Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our method achieves state-of-the-art results in the ND setting, and is especially effective for the near-ND setting, where we improve over existing work by a large margin of up to 8% in AUROC.

The review comment to be evaluated is:

The paper investigates the near anomalies detection and points a significant decrease in performance of state of the art approaches on near anomalies cases. A new solution is proposed: 1) using SDE models trained to generate data from the training distribution. 2) generate a dataset from the trained SDE. 3) train a binary classifier on top of a feature extractor to distinguish between the generated data and real data. 4) use nearest neighbour for detecting anomalies.
Extensive experiments and empirical analysis are carried on. The method shows good results.

The evidence to be evaluated is:In this section, we compare our method against the SOTA in the ND and near-ND setups.

We compare our approach with current top self-supervised and pre-trained feature adaptation methods in the literature (44; 34; 33; 7; 16).

Table 1(a) provides the results of the case, which indicates our superiority. To evaluate the performance on near-ND benchmark, we use 4 different datasets that naturally show slight differences between their class distributions. These datasets are not commonly used in ND but used in fine-grained image classification or segmentation, containing hard-to-distinguish object classes. Thus they are suitable for benchmarking near-ND.

Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

In this setup, each class of CIFAR-10 (\(C_{i}^{10}\)) and its closest class in CIFAR-100 (\(C_{j}^{100}\)) are selected, to represent normal and abnormal classes, respectively, where \(i\) and \(j\) are class numbers that are less than 10 and 100, respectively. Then, the model is trained on the training samples of (\(C_{i}^{10}\)) and tested against the aggregation of (\(C_{j}^{100}\)) and (\(C_{i}^{10}\)) test sets. As it is shown, almost all the methods considerably lag behind our method by a large margin.

Table 1 (c) compares our method with other SOTA methods in detecting such samples. Due to learning more distinguishing features, we pass the SOTA by 12 percentage points in the AUROC score, showing the applicability of our approach in detecting a wide range of anomalies. While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting.

The results are provided for both the ND and near-ND evaluation setups.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Table 3 shows clear superiority of using a SDE-based model compared to the well-known generative models (20; 42; 5; 14) with 2% to 20% better AUROCs based on the chosen dataset.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting. The improvements that are made by our method are consistent across wide variety of datasets, and choices of the backbone model.

As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 3: Comparison of our model performance (in AUROC %) upon using different generative models for each class of the CIFAR-10, Weather, WBC and MvTecAD dataset. The performance of the base backbone is also included to measure the effectiveness of the training process. Diffusionbased data generation considerably passes the GAN-based SOTAs in the novelty detection task. Coverage&Density metrics are calculated according to the dataset and generative model. These metrics measure the diversity and fidelity of the samples that are generated by generative models.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Summary: The evidence strongly supports this claim through comprehensive experimental results. Tables 1(a,b,c) show clear performance improvements in both ND and near-ND settings across multiple datasets. For the near-ND setting specifically, Table 1(b) demonstrates consistent improvements of ~4-8% AUROC over SOTA methods across different datasets. Table 1(c) further validates the 8% improvement claim for CIFAR-10vs100. The review comment corroborates the method's effectiveness. While the 8% improvement claim is specific to certain datasets/settings and could have been more precisely qualified, the evidence thoroughly substantiates the core performance claims.

<score>1</score>

Justification:
1. The claim is precise in specifying "up to 8%" rather than claiming a universal 8% improvement
2. Tables 1(a,b,c) provide extensive empirical evidence showing consistent improvements
3. The improvements are demonstrated across multiple datasets and experimental settings
4. The review comment independently validates the method's strong performance
5. Performance gains are well-documented with specific metrics and baselines


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our method achieves state-of-the-art results in the ND setting, and is especially effective for the near-ND setting, where we improve over existing work by a large margin of up to 8% in AUROC.

The review comment to be evaluated is:

* There is a large room for improvement in clarity.
    * The paper should be clearer on how exactly the generative model is trained and used. The most important fact, in my opinion, is that the score-based generative model is trained on in-distribution data and terminated prematurely. However, the fact is only briefly mentioned in Page 2, and then mentioned in Figure 2. Given the importance of this information, this point should be described in detail in Section 2.1. Also, other details and design choices, for example, how early the training of the score-based model should be stopped, needs to be elaborated.
    * The experimental setup described in Section 3.2 is difficult to follow. There are several comparison settings but are not well organized.
* Discussion on the differences among diffusion models, GANs, and VAEs in Section 2.2 are not very persuasive. It does not have concrete examples or quantitative arguments. Plus, the argument of Section 3.2 is mostly about generative modeling itself and not about its application to generating auxiliary data in novelty detection.
* When performing CIFAR-10 vs CIFAR-100 experiment, it is more standard and more challenging to incorporate all the classes, instead of using only one class from each dataset. Overall, I am not convinced that the experiment settings used in the paper are standard.
* Replacing a GAN with a diffusion model in the data generator for novelty detection itself is not a very novel contribution to the community.

The evidence to be evaluated is:In this section, we compare our method against the SOTA in the ND and near-ND setups.

We compare our approach with current top self-supervised and pre-trained feature adaptation methods in the literature (44; 34; 33; 7; 16).

Table 1(a) provides the results of the case, which indicates our superiority. To evaluate the performance on near-ND benchmark, we use 4 different datasets that naturally show slight differences between their class distributions. These datasets are not commonly used in ND but used in fine-grained image classification or segmentation, containing hard-to-distinguish object classes. Thus they are suitable for benchmarking near-ND.

Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

In this setup, each class of CIFAR-10 (\(C_{i}^{10}\)) and its closest class in CIFAR-100 (\(C_{j}^{100}\)) are selected, to represent normal and abnormal classes, respectively, where \(i\) and \(j\) are class numbers that are less than 10 and 100, respectively. Then, the model is trained on the training samples of (\(C_{i}^{10}\)) and tested against the aggregation of (\(C_{j}^{100}\)) and (\(C_{i}^{10}\)) test sets. As it is shown, almost all the methods considerably lag behind our method by a large margin.

Table 1 (c) compares our method with other SOTA methods in detecting such samples. Due to learning more distinguishing features, we pass the SOTA by 12 percentage points in the AUROC score, showing the applicability of our approach in detecting a wide range of anomalies. While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting.

The results are provided for both the ND and near-ND evaluation setups.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Table 3 shows clear superiority of using a SDE-based model compared to the well-known generative models (20; 42; 5; 14) with 2% to 20% better AUROCs based on the chosen dataset.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting. The improvements that are made by our method are consistent across wide variety of datasets, and choices of the backbone model.

As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 3: Comparison of our model performance (in AUROC %) upon using different generative models for each class of the CIFAR-10, Weather, WBC and MvTecAD dataset. The performance of the base backbone is also included to measure the effectiveness of the training process. Diffusionbased data generation considerably passes the GAN-based SOTAs in the novelty detection task. Coverage&Density metrics are calculated according to the dataset and generative model. These metrics measure the diversity and fidelity of the samples that are generated by generative models.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence:

The claim states their method achieves SOTA results in ND and improves over existing work by "up to 8%" in near-ND AUROC. The evidence supports this through:
- Table 1 showing superior performance across datasets
- Quantitative results demonstrating 8-12% improvements in near-ND scenarios
- Extensive experimental validation across multiple datasets and backbones
- Detailed ablation studies and comparisons with other methods

While the improvements are significant and well-documented, the claim is modestly stated and avoids exaggeration by:
- Using precise language ("up to 8%")
- Being specific about the settings
- Having comprehensive evidence to back the stated margins

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our method achieves state-of-the-art results in the ND setting, and is especially effective for the near-ND setting, where we improve over existing work by a large margin of up to 8% in AUROC.

The review comment to be evaluated is:

The idea of this paper is very natural: try to generate near-distribution outliers to augment the original training dataset which only contains normal data (semi-supervised learning), such that we can fine-tune the pre-trained feature extractor by training a binary classifier.

Strengths:
(1) Using generated outliers to improve the accuracy of novelty detection is not new. There have been some similar works which use GANs instead of SDE.  The main contribution of this paper is that they replace the GANs with the diffusion model (which is increasingly popular recently).  The motivation here is that the outliers generated by SDE are “closer“ to the nominal distribution, and then using these "boarder" points  to deal with the near-ND setting.  Even though the pipeline and SDE are all from existing works, the novel combination is meaningful considering the theoretically and experimental results.

(2) The experimental results show that the proposed method considerably improves over existing models (especially for the near-ND setting).  It is not surprising that SDE over-perform GANs for the near-ND setting, but I am surprised that the improvement is so obvious.  It will be great if the code can be provided.

Weakness:
(1) In Section 2.2 (Diffusion models vs. GANs), they try to explain, for near-ND, why choosing diffusion models instead of other generative modes (GAN and VAE), it would be better to include flow-based models in this part. Also,  VAE-related results are not included in figure3.

(2) In Figure3,  density, coverage and FID are used as the metrics to measure the fidelity and diversity.  Could you add a brief introductions/definitions for these metrics? Also, I am confused how to get the "density" from GAN-based models as they are not tractable models.

(3)About Stopping point: In section 4, "the method is robust against the stopping point with at most 2% variation for the reasonable FID scores". However,  from table4,  it is difficult to get this conclusion. what is "reasonable" FID? Actually, the stopping point decide how similar the fake points and the real points. How to decide when to stop ?

Other minor comments:
The "normal distribution" used in this paper sometimes are confusing, which means the distribution of nominal data instead of Gaussian distribution. It would be better to use other notations.

The evidence to be evaluated is:In this section, we compare our method against the SOTA in the ND and near-ND setups.

We compare our approach with current top self-supervised and pre-trained feature adaptation methods in the literature (44; 34; 33; 7; 16).

Table 1(a) provides the results of the case, which indicates our superiority. To evaluate the performance on near-ND benchmark, we use 4 different datasets that naturally show slight differences between their class distributions. These datasets are not commonly used in ND but used in fine-grained image classification or segmentation, containing hard-to-distinguish object classes. Thus they are suitable for benchmarking near-ND.

Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

In this setup, each class of CIFAR-10 (\(C_{i}^{10}\)) and its closest class in CIFAR-100 (\(C_{j}^{100}\)) are selected, to represent normal and abnormal classes, respectively, where \(i\) and \(j\) are class numbers that are less than 10 and 100, respectively. Then, the model is trained on the training samples of (\(C_{i}^{10}\)) and tested against the aggregation of (\(C_{j}^{100}\)) and (\(C_{i}^{10}\)) test sets. As it is shown, almost all the methods considerably lag behind our method by a large margin.

Table 1 (c) compares our method with other SOTA methods in detecting such samples. Due to learning more distinguishing features, we pass the SOTA by 12 percentage points in the AUROC score, showing the applicability of our approach in detecting a wide range of anomalies. While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting.

The results are provided for both the ND and near-ND evaluation setups.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Table 3 shows clear superiority of using a SDE-based model compared to the well-known generative models (20; 42; 5; 14) with 2% to 20% better AUROCs based on the chosen dataset.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting. The improvements that are made by our method are consistent across wide variety of datasets, and choices of the backbone model.

As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 3: Comparison of our model performance (in AUROC %) upon using different generative models for each class of the CIFAR-10, Weather, WBC and MvTecAD dataset. The performance of the base backbone is also included to measure the effectiveness of the training process. Diffusionbased data generation considerably passes the GAN-based SOTAs in the novelty detection task. Coverage&Density metrics are calculated according to the dataset and generative model. These metrics measure the diversity and fidelity of the samples that are generated by generative models.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim appears to be well-supported by comprehensive experimental results shown in Tables 1 and 3. The method demonstrates consistent improvement over SOTA across multiple datasets, particularly for near-ND settings with improvements up to 8% AUROC (e.g., WBC dataset shows 91.2% vs 85.1% baseline). The evidence includes detailed performance comparisons, ablation studies, and visual examples that validate the claimed improvements. The reviewer also notes the "obvious improvement" and validates the novelty of using diffusion models for this task. The claim appropriately reflects the evidence without exaggeration.

<score>1</score>

Justification:
1. Tables 1(a,b,c) provide direct numerical evidence supporting the claimed improvements in both ND and near-ND settings
2. Table 3 demonstrates superiority over other generative approaches
3. Visual examples in Figures 10-12 demonstrate the quality of generated samples
4. Reviewer confirms the significance of improvements
5. Claim language is precise and quantified ("up to 8%") rather than using vague superlatives
6. Results are consistent across multiple datasets and experimental settings


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our method achieves state-of-the-art results in the ND setting, and is especially effective for the near-ND setting, where we improve over existing work by a large margin of up to 8% in AUROC.

The review comment to be evaluated is:

### Strengths

- The paper is, for the most part, well-written, easy-to-follow, and does a good job of
contextualising and motivating the work.
- The proposed method is simple and intuitive, and seems to perform well empirically
compared with the baselines in both the regular ND setting and the near ND setting, showing a
significant reduction in the performance gap of going from the former to the latter.
the NR to near-NR setting.
- The figures are well-chosen and well-put-together: they convey well the essential points of the
  paper, such as the shortcomings of existing methods and the feasibility of using GANs vs diffusion
  models for creating the synthetic data.
- Evaluation performed using a good range and selection of baseline methods and benchmark datasets.
- Results are aggregated over a good number (10) of replicates.

### Weaknesses
- Despite the authors claiming otherwise, the method is quite sensitive to the stopping criterion in
  the near ND setting with an empirical range of [68.2, 90.0]; in practice, different datasets may
  require wildly different FID thresholds and there is no reliable way of validating this given the
  nature of the problem. Also, I wonder if the number of sampling steps and the choice of sampler have a
  significant effect on the results -- there doesn't seem to be any mention of this.
  Taking this a step further, might it be feasible to train the diffusion model fully and then rely
  upon early-stopping the reverse diffusion process according to the FID score?
- The need to train, and indeed sample, from a diffusion model can be computationally burdensome for
  higher-resolution datasets.
- The description of the SDE model is a little overdone; a much briefer description with a pointing of the
  reader to the original paper would be sufficient. This is especially the case given that, as
  I understand it, the proposed method should work with any diffusion based model (Imagen, Parti,
  DALLE-2, etc.) or are there theoretically some constraints on the form said model and the
  associated sampler should take?
- While this applies to several methods of the same class, the method seems to make certain
assumptions about the distribution of the ID and OOD data that might not hold in the wild.
- While results are aggregated over a good number (10) of replicates no measure of dispersion
  (standard deviation/error) seems to be reported alongside the mean in each case.
- Some typographic and syntactical errors (e.g. 'that unlikely come' (line 1 of the introduction)).

The evidence to be evaluated is:In this section, we compare our method against the SOTA in the ND and near-ND setups.

We compare our approach with current top self-supervised and pre-trained feature adaptation methods in the literature (44; 34; 33; 7; 16).

Table 1(a) provides the results of the case, which indicates our superiority. To evaluate the performance on near-ND benchmark, we use 4 different datasets that naturally show slight differences between their class distributions. These datasets are not commonly used in ND but used in fine-grained image classification or segmentation, containing hard-to-distinguish object classes. Thus they are suitable for benchmarking near-ND.

Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

In this setup, each class of CIFAR-10 (\(C_{i}^{10}\)) and its closest class in CIFAR-100 (\(C_{j}^{100}\)) are selected, to represent normal and abnormal classes, respectively, where \(i\) and \(j\) are class numbers that are less than 10 and 100, respectively. Then, the model is trained on the training samples of (\(C_{i}^{10}\)) and tested against the aggregation of (\(C_{j}^{100}\)) and (\(C_{i}^{10}\)) test sets. As it is shown, almost all the methods considerably lag behind our method by a large margin.

Table 1 (c) compares our method with other SOTA methods in detecting such samples. Due to learning more distinguishing features, we pass the SOTA by 12 percentage points in the AUROC score, showing the applicability of our approach in detecting a wide range of anomalies. While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting.

The results are provided for both the ND and near-ND evaluation setups.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Table 3 shows clear superiority of using a SDE-based model compared to the well-known generative models (20; 42; 5; 14) with 2% to 20% better AUROCs based on the chosen dataset.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting. The improvements that are made by our method are consistent across wide variety of datasets, and choices of the backbone model.

As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 3: Comparison of our model performance (in AUROC %) upon using different generative models for each class of the CIFAR-10, Weather, WBC and MvTecAD dataset. The performance of the base backbone is also included to measure the effectiveness of the training process. Diffusionbased data generation considerably passes the GAN-based SOTAs in the novelty detection task. Coverage&Density metrics are calculated according to the dataset and generative model. These metrics measure the diversity and fidelity of the samples that are generated by generative models.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim of achieving state-of-the-art results and improving over existing work by up to 8% AUROC in the near-ND setting is supported by the comprehensive empirical results shown in Table 1(b). The results demonstrate consistent improvements across multiple datasets, with performance gains ranging from 4-8% over previous SOTA methods. However, the reviewer points out sensitivity to stopping criteria with empirical range of [68.2, 90.0], suggesting some instability in performance that isn't acknowledged in the claim. While the core performance claim is accurate, it would be more well-stated if it acknowledged this limitation.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We address this challenge through variants of PSRO in which all RL is applied to environments where opponents play pure strategies.

The evidence to be evaluated is:A _normal form games_ (NFG) \(\Gamma\!=\!(\Pi,\!U,\!n)\) describes a one-shot strategic interaction among \(n\) players.

Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In PSRO, the strategy exploration problem is decomposed into two steps: solution and BR via RL.

In the BR step, PSRO generates a new policy via RL (the _response oracle_), training against the target opponent profile computed by the MSS.

In experiments reported here, we adopt NE as the baseline MSS; however, our methods readily apply to any MSS. We observe and address two main problems with the standard version of PSRO (Algorithm 1). First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs. This suggests a compelling opportunity to transfer learning across epochs; however, the BR calculation step works by training anew. Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG. Secondly, the opponent profiles are mixtures (i.e., distribution over opponent policies). This makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

Consider the Rock-Paper-Scissors (R-P-S) game in figure 1. Player 1's initial strategy \(\pi_{1}^{1}\) mostly plays S, which induces a best response of R. Player 1's second strategy \(\pi_{1}^{2}\) is an approximate best response to R, scoring \(80\%\) against it. The best response to \(\pi_{1}^{2}\) is P. This leads to an empirical game similar to matching pennies, with player 2 playing an even mixture of Rock and Paper; in particular, player 2's meta-strategy never plays S, despite it being good against both player 1 strategies so far. If, as in PSRO, player 1 then adds their BR to this meta-strategy, which is P, then their strategy set still does not contain the Nash equilibrium of the game.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

Input: Initial policy sets for all players \(\Pi^{0}\)  Simulate utilities \(\tilde{U}^{\Pi^{0}}\) for each joint \(\pi\!\in\!\Pi^{0}\)  Initialize solutions \(\sigma_{i}^{*,0}\!=\!\text{Uniform}(\Pi_{i}^{0})\) whileepoch e in \(\{1,2,...\}\)do for player \(i\!\in\![[n]]\)do for many episodesdo \(\pi_{-i}\!\sim\!\sigma_{-i}^{*-1}\)  Train \(\pi_{i}^{e}\) over \(\tau\!\sim\!(\pi_{i}^{e}\!,\!\pi_{-i})\) \(\Pi_{i}^{e}\!=\!\Pi_{i}^{-1}\!\cup\!\{\pi_{i}^{e}\}\)  Simulate missing entries in \(\tilde{U}^{\Pi^{e}}\) from \(\Pi^{e}\)  Compute a solution \(\sigma^{*,e}\) from \(\tilde{\Gamma}^{e}\) Output: Current solution \(\sigma_{i}^{*,e}\) for player \(i\) ```**Algorithm 1**Policy-Space Response

First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy. Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters. A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: _pure-hparams_ performed best on-average against each individual test-opponent, and _mix-hparams_ performed best against a uniform mixture of the test-opponents. These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.

We used two sets of hyperparameters: specialized for low- and high-variance outcomes of state, which are a result of facing pure- and mixed-strategy opponents respectively.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget.

The trends observed in the previous experiment reoccur: Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Schvartzman and Wellman (2009b) followed this up by introducing reinforcement learning as the approximate oracle method. These algorithms, alongside a trend of deep reinforcement learning being used to solve difficult games, set the stage for the PSRO framework. PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

PSRO's strategy exploration happens in two phases: first, a MSS produces an opponent-profile as a learning objective, then reinforcement learning is used as an approximate oracle to best-respond to the objective.

Similarly, self-play's resulting profile is a pure strategy containing only the newest policy.

PSRO's strategy exploration occurs in two steps: solution and BR via RL.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that it is a well-stated claim supported by the detailed technical evidence provided. The paper thoroughly demonstrates how both Mixed-Oracles and Mixed-Opponents algorithms modify PSRO to work with pure strategies. The evidence includes mathematical formulations, algorithmic details, and experimental results (shown in Figure 4) that support this approach. The text explicitly states how Mixed-Opponents "modifies the BR training by constructing a single opponent policy" and Mixed-Oracles learns best responses to individual opponent policies rather than mixed profiles. The claim accurately reflects what is demonstrated without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We address this challenge through variants of PSRO in which all RL is applied to environments where opponents play pure strategies.

The review comment to be evaluated is:

The paper suggests two techniques to improve the calculation of empirically figuring out a Nash equilibrium using an iterative application of best-response dynamics. One method learns the best-response to the previously used strategy. The other uses that technique to model the opponent, and then best-responds to the modeled opponent. The experiments show a faster reaching to NE than without these changes.

The paper is well-written and explained, and is accessible even to researchers not well-versed in ML topics. While, the suggested changes are rather straightforward, they do indeed lead to the expected advance (shorter times to reach NE). I was convinced by further introspection that this is a significant enough contribution, to merit acceptance.

Of course, a more significant change to the algorithm, leading not only to a shorter time but to convergence to better equilibria (in cases where multiple exist) would be far more compelling.

The evidence to be evaluated is:A _normal form games_ (NFG) \(\Gamma\!=\!(\Pi,\!U,\!n)\) describes a one-shot strategic interaction among \(n\) players.

Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In PSRO, the strategy exploration problem is decomposed into two steps: solution and BR via RL.

In the BR step, PSRO generates a new policy via RL (the _response oracle_), training against the target opponent profile computed by the MSS.

In experiments reported here, we adopt NE as the baseline MSS; however, our methods readily apply to any MSS. We observe and address two main problems with the standard version of PSRO (Algorithm 1). First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs. This suggests a compelling opportunity to transfer learning across epochs; however, the BR calculation step works by training anew. Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG. Secondly, the opponent profiles are mixtures (i.e., distribution over opponent policies). This makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

Consider the Rock-Paper-Scissors (R-P-S) game in figure 1. Player 1's initial strategy \(\pi_{1}^{1}\) mostly plays S, which induces a best response of R. Player 1's second strategy \(\pi_{1}^{2}\) is an approximate best response to R, scoring \(80\%\) against it. The best response to \(\pi_{1}^{2}\) is P. This leads to an empirical game similar to matching pennies, with player 2 playing an even mixture of Rock and Paper; in particular, player 2's meta-strategy never plays S, despite it being good against both player 1 strategies so far. If, as in PSRO, player 1 then adds their BR to this meta-strategy, which is P, then their strategy set still does not contain the Nash equilibrium of the game.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

Input: Initial policy sets for all players \(\Pi^{0}\)  Simulate utilities \(\tilde{U}^{\Pi^{0}}\) for each joint \(\pi\!\in\!\Pi^{0}\)  Initialize solutions \(\sigma_{i}^{*,0}\!=\!\text{Uniform}(\Pi_{i}^{0})\) whileepoch e in \(\{1,2,...\}\)do for player \(i\!\in\![[n]]\)do for many episodesdo \(\pi_{-i}\!\sim\!\sigma_{-i}^{*-1}\)  Train \(\pi_{i}^{e}\) over \(\tau\!\sim\!(\pi_{i}^{e}\!,\!\pi_{-i})\) \(\Pi_{i}^{e}\!=\!\Pi_{i}^{-1}\!\cup\!\{\pi_{i}^{e}\}\)  Simulate missing entries in \(\tilde{U}^{\Pi^{e}}\) from \(\Pi^{e}\)  Compute a solution \(\sigma^{*,e}\) from \(\tilde{\Gamma}^{e}\) Output: Current solution \(\sigma_{i}^{*,e}\) for player \(i\) ```**Algorithm 1**Policy-Space Response

First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy. Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters. A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: _pure-hparams_ performed best on-average against each individual test-opponent, and _mix-hparams_ performed best against a uniform mixture of the test-opponents. These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.

We used two sets of hyperparameters: specialized for low- and high-variance outcomes of state, which are a result of facing pure- and mixed-strategy opponents respectively.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget.

The trends observed in the previous experiment reoccur: Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Schvartzman and Wellman (2009b) followed this up by introducing reinforcement learning as the approximate oracle method. These algorithms, alongside a trend of deep reinforcement learning being used to solve difficult games, set the stage for the PSRO framework. PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

PSRO's strategy exploration happens in two phases: first, a MSS produces an opponent-profile as a learning objective, then reinforcement learning is used as an approximate oracle to best-respond to the objective.

Similarly, self-play's resulting profile is a pure strategy containing only the newest policy.

PSRO's strategy exploration occurs in two steps: solution and BR via RL.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my analysis of the evidence and review comments:

The claim "We address this challenge through variants of PSRO in which all RL is applied to environments where opponents play pure strategies" appears well-stated and supported by the evidence. The paper clearly demonstrates two variants (Mixed-Oracles and Mixed-Opponents) that modify PSRO to train against pure strategies rather than mixed strategies. The evidence shows detailed mathematical formulations and experimental results supporting this approach, with Figure 4 demonstrating improved performance. The review comments also confirm these findings, noting that while the changes are straightforward, they lead to faster Nash equilibrium convergence.

<score>1</score>

Justification:
- The evidence explicitly shows the algorithms modify PSRO to train against pure strategies
- Mathematical formulations and experimental results directly support the claim
- Review comments validate that the approach works as claimed
- No exaggeration of results or unsupported assertions
- Clear acknowledgment of limitations in both paper and review
- Results presented with appropriate scope and context


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We address this challenge through variants of PSRO in which all RL is applied to environments where opponents play pure strategies.

The review comment to be evaluated is:

The paper focuses on resolving the computational and sample efficiency challenges with current PSRO style approaches. To this end it proposes two different modifications to the standard PSRO setup: 1) Mixed Oracles, and 2) Mixed Opponents. These approaches allow avoiding resetting learning after each outer loop epoch and reduce the stochasticity of dynamics during training. Thee efficacy is demonstrated on relatively simple games but using Deep RL policies where the proposed approaches are at least on par with standard PSRO approach in terms of final performance while drastically improving the sample efficiency.

Although the paper has quite a few typos and unclear writing in places, it has a fantastic setup with clear description of the problems with standard PSRO it's trying to resolve. The paper points out the clear computational deficiencies with current PSRO style approaches: 1) starting anew everytime, 2) slower exploration of strategy space, 3) stochastic dynamics making learning difficult.

The paper is mostly focused on two-player zero sum games.
Approach heavily dependent on efficacy of Q-mixing approach (although the idea is more general), so currently limited to discrete action problems.
Moreover, since there are no alternatives to Q-mixing, we don't actually get any understanding/intuition for why this works and how important is the performance of "TransferOracle" or "OpponentOracle".

Although section 4.4 suggests that the two stage hyperparameter selection wasn't as important, results in Fig 6b are too noisy to fully accept. It's unclear how much compute/samples were required for the hyper parameter selection. In general the results are a lot more noisier for the proposed approaches vs standard PSRO.

**Minor**
- Environment dynamics should be SxA rather than OxA
- Algorithm 4.1 does not exist?
- Fig 1. Is it because of PSRO with Nash. Would a different MSS work differently?
- "proposes MCC as a solution concept": not everyone would know what MCC refers to.

The evidence to be evaluated is:A _normal form games_ (NFG) \(\Gamma\!=\!(\Pi,\!U,\!n)\) describes a one-shot strategic interaction among \(n\) players.

Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In PSRO, the strategy exploration problem is decomposed into two steps: solution and BR via RL.

In the BR step, PSRO generates a new policy via RL (the _response oracle_), training against the target opponent profile computed by the MSS.

In experiments reported here, we adopt NE as the baseline MSS; however, our methods readily apply to any MSS. We observe and address two main problems with the standard version of PSRO (Algorithm 1). First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs. This suggests a compelling opportunity to transfer learning across epochs; however, the BR calculation step works by training anew. Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG. Secondly, the opponent profiles are mixtures (i.e., distribution over opponent policies). This makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

Consider the Rock-Paper-Scissors (R-P-S) game in figure 1. Player 1's initial strategy \(\pi_{1}^{1}\) mostly plays S, which induces a best response of R. Player 1's second strategy \(\pi_{1}^{2}\) is an approximate best response to R, scoring \(80\%\) against it. The best response to \(\pi_{1}^{2}\) is P. This leads to an empirical game similar to matching pennies, with player 2 playing an even mixture of Rock and Paper; in particular, player 2's meta-strategy never plays S, despite it being good against both player 1 strategies so far. If, as in PSRO, player 1 then adds their BR to this meta-strategy, which is P, then their strategy set still does not contain the Nash equilibrium of the game.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

Input: Initial policy sets for all players \(\Pi^{0}\)  Simulate utilities \(\tilde{U}^{\Pi^{0}}\) for each joint \(\pi\!\in\!\Pi^{0}\)  Initialize solutions \(\sigma_{i}^{*,0}\!=\!\text{Uniform}(\Pi_{i}^{0})\) whileepoch e in \(\{1,2,...\}\)do for player \(i\!\in\![[n]]\)do for many episodesdo \(\pi_{-i}\!\sim\!\sigma_{-i}^{*-1}\)  Train \(\pi_{i}^{e}\) over \(\tau\!\sim\!(\pi_{i}^{e}\!,\!\pi_{-i})\) \(\Pi_{i}^{e}\!=\!\Pi_{i}^{-1}\!\cup\!\{\pi_{i}^{e}\}\)  Simulate missing entries in \(\tilde{U}^{\Pi^{e}}\) from \(\Pi^{e}\)  Compute a solution \(\sigma^{*,e}\) from \(\tilde{\Gamma}^{e}\) Output: Current solution \(\sigma_{i}^{*,e}\) for player \(i\) ```**Algorithm 1**Policy-Space Response

First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy. Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters. A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: _pure-hparams_ performed best on-average against each individual test-opponent, and _mix-hparams_ performed best against a uniform mixture of the test-opponents. These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.

We used two sets of hyperparameters: specialized for low- and high-variance outcomes of state, which are a result of facing pure- and mixed-strategy opponents respectively.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget.

The trends observed in the previous experiment reoccur: Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Schvartzman and Wellman (2009b) followed this up by introducing reinforcement learning as the approximate oracle method. These algorithms, alongside a trend of deep reinforcement learning being used to solve difficult games, set the stage for the PSRO framework. PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

PSRO's strategy exploration happens in two phases: first, a MSS produces an opponent-profile as a learning objective, then reinforcement learning is used as an approximate oracle to best-respond to the objective.

Similarly, self-play's resulting profile is a pure strategy containing only the newest policy.

PSRO's strategy exploration occurs in two steps: solution and BR via RL.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "We address this challenge through variants of PSRO in which all RL is applied to environments where opponents play pure strategies" appears somewhat overstated. While the Mixed-Oracles and Mixed-Opponents algorithms do involve training against pure strategies in some capacity, the evidence shows that Mixed-Opponents still uses mixed strategies through Q-Mixing to combine opponent behaviors. The reviewer also notes limitations regarding Q-mixing dependency. The claim oversimplifies the actual approach, which involves both pure and mixed strategy elements.

<score>4</score>

Justification:
1. The evidence shows that Mixed-Oracles learns best responses to individual pure strategies, but then combines these responses using Q-Mixing for mixed opponent profiles.
2. Mixed-Opponents combines opponent strategies into a single policy using Q-Mixing, which isn't purely training against pure strategies.
3. The reviewer points out the heavy dependence on Q-mixing, suggesting the approach isn't as straightforward as claimed.
4. While the methods do leverage pure strategy training, the claim overstates this aspect by suggesting "all RL" is applied to pure strategy environments.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We address this challenge through variants of PSRO in which all RL is applied to environments where opponents play pure strategies.

The review comment to be evaluated is:

Summary
The paper proposes two new methods in the Policy-Space Response Oracle framework. These approaches permit to reuse past knowledge in order to reduce the amount of data required for the RL training. The first algorithm Mixed-Oracles transfers the previous iteration of Deep RL, instead of the second one, Mixed-Opponents, transfers existing strategy action-value estimates.

Strengths
The paper proposes two convincing alternatives to reuse previous knowledge in the PSRO framework. The two ideas are based on Q-mixing approach: the first one uses this method to transfer Q-values across epochs, the second one to design a new training objective.
The experiments show that the proposed methods find a good solution using less simulation than the original PSRO framework.

Weakness
The paper is not very novel, since it uses previous approaches (PSRO and Q-mixing) to transfer knowledge for the PSRO framework.
I am not aware of recent works in this framework but could be useful to compare the proposed approaches with P2RO [1].
The Mixed-Opponent section needs a better explanation of the use of Q-mixing as a training objective.

[1]  McAleer, S., Lanier, J., Fox, R., & Baldi, P. (2020). Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. arXiv preprint arXiv:2006.08555.

The evidence to be evaluated is:A _normal form games_ (NFG) \(\Gamma\!=\!(\Pi,\!U,\!n)\) describes a one-shot strategic interaction among \(n\) players.

Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In PSRO, the strategy exploration problem is decomposed into two steps: solution and BR via RL.

In the BR step, PSRO generates a new policy via RL (the _response oracle_), training against the target opponent profile computed by the MSS.

In experiments reported here, we adopt NE as the baseline MSS; however, our methods readily apply to any MSS. We observe and address two main problems with the standard version of PSRO (Algorithm 1). First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs. This suggests a compelling opportunity to transfer learning across epochs; however, the BR calculation step works by training anew. Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG. Secondly, the opponent profiles are mixtures (i.e., distribution over opponent policies). This makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

Consider the Rock-Paper-Scissors (R-P-S) game in figure 1. Player 1's initial strategy \(\pi_{1}^{1}\) mostly plays S, which induces a best response of R. Player 1's second strategy \(\pi_{1}^{2}\) is an approximate best response to R, scoring \(80\%\) against it. The best response to \(\pi_{1}^{2}\) is P. This leads to an empirical game similar to matching pennies, with player 2 playing an even mixture of Rock and Paper; in particular, player 2's meta-strategy never plays S, despite it being good against both player 1 strategies so far. If, as in PSRO, player 1 then adds their BR to this meta-strategy, which is P, then their strategy set still does not contain the Nash equilibrium of the game.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

Input: Initial policy sets for all players \(\Pi^{0}\)  Simulate utilities \(\tilde{U}^{\Pi^{0}}\) for each joint \(\pi\!\in\!\Pi^{0}\)  Initialize solutions \(\sigma_{i}^{*,0}\!=\!\text{Uniform}(\Pi_{i}^{0})\) whileepoch e in \(\{1,2,...\}\)do for player \(i\!\in\![[n]]\)do for many episodesdo \(\pi_{-i}\!\sim\!\sigma_{-i}^{*-1}\)  Train \(\pi_{i}^{e}\) over \(\tau\!\sim\!(\pi_{i}^{e}\!,\!\pi_{-i})\) \(\Pi_{i}^{e}\!=\!\Pi_{i}^{-1}\!\cup\!\{\pi_{i}^{e}\}\)  Simulate missing entries in \(\tilde{U}^{\Pi^{e}}\) from \(\Pi^{e}\)  Compute a solution \(\sigma^{*,e}\) from \(\tilde{\Gamma}^{e}\) Output: Current solution \(\sigma_{i}^{*,e}\) for player \(i\) ```**Algorithm 1**Policy-Space Response

First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy. Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters. A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: _pure-hparams_ performed best on-average against each individual test-opponent, and _mix-hparams_ performed best against a uniform mixture of the test-opponents. These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.

We used two sets of hyperparameters: specialized for low- and high-variance outcomes of state, which are a result of facing pure- and mixed-strategy opponents respectively.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget.

The trends observed in the previous experiment reoccur: Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Schvartzman and Wellman (2009b) followed this up by introducing reinforcement learning as the approximate oracle method. These algorithms, alongside a trend of deep reinforcement learning being used to solve difficult games, set the stage for the PSRO framework. PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

PSRO's strategy exploration happens in two phases: first, a MSS produces an opponent-profile as a learning objective, then reinforcement learning is used as an approximate oracle to best-respond to the objective.

Similarly, self-play's resulting profile is a pure strategy containing only the newest policy.

PSRO's strategy exploration occurs in two steps: solution and BR via RL.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Looking at the evidence, I'll evaluate the claim: "We address this challenge through variants of PSRO in which all RL is applied to environments where opponents play pure strategies."

This claim appears somewhat overstated. While the evidence shows that Mixed-Oracles does train against pure strategies (individual opponent policies), Mixed-Opponents actually combines opponent strategies using Q-Mixing to create a single opponent policy. The paper clearly describes both approaches, and Mixed-Opponents specifically deals with mixed strategies through policy combination. The experimental results (Figure 4) demonstrate both methods are effective, but the claim oversimplifies the actual mechanisms used, particularly for Mixed-Opponents.

<score>4</score>

The score of 4 reflects that the claim overstates what the evidence shows by suggesting all RL is applied to pure strategy environments, when in fact only Mixed-Oracles strictly follows this approach, while Mixed-Opponents uses a different mechanism involving mixed strategies.