Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Learning against a single policy mitigates variance in state outcomes that is induced by an unobserved distribution of opponents.

The review comment to be evaluated is:

The paper suggests two techniques to improve the calculation of empirically figuring out a Nash equilibrium using an iterative application of best-response dynamics. One method learns the best-response to the previously used strategy. The other uses that technique to model the opponent, and then best-responds to the modeled opponent. The experiments show a faster reaching to NE than without these changes.

The paper is well-written and explained, and is accessible even to researchers not well-versed in ML topics. While, the suggested changes are rather straightforward, they do indeed lead to the expected advance (shorter times to reach NE). I was convinced by further introspection that this is a significant enough contribution, to merit acceptance.

Of course, a more significant change to the algorithm, leading not only to a shorter time but to convergence to better equilibria (in cases where multiple exist) would be far more compelling.

The evidence to be evaluated is:A _normal form games_ (NFG) \(\Gamma\!=\!(\Pi,\!U,\!n)\) describes a one-shot strategic interaction among \(n\) players. When there is more than one agent, we denote agent-specific components with subscripts (e.g., \(\pi_{i}\)). Negated subscripts represent the joint elements of all other agents (e.g., \(\pi_{-i}\)). Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

The sum of regrets across all players \(\text{SumRegret}(\sigma,\overline{\Pi})=\sum_{i\in n}\text{Regret}_{i}(\sigma, \overline{\Pi}_{i})\), sometimes called the Nash convergence, is a measure of how stable a solution is. At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs.

Secondly, the opponent profiles are mixtures (i.e., distribution over opponent policies). This makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult.

In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy. Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters. A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: _pure-hparams_ performed best on-average against each individual test-opponent, and _mix-hparams_ performed best against a uniform mixture of the test-opponents. These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.

We used two sets of hyperparameters: specialized for low- and high-variance outcomes of state, which are a result of facing pure- and mixed-strategy opponents respectively. This was motivated by the assumption that lower variance would require less training.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "Learning against a single policy mitigates variance in state outcomes that is induced by an unobserved distribution of opponents" appears to be well-supported. The paper presents two methods (Mixed-Oracles and Mixed-Opponents) that address variance by either learning against individual opponent policies or creating an aggregate opponent policy. The evidence includes theoretical justification and experimental results (Figure 4) showing improved performance. While the claim is relatively modest in scope and accurately reflects the demonstrated outcomes, it could acknowledge that this is shown primarily in specific game contexts.

<score>1</score>

Justification:
1. The paper explicitly discusses how opponent profiles being mixtures "makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult."

2. The Mixed-Opponents algorithm directly demonstrates how learning against a combined single opponent policy helps address the variance issue.

3. Figure 4 provides empirical evidence showing improved performance (lower SumRegret) when using Mixed-Opponents compared to standard PSRO.

4. The claim doesn't overreach - it simply states that learning against a single policy helps mitigate variance, which is well-supported by both the theoretical framework and experimental results.

5. The review comment, while generally positive, doesn't specifically address this claim's validity.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Learning against a single policy mitigates variance in state outcomes that is induced by an unobserved distribution of opponents.

The review comment to be evaluated is:

The paper focuses on resolving the computational and sample efficiency challenges with current PSRO style approaches. To this end it proposes two different modifications to the standard PSRO setup: 1) Mixed Oracles, and 2) Mixed Opponents. These approaches allow avoiding resetting learning after each outer loop epoch and reduce the stochasticity of dynamics during training. Thee efficacy is demonstrated on relatively simple games but using Deep RL policies where the proposed approaches are at least on par with standard PSRO approach in terms of final performance while drastically improving the sample efficiency.

Although the paper has quite a few typos and unclear writing in places, it has a fantastic setup with clear description of the problems with standard PSRO it's trying to resolve. The paper points out the clear computational deficiencies with current PSRO style approaches: 1) starting anew everytime, 2) slower exploration of strategy space, 3) stochastic dynamics making learning difficult.

The paper is mostly focused on two-player zero sum games.
Approach heavily dependent on efficacy of Q-mixing approach (although the idea is more general), so currently limited to discrete action problems.
Moreover, since there are no alternatives to Q-mixing, we don't actually get any understanding/intuition for why this works and how important is the performance of "TransferOracle" or "OpponentOracle".

Although section 4.4 suggests that the two stage hyperparameter selection wasn't as important, results in Fig 6b are too noisy to fully accept. It's unclear how much compute/samples were required for the hyper parameter selection. In general the results are a lot more noisier for the proposed approaches vs standard PSRO.

**Minor**
- Environment dynamics should be SxA rather than OxA
- Algorithm 4.1 does not exist?
- Fig 1. Is it because of PSRO with Nash. Would a different MSS work differently?
- "proposes MCC as a solution concept": not everyone would know what MCC refers to.

The evidence to be evaluated is:A _normal form games_ (NFG) \(\Gamma\!=\!(\Pi,\!U,\!n)\) describes a one-shot strategic interaction among \(n\) players. When there is more than one agent, we denote agent-specific components with subscripts (e.g., \(\pi_{i}\)). Negated subscripts represent the joint elements of all other agents (e.g., \(\pi_{-i}\)). Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

The sum of regrets across all players \(\text{SumRegret}(\sigma,\overline{\Pi})=\sum_{i\in n}\text{Regret}_{i}(\sigma, \overline{\Pi}_{i})\), sometimes called the Nash convergence, is a measure of how stable a solution is. At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs.

Secondly, the opponent profiles are mixtures (i.e., distribution over opponent policies). This makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult.

In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy. Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters. A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: _pure-hparams_ performed best on-average against each individual test-opponent, and _mix-hparams_ performed best against a uniform mixture of the test-opponents. These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.

We used two sets of hyperparameters: specialized for low- and high-variance outcomes of state, which are a result of facing pure- and mixed-strategy opponents respectively. This was motivated by the assumption that lower variance would require less training.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "Learning against a single policy mitigates variance in state outcomes that is induced by an unobserved distribution of opponents."

Summary of evaluation:
The evidence partially supports this claim but suggests it may be slightly overstated. While both Mixed-Oracles and Mixed-Opponents approaches aim to reduce variance by learning against single policies rather than mixtures, the evidence shows mixed results. The graphs in Figure 4 show high variance in both approaches, though Mixed-Opponents appears to achieve lower regret. The paper acknowledges the need for different hyperparameters for pure vs. mixed strategies, indicating the complexity of variance reduction.

<score>3</score>

Justification:
1. The evidence does support that learning against single policies can help address variance issues:
- The text states "Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning."
- The paper explains how opponent mixtures make environment dynamics more stochastic and learning more difficult.

2. However, several factors suggest the claim is moderately overstated:
- Figure 4 shows considerable variance remains in both approaches (wide shaded regions)
- The need for different hyperparameters for pure vs. mixed strategies suggests variance isn't fully mitigated
- The reviewer notes "results are a lot more noisier for the proposed approaches vs standard PSRO"
- The claim doesn't acknowledge limitations or degrees of effectiveness

The claim would be better stated with qualifiers about the degree of variance mitigation and acknowledgment of remaining challenges.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Learning against a single policy mitigates variance in state outcomes that is induced by an unobserved distribution of opponents.

The review comment to be evaluated is:

Summary
The paper proposes two new methods in the Policy-Space Response Oracle framework. These approaches permit to reuse past knowledge in order to reduce the amount of data required for the RL training. The first algorithm Mixed-Oracles transfers the previous iteration of Deep RL, instead of the second one, Mixed-Opponents, transfers existing strategy action-value estimates.

Strengths
The paper proposes two convincing alternatives to reuse previous knowledge in the PSRO framework. The two ideas are based on Q-mixing approach: the first one uses this method to transfer Q-values across epochs, the second one to design a new training objective.
The experiments show that the proposed methods find a good solution using less simulation than the original PSRO framework.

Weakness
The paper is not very novel, since it uses previous approaches (PSRO and Q-mixing) to transfer knowledge for the PSRO framework.
I am not aware of recent works in this framework but could be useful to compare the proposed approaches with P2RO [1].
The Mixed-Opponent section needs a better explanation of the use of Q-mixing as a training objective.

[1]  McAleer, S., Lanier, J., Fox, R., & Baldi, P. (2020). Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. arXiv preprint arXiv:2006.08555.

The evidence to be evaluated is:A _normal form games_ (NFG) \(\Gamma\!=\!(\Pi,\!U,\!n)\) describes a one-shot strategic interaction among \(n\) players. When there is more than one agent, we denote agent-specific components with subscripts (e.g., \(\pi_{i}\)). Negated subscripts represent the joint elements of all other agents (e.g., \(\pi_{-i}\)). Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

The sum of regrets across all players \(\text{SumRegret}(\sigma,\overline{\Pi})=\sum_{i\in n}\text{Regret}_{i}(\sigma, \overline{\Pi}_{i})\), sometimes called the Nash convergence, is a measure of how stable a solution is. At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs.

Secondly, the opponent profiles are mixtures (i.e., distribution over opponent policies). This makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult.

In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy. Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters. A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: _pure-hparams_ performed best on-average against each individual test-opponent, and _mix-hparams_ performed best against a uniform mixture of the test-opponents. These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.

We used two sets of hyperparameters: specialized for low- and high-variance outcomes of state, which are a result of facing pure- and mixed-strategy opponents respectively. This was motivated by the assumption that lower variance would require less training.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "Learning against a single policy mitigates variance in state outcomes induced by an unobserved distribution of opponents" appears to be well-supported. The evidence shows that both Mixed-Oracles and Mixed-Opponents algorithms address the variance issue by converting mixed-strategy opponents into single policies for training. The experimental results in Figure 4 demonstrate that Mixed-Opponents achieves lower SumRegret compared to standard PSRO, suggesting improved learning stability. The paper explicitly discusses how these methods "mitigate the variance in state outcomes, by removing an unobserved distribution of opponents."

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Learning against a single policy mitigates variance in state outcomes that is induced by an unobserved distribution of opponents.

The review comment to be evaluated is:

##########################################################################
Summary:

The paper provides an interesting approach to speeding up the convergence time of the Policy-Space Response Oracles framework by re-using the Q-functions of past best-responses to transfer knowledge across epochs.
##########################################################################
Reasons for score:

Overall, I am low confidence on my assessment of this paper due to the exposition in the algorithm section being relatively confusing. The experimental results are interesting which suggests that the method has value but there is key missing information on how the best response policies are constructed that make it difficult to assess the paper and lead to my not wanting to recommend its acceptance. I would highly recommend being more detailed in Sec. 3 to allow me to reassess the paper. I would certainly be willing to update my score if the paper was clearer to read.
 ##########################################################################Pros:

1. The experimental results on Leduc Poker are very speedy in terms of time-steps.

2. The idea of reusing the prior Q functions and just mixing them together rather than re-learning all of the policies is very good.

##########################################################################
Cons:

1. The algorithm boxes are so high level that I am struggling to understand how the algorithms work. I would not be able to implement it from reading the paper.

#########################################################################
Things that would improve readability:

- It would be nice in the algorithm boxes to connect Q-mixing to how the best policy is explicitly output. I was not able to understand how Q-mixing was connected to either Algorithm 2 or 3 and subsequently had difficulty following the paper.
- \lambda does not appear to be defined anywhere but appears in the Mixed Oracles algorithm box
- What is the OpponentOracle and the TransferOracle? They are defined in the algorithm boxes but are not clearly defined elsewhere.
- The specific example of RPS in section 3.2 does not provide useful intuition by going through the numerics, it may be more helpful to walk through a more high level description.
- It would probably be useful to move more of the experimental results to an appendix to leave room for the exposition of the algorithms.

The evidence to be evaluated is:A _normal form games_ (NFG) \(\Gamma\!=\!(\Pi,\!U,\!n)\) describes a one-shot strategic interaction among \(n\) players. When there is more than one agent, we denote agent-specific components with subscripts (e.g., \(\pi_{i}\)). Negated subscripts represent the joint elements of all other agents (e.g., \(\pi_{-i}\)). Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

The sum of regrets across all players \(\text{SumRegret}(\sigma,\overline{\Pi})=\sum_{i\in n}\text{Regret}_{i}(\sigma, \overline{\Pi}_{i})\), sometimes called the Nash convergence, is a measure of how stable a solution is. At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs.

Secondly, the opponent profiles are mixtures (i.e., distribution over opponent policies). This makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult.

In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy. Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters. A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: _pure-hparams_ performed best on-average against each individual test-opponent, and _mix-hparams_ performed best against a uniform mixture of the test-opponents. These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.

We used two sets of hyperparameters: specialized for low- and high-variance outcomes of state, which are a result of facing pure- and mixed-strategy opponents respectively. This was motivated by the assumption that lower variance would require less training.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "Learning against a single policy mitigates variance in state outcomes that is induced by an unobserved distribution of opponents" appears to be well-supported by multiple pieces of evidence. The text explicitly states this concept in multiple ways and provides experimental validation. The evidence shows that Mixed-Opponents method constructs a single opponent policy from a mixture, which reduces variance compared to learning against multiple opponents. Figure 4 demonstrates improved performance of Mixed-Opponents over standard PSRO, supporting this claim. The authors also carefully acknowledge the relationship between variance and training through their hyperparameter selection process.

<score>1</score>

Justification:
1. Direct evidence states "Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning."
2. The Mixed-Opponents algorithm specifically addresses this by "constructing a single opponent policy that represents an aggregate of the mixed strategy"
3. Figure 4 provides empirical validation showing improved performance
4. The authors demonstrate understanding of variance effects through their hyperparameter selection process for "low- and high-variance situations"
5. The claim is carefully worded to match what the evidence demonstrates without overextending beyond the supported conclusions


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We empirically demonstrate that these algorithms substantially reduce the amount of simulation during training required by PSRO, while producing equivalent or better solutions to the game.

The evidence to be evaluated is:In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent. In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps? We begin by running both algorithms on the Gathering-Small and Leduc-Poker games. We record regrets over PSRO epochs and cumulative training timesteps. Next, we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?

We run both algorithms on the Gathering-Open game with three players.

Figure 5(a) shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the time.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget.

The trends observed in the previous experiment reoccur: Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

In this study, we investigate PSRO's approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 5: PSRO compared against Mixed-Opponents on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 2: PSRO compared against Mixed-Oracles on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 3: PSRO compared against Mixed-Oracles on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be fairly well-stated, though slightly overstated. While the graphs (Figures 2-5) demonstrate that both Mixed-Oracles and Mixed-Opponents achieve similar or better regret scores compared to PSRO with fewer training timesteps across different games (Gathering-Small, Leduc-Poker), the word "substantially" may be too strong. The evidence shows improvements in efficiency, but the magnitude varies across experiments. For instance, Mixed-Opponents finds similar quality solutions in "half the time" in some cases, but the improvement isn't consistently "substantial" across all scenarios.

<score>2</score>

Here's my detailed justification:

1. The evidence supports improved efficiency:
- Figures demonstrate both algorithms achieve similar or better regret scores with fewer timesteps
- Explicit mention that "Mixed-Opponents finds a similar quality solution to PSRO in half of the time"
- Results consistently show reduced training time across multiple games

2. Areas where the claim could be more precise:
- "Substantially" implies a dramatic improvement, but the magnitude varies across experiments
- The efficiency gains, while clear, aren't uniformly "substantial" across all test cases
- Some results show modest rather than substantial improvements

3. Supporting quotes:
- "Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps"
- "Mixed-Opponents find a better solution than PSRO in half the simulation timesteps"

4. The claim would be better stated if it used more moderate language like "consistently" or "notably" rather than "substantially."

The score of 2 reflects that while the claim is generally supported by evidence, the use of "substantially" slightly overstates the consistent but varying degree of improvement shown in the results.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We empirically demonstrate that these algorithms substantially reduce the amount of simulation during training required by PSRO, while producing equivalent or better solutions to the game.

The review comment to be evaluated is:

The paper suggests two techniques to improve the calculation of empirically figuring out a Nash equilibrium using an iterative application of best-response dynamics. One method learns the best-response to the previously used strategy. The other uses that technique to model the opponent, and then best-responds to the modeled opponent. The experiments show a faster reaching to NE than without these changes.

The paper is well-written and explained, and is accessible even to researchers not well-versed in ML topics. While, the suggested changes are rather straightforward, they do indeed lead to the expected advance (shorter times to reach NE). I was convinced by further introspection that this is a significant enough contribution, to merit acceptance.

Of course, a more significant change to the algorithm, leading not only to a shorter time but to convergence to better equilibria (in cases where multiple exist) would be far more compelling.

The evidence to be evaluated is:In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent. In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps? We begin by running both algorithms on the Gathering-Small and Leduc-Poker games. We record regrets over PSRO epochs and cumulative training timesteps. Next, we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?

We run both algorithms on the Gathering-Open game with three players.

Figure 5(a) shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the time.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget.

The trends observed in the previous experiment reoccur: Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

In this study, we investigate PSRO's approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 5: PSRO compared against Mixed-Opponents on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 2: PSRO compared against Mixed-Oracles on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 3: PSRO compared against Mixed-Oracles on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears well-supported but slightly overstated. The graphs show that both Mixed-Oracles and Mixed-Opponents algorithms generally achieve similar or better regret values compared to PSRO while using fewer timesteps. However, the claim states they "substantially reduce" simulation time, which is somewhat strong language given that the improvements shown vary across experiments - in some cases showing roughly half the timesteps needed (e.g., Figure 5) while in others showing more modest gains. The solutions are indeed equivalent or better in terms of final regret values.

<score>2</score>

Justification:
1. The evidence demonstrates reduced simulation time:
- Figure 5(a) shows "Mixed-Opponents finds a similar quality solution to PSRO in half the time"
- Under pure-hparams, "Mixed-Opponents find a better solution than PSRO in half the simulation timesteps"
- Figures 2-5 consistently show faster convergence with fewer timesteps

2. The slightly higher score (2 instead of 1) is given because:
- The term "substantially reduce" is somewhat strong given the variable improvements shown across different experiments
- The reviewer comment notes the changes are "rather straightforward" and lead to "shorter times" but doesn't characterize the improvement as substantial
- The evidence shows improvements but ranges from modest to 50% reduction depending on the scenario

The claim is generally accurate but could be more precisely worded to reflect the variable nature of the improvements observed.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We empirically demonstrate that these algorithms substantially reduce the amount of simulation during training required by PSRO, while producing equivalent or better solutions to the game.

The review comment to be evaluated is:

The paper focuses on resolving the computational and sample efficiency challenges with current PSRO style approaches. To this end it proposes two different modifications to the standard PSRO setup: 1) Mixed Oracles, and 2) Mixed Opponents. These approaches allow avoiding resetting learning after each outer loop epoch and reduce the stochasticity of dynamics during training. Thee efficacy is demonstrated on relatively simple games but using Deep RL policies where the proposed approaches are at least on par with standard PSRO approach in terms of final performance while drastically improving the sample efficiency.

Although the paper has quite a few typos and unclear writing in places, it has a fantastic setup with clear description of the problems with standard PSRO it's trying to resolve. The paper points out the clear computational deficiencies with current PSRO style approaches: 1) starting anew everytime, 2) slower exploration of strategy space, 3) stochastic dynamics making learning difficult.

The paper is mostly focused on two-player zero sum games.
Approach heavily dependent on efficacy of Q-mixing approach (although the idea is more general), so currently limited to discrete action problems.
Moreover, since there are no alternatives to Q-mixing, we don't actually get any understanding/intuition for why this works and how important is the performance of "TransferOracle" or "OpponentOracle".

Although section 4.4 suggests that the two stage hyperparameter selection wasn't as important, results in Fig 6b are too noisy to fully accept. It's unclear how much compute/samples were required for the hyper parameter selection. In general the results are a lot more noisier for the proposed approaches vs standard PSRO.

**Minor**
- Environment dynamics should be SxA rather than OxA
- Algorithm 4.1 does not exist?
- Fig 1. Is it because of PSRO with Nash. Would a different MSS work differently?
- "proposes MCC as a solution concept": not everyone would know what MCC refers to.

The evidence to be evaluated is:In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent. In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps? We begin by running both algorithms on the Gathering-Small and Leduc-Poker games. We record regrets over PSRO epochs and cumulative training timesteps. Next, we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?

We run both algorithms on the Gathering-Open game with three players.

Figure 5(a) shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the time.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget.

The trends observed in the previous experiment reoccur: Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

In this study, we investigate PSRO's approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 5: PSRO compared against Mixed-Opponents on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 2: PSRO compared against Mixed-Oracles on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 3: PSRO compared against Mixed-Oracles on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I'll evaluate the claim "We empirically demonstrate that these algorithms substantially reduce the amount of simulation during training required by PSRO, while producing equivalent or better solutions to the game."

Summary: The figures show that both Mixed-Oracles and Mixed-Opponents algorithms achieve similar or better regret values compared to PSRO while using fewer timesteps across multiple games (Gathering-Small, Leduc-Poker). Figures 2-5 demonstrate this reduction in required simulation timesteps while maintaining performance quality. However, as noted in the review comments, the results show higher noise levels for the proposed approaches, and there are limitations regarding the dependency on Q-mixing and applicability to discrete action problems only.

The claim appears slightly overstated given these limitations and noise levels, but the core assertion about reduced simulation requirements while maintaining solution quality is supported by the evidence.

<score>2</score>

Justification:
1. The evidence clearly shows reduced simulation requirements through graphs comparing timesteps
2. Performance is generally equivalent or better, supported by multiple game environments
3. However, some caveats exist:
   - Higher result variability (noted in review comments)
   - Limited to discrete action problems
   - Dependency on Q-mixing effectiveness
4. The word "substantially" is somewhat strong but generally supported by the ~50% reduction in required timesteps shown in the figures
5. The limitations don't invalidate the core claim but suggest slightly more measured language would be more appropriate


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We empirically demonstrate that these algorithms substantially reduce the amount of simulation during training required by PSRO, while producing equivalent or better solutions to the game.

The review comment to be evaluated is:

Summary
The paper proposes two new methods in the Policy-Space Response Oracle framework. These approaches permit to reuse past knowledge in order to reduce the amount of data required for the RL training. The first algorithm Mixed-Oracles transfers the previous iteration of Deep RL, instead of the second one, Mixed-Opponents, transfers existing strategy action-value estimates.

Strengths
The paper proposes two convincing alternatives to reuse previous knowledge in the PSRO framework. The two ideas are based on Q-mixing approach: the first one uses this method to transfer Q-values across epochs, the second one to design a new training objective.
The experiments show that the proposed methods find a good solution using less simulation than the original PSRO framework.

Weakness
The paper is not very novel, since it uses previous approaches (PSRO and Q-mixing) to transfer knowledge for the PSRO framework.
I am not aware of recent works in this framework but could be useful to compare the proposed approaches with P2RO [1].
The Mixed-Opponent section needs a better explanation of the use of Q-mixing as a training objective.

[1]  McAleer, S., Lanier, J., Fox, R., & Baldi, P. (2020). Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. arXiv preprint arXiv:2006.08555.

The evidence to be evaluated is:In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent. In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps? We begin by running both algorithms on the Gathering-Small and Leduc-Poker games. We record regrets over PSRO epochs and cumulative training timesteps. Next, we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?

We run both algorithms on the Gathering-Open game with three players.

Figure 5(a) shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the time.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget.

The trends observed in the previous experiment reoccur: Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

In this study, we investigate PSRO's approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 5: PSRO compared against Mixed-Opponents on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 2: PSRO compared against Mixed-Oracles on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 3: PSRO compared against Mixed-Oracles on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence carefully, I would evaluate the claim as generally well-stated, but with minor overstatement. The figures clearly show that both Mixed-Oracles and Mixed-Opponents achieve similar or better regret scores while using fewer timesteps compared to PSRO across multiple games (Gathering-Small, Leduc-Poker). However, the word "substantially" might be slightly strong since the reduction varies across experiments - in some cases it's about half the timesteps, which is significant but perhaps not "substantial." The results support the core claim about equivalent/better solutions with reduced simulation, but the magnitude description could be more precise.

<score>2</score>

Justification:
1. Evidence directly supports reduced simulation time:
- Figures 2-5 demonstrate fewer timesteps needed for both algorithms
- "Mixed-Opponents finds a similar quality solution to PSRO in half of the time"
- "Mixed-Opponents find a better solution than PSRO in half the simulation timesteps"

2. Evidence supports equivalent/better solutions:
- Regret plots show similar or improved performance
- Multiple games tested (Gathering-Small, Leduc-Poker, Gathering-Open)
- Consistent results under different hyperparameter settings

3. Minor overstatement concerns:
- "Substantially" implies a dramatic reduction, when results show ~50% improvement
- Reduction varies across experiments
- Reviewer notes the approaches are "not very novel" as they build on existing methods

The claim would be more precise if it quantified the reduction (e.g., "reduce by approximately half") rather than using the broader term "substantially."


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We empirically demonstrate that these algorithms substantially reduce the amount of simulation during training required by PSRO, while producing equivalent or better solutions to the game.

The review comment to be evaluated is:

##########################################################################
Summary:

The paper provides an interesting approach to speeding up the convergence time of the Policy-Space Response Oracles framework by re-using the Q-functions of past best-responses to transfer knowledge across epochs.
##########################################################################
Reasons for score:

Overall, I am low confidence on my assessment of this paper due to the exposition in the algorithm section being relatively confusing. The experimental results are interesting which suggests that the method has value but there is key missing information on how the best response policies are constructed that make it difficult to assess the paper and lead to my not wanting to recommend its acceptance. I would highly recommend being more detailed in Sec. 3 to allow me to reassess the paper. I would certainly be willing to update my score if the paper was clearer to read.
 ##########################################################################Pros:

1. The experimental results on Leduc Poker are very speedy in terms of time-steps.

2. The idea of reusing the prior Q functions and just mixing them together rather than re-learning all of the policies is very good.

##########################################################################
Cons:

1. The algorithm boxes are so high level that I am struggling to understand how the algorithms work. I would not be able to implement it from reading the paper.

#########################################################################
Things that would improve readability:

- It would be nice in the algorithm boxes to connect Q-mixing to how the best policy is explicitly output. I was not able to understand how Q-mixing was connected to either Algorithm 2 or 3 and subsequently had difficulty following the paper.
- \lambda does not appear to be defined anywhere but appears in the Mixed Oracles algorithm box
- What is the OpponentOracle and the TransferOracle? They are defined in the algorithm boxes but are not clearly defined elsewhere.
- The specific example of RPS in section 3.2 does not provide useful intuition by going through the numerics, it may be more helpful to walk through a more high level description.
- It would probably be useful to move more of the experimental results to an appendix to leave room for the exposition of the algorithms.

The evidence to be evaluated is:In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent. In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps? We begin by running both algorithms on the Gathering-Small and Leduc-Poker games. We record regrets over PSRO epochs and cumulative training timesteps. Next, we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?

We run both algorithms on the Gathering-Open game with three players.

Figure 5(a) shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the time.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget.

The trends observed in the previous experiment reoccur: Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

In this study, we investigate PSRO's approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 5: PSRO compared against Mixed-Opponents on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 2: PSRO compared against Mixed-Oracles on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 3: PSRO compared against Mixed-Oracles on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "We empirically demonstrate that these algorithms substantially reduce the amount of simulation during training required by PSRO, while producing equivalent or better solutions to the game."

Summary of evaluation:
The evidence largely supports this claim through multiple experimental results shown in Figures 2-5. The graphs consistently demonstrate that both Mixed-Oracles and Mixed-Opponents achieve similar or better regret values compared to PSRO while using fewer timesteps. Specifically, Mixed-Opponents finds solutions "at least as good as PSRO in fewer timesteps" and "in half the simulation timesteps" under certain conditions. The results are shown across different games (Gathering-Small, Leduc-Poker, Gathering-Open), providing robust support.

While there are some limitations noted in the review comments about algorithm clarity, these don't directly impact the empirical performance claims, which are well-supported by the graphical evidence.

<score>1</score>

Justification:
1. The graphs clearly show reduced training timesteps while maintaining or improving performance
2. Results are demonstrated across multiple games and scenarios
3. The claims are carefully qualified ("equivalent or better" rather than claiming universal superiority)
4. Empirical evidence directly supports both aspects of the claim (reduced simulation time and solution quality)
5. The reviewer's concerns focus on exposition rather than results validity