Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Both algorithms modify how PSRO adds new policies to the empirical game, based on learned responses to a single opponent policy.

The review comment to be evaluated is:

Summary
The paper proposes two new methods in the Policy-Space Response Oracle framework. These approaches permit to reuse past knowledge in order to reduce the amount of data required for the RL training. The first algorithm Mixed-Oracles transfers the previous iteration of Deep RL, instead of the second one, Mixed-Opponents, transfers existing strategy action-value estimates.

Strengths
The paper proposes two convincing alternatives to reuse previous knowledge in the PSRO framework. The two ideas are based on Q-mixing approach: the first one uses this method to transfer Q-values across epochs, the second one to design a new training objective.
The experiments show that the proposed methods find a good solution using less simulation than the original PSRO framework.

Weakness
The paper is not very novel, since it uses previous approaches (PSRO and Q-mixing) to transfer knowledge for the PSRO framework.
I am not aware of recent works in this framework but could be useful to compare the proposed approaches with P2RO [1].
The Mixed-Opponent section needs a better explanation of the use of Q-mixing as a training objective.

[1]  McAleer, S., Lanier, J., Fox, R., & Baldi, P. (2020). Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. arXiv preprint arXiv:2006.08555.

The evidence to be evaluated is:At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In PSRO, the strategy exploration problem is decomposed into two steps: solution and BR via RL.

In the BR step, PSRO generates a new policy via RL (the _response oracle_), training against the target opponent profile computed by the MSS. The choice of MSS and response oracle algorithm constitutes a strategy exploration approach and determines the convergence speed of PSRO.

In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent. In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies. The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

If, as in PSRO, player 1 then adds their BR to this meta-strategy, which is P, then their strategy set still does not contain the Nash equilibrium of the game. The problem is that there are already strategies in the empirical game that can defeat player 2's strategies, so the new strategy is not useful.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Input: Initial policy sets for all players \(\Pi^{0}\)  Simulate utilities \(\tilde{U}^{\Pi^{0}}\) for each joint \(\pi\!\in\!\Pi^{0}\)  Initialize solutions \(\sigma_{i}^{*,0}\!=\!\text{Uniform}(\Pi_{i}^{0})\) whileepoch e in \(\{1,2,...\}\)do for player \(i\!\in\![[n]]\)do for many episodesdo \(\pi_{-i}\!\sim\!\sigma_{-i}^{*-1}\)  Train \(\pi_{i}^{e}\) over \(\tau\!\sim\!(\pi_{i}^{e}\!,\!\pi_{-i})\) \(\Pi_{i}^{e}\!=\!\Pi_{i}^{-1}\!\cup\!\{\pi_{i}^{e}\}\)  Simulate missing entries in \(\tilde{U}^{\Pi^{e}}\) from \(\Pi^{e}\)  Compute a solution \(\sigma^{*,e}\) from \(\tilde{\Gamma}^{e}\) Output: Current solution \(\sigma_{i}^{*,e}\) for player \(i\)

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Their approach is still seen in PSRO today, where iterative BRs to equilibrium are used to build an empirical game.

PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

Wright et al. (2019) extends DO's MSS to return a decay-weighted linear combination of the current and previous solutions. Omidshafiei et al. (2019) proposes Markov-Conley Chains (MCC) as a solution concept that relates to their new proposed quality measure of policy called \(\alpha\)-Rank, and Muller et al. (2020) investigated its use as a MSS.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim: "Both algorithms modify how PSRO adds new policies to the empirical game, based on learned responses to a single opponent policy."

Summary of evaluation:
The evidence shows that both Mixed-Oracles and Mixed-Opponents do modify PSRO's policy addition process, but in different ways:

1. Mixed-Oracles learns BR against a single new opponent policy but combines knowledge from previous epochs using Q-Mixing.
2. Mixed-Opponents combines opponent strategies into a single policy target for training.

While both methods involve "single opponent" elements, the claim somewhat oversimplifies their mechanisms. Mixed-Oracles still uses knowledge from multiple opponents through Q-Mixing, while Mixed-Opponents creates a composite opponent. The claim is slightly overstated in suggesting both methods are purely single-opponent based.

<score>2</score>

Justification:
- The evidence confirms both algorithms modify PSRO's policy addition process
- Mixed-Oracles: "During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy"
- Mixed-Opponents: "combines the strategies of the opponent mixture themselves to construct a single new opponent policy"
- However, both still incorporate multiple opponent information:
  - Mixed-Oracles: "reusing learned behaviors from previous epochs"
  - Mixed-Opponents: "combines the strategies of the opponent mixture"
- The reviewer's comments don't contradict the fundamental mechanisms described, supporting that both approaches modify PSRO's policy addition process

The claim is mostly accurate but slightly oversimplifies the mechanisms, warranting a score of 2.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Both algorithms modify how PSRO adds new policies to the empirical game, based on learned responses to a single opponent policy.

The review comment to be evaluated is:

##########################################################################
Summary:

The paper provides an interesting approach to speeding up the convergence time of the Policy-Space Response Oracles framework by re-using the Q-functions of past best-responses to transfer knowledge across epochs.
##########################################################################
Reasons for score:

Overall, I am low confidence on my assessment of this paper due to the exposition in the algorithm section being relatively confusing. The experimental results are interesting which suggests that the method has value but there is key missing information on how the best response policies are constructed that make it difficult to assess the paper and lead to my not wanting to recommend its acceptance. I would highly recommend being more detailed in Sec. 3 to allow me to reassess the paper. I would certainly be willing to update my score if the paper was clearer to read.
 ##########################################################################Pros:

1. The experimental results on Leduc Poker are very speedy in terms of time-steps.

2. The idea of reusing the prior Q functions and just mixing them together rather than re-learning all of the policies is very good.

##########################################################################
Cons:

1. The algorithm boxes are so high level that I am struggling to understand how the algorithms work. I would not be able to implement it from reading the paper.

#########################################################################
Things that would improve readability:

- It would be nice in the algorithm boxes to connect Q-mixing to how the best policy is explicitly output. I was not able to understand how Q-mixing was connected to either Algorithm 2 or 3 and subsequently had difficulty following the paper.
- \lambda does not appear to be defined anywhere but appears in the Mixed Oracles algorithm box
- What is the OpponentOracle and the TransferOracle? They are defined in the algorithm boxes but are not clearly defined elsewhere.
- The specific example of RPS in section 3.2 does not provide useful intuition by going through the numerics, it may be more helpful to walk through a more high level description.
- It would probably be useful to move more of the experimental results to an appendix to leave room for the exposition of the algorithms.

The evidence to be evaluated is:At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In PSRO, the strategy exploration problem is decomposed into two steps: solution and BR via RL.

In the BR step, PSRO generates a new policy via RL (the _response oracle_), training against the target opponent profile computed by the MSS. The choice of MSS and response oracle algorithm constitutes a strategy exploration approach and determines the convergence speed of PSRO.

In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent. In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies. The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

If, as in PSRO, player 1 then adds their BR to this meta-strategy, which is P, then their strategy set still does not contain the Nash equilibrium of the game. The problem is that there are already strategies in the empirical game that can defeat player 2's strategies, so the new strategy is not useful.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Input: Initial policy sets for all players \(\Pi^{0}\)  Simulate utilities \(\tilde{U}^{\Pi^{0}}\) for each joint \(\pi\!\in\!\Pi^{0}\)  Initialize solutions \(\sigma_{i}^{*,0}\!=\!\text{Uniform}(\Pi_{i}^{0})\) whileepoch e in \(\{1,2,...\}\)do for player \(i\!\in\![[n]]\)do for many episodesdo \(\pi_{-i}\!\sim\!\sigma_{-i}^{*-1}\)  Train \(\pi_{i}^{e}\) over \(\tau\!\sim\!(\pi_{i}^{e}\!,\!\pi_{-i})\) \(\Pi_{i}^{e}\!=\!\Pi_{i}^{-1}\!\cup\!\{\pi_{i}^{e}\}\)  Simulate missing entries in \(\tilde{U}^{\Pi^{e}}\) from \(\Pi^{e}\)  Compute a solution \(\sigma^{*,e}\) from \(\tilde{\Gamma}^{e}\) Output: Current solution \(\sigma_{i}^{*,e}\) for player \(i\)

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Their approach is still seen in PSRO today, where iterative BRs to equilibrium are used to build an empirical game.

PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

Wright et al. (2019) extends DO's MSS to return a decay-weighted linear combination of the current and previous solutions. Omidshafiei et al. (2019) proposes Markov-Conley Chains (MCC) as a solution concept that relates to their new proposed quality measure of policy called \(\alpha\)-Rank, and Muller et al. (2020) investigated its use as a MSS.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim "Both algorithms modify how PSRO adds new policies to the empirical game, based on learned responses to a single opponent policy."

Based on the evidence:

1. Mixed-Oracles: Trains against single new opponent policies but combines knowledge from previous best responses using Q-Mixing. It doesn't strictly learn responses to just a single opponent policy.

2. Mixed-Opponents: Creates a combined opponent policy from the mixture and trains against this single aggregated opponent.

The claim is somewhat overstated because while both methods aim to reduce training against multiple opponents, Mixed-Oracles actually uses multiple opponent policies through Q-Mixing, not just a single opponent policy as stated. Mixed-Opponents does use a single (but aggregated) opponent policy.

The review comment raises concerns about algorithm clarity but doesn't directly contradict or support this specific claim.

<score>3</score>

Justification:
- Evidence shows Mixed-Oracles "reuses learned behaviors from previous epochs" and uses Q-Mixing to combine "Q-values learned against each opponent policy"
- Mixed-Opponents does "combine the strategies of the opponent mixture themselves to construct a single new opponent policy"
- The claim oversimplifies Mixed-Oracles' approach by suggesting it only uses single opponent responses
- The statement accurately describes Mixed-Opponents but is imprecise for Mixed-Oracles


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The first, Mixed-Oracles, transfers knowledge from previous iterations of Deep RL, requiring training only against the opponent's newest policy.

The evidence to be evaluated is:First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs. This suggests a compelling opportunity to transfer learning across epochs; however, the BR calculation step works by training anew. Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG.

In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Wright et al. (2019) extends DO's MSS to return a decay-weighted linear combination of the current and previous solutions.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution. This affords our algorithm the ability to transfer learning about previously encountered opponents.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "The first, Mixed-Oracles, transfers knowledge from previous iterations of Deep RL, requiring training only against the opponent's newest policy."

Analysis:
The evidence strongly supports this claim. Multiple sections describe Mixed-Oracles' ability to transfer knowledge between epochs and focus training on new opponent policies. The mechanism is clearly explained through Q-Mixing, which allows combining Q-values learned from previous opponents. The evidence directly states "By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies." The claim accurately represents the core functionality without exaggerating its capabilities or effectiveness.

<score>1</score>

Justification:
1. The claim precisely matches the implementation details described in the evidence
2. The evidence explicitly confirms both key aspects:
   - Knowledge transfer: "Mixed-Oracles which uses Q-Mixing to transfer Q-values across epochs"
   - Training against new policies only: "During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy"
3. The mechanism for knowledge transfer (Q-Mixing) is well-explained with mathematical formulation
4. Results support the claimed functionality, showing Mixed-Oracles achieves solutions "at least as good as PSRO in fewer timesteps"

The claim is well-stated as it accurately represents the core functionality without overextending beyond what the evidence demonstrates.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The first, Mixed-Oracles, transfers knowledge from previous iterations of Deep RL, requiring training only against the opponent's newest policy.

The review comment to be evaluated is:

The paper suggests two techniques to improve the calculation of empirically figuring out a Nash equilibrium using an iterative application of best-response dynamics. One method learns the best-response to the previously used strategy. The other uses that technique to model the opponent, and then best-responds to the modeled opponent. The experiments show a faster reaching to NE than without these changes.

The paper is well-written and explained, and is accessible even to researchers not well-versed in ML topics. While, the suggested changes are rather straightforward, they do indeed lead to the expected advance (shorter times to reach NE). I was convinced by further introspection that this is a significant enough contribution, to merit acceptance.

Of course, a more significant change to the algorithm, leading not only to a shorter time but to convergence to better equilibria (in cases where multiple exist) would be far more compelling.

The evidence to be evaluated is:First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs. This suggests a compelling opportunity to transfer learning across epochs; however, the BR calculation step works by training anew. Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG.

In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Wright et al. (2019) extends DO's MSS to return a decay-weighted linear combination of the current and previous solutions.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution. This affords our algorithm the ability to transfer learning about previously encountered opponents.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim based on the provided evidence:

The claim states that Mixed-Oracles transfers knowledge from previous iterations and only requires training against the newest opponent policy. The evidence strongly supports this claim through:

1. Clear explanation of how Mixed-Oracles reuses learned behaviors from previous epochs by using Q-Mixing to transfer Q-values
2. Explicit statement that it "focuses training exclusively on new opponent policies"
3. Empirical validation showing Mixed-Oracles finds solutions at least as good as PSRO in fewer timesteps
4. Detailed mathematical formulation of how Q-Mixing enables knowledge transfer

The claim accurately reflects the technical approach and experimental results without exaggeration. The review comment further validates this by acknowledging the clear explanation and demonstrated improvements.

<score>1</score>

I assign a score of 1 (Well-stated) because the claim is precise, technically accurate, and fully supported by both the theoretical framework and experimental results presented in the evidence. There is no overstatement of capabilities or results.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The first, Mixed-Oracles, transfers knowledge from previous iterations of Deep RL, requiring training only against the opponent's newest policy.

The review comment to be evaluated is:

The paper focuses on resolving the computational and sample efficiency challenges with current PSRO style approaches. To this end it proposes two different modifications to the standard PSRO setup: 1) Mixed Oracles, and 2) Mixed Opponents. These approaches allow avoiding resetting learning after each outer loop epoch and reduce the stochasticity of dynamics during training. Thee efficacy is demonstrated on relatively simple games but using Deep RL policies where the proposed approaches are at least on par with standard PSRO approach in terms of final performance while drastically improving the sample efficiency.

Although the paper has quite a few typos and unclear writing in places, it has a fantastic setup with clear description of the problems with standard PSRO it's trying to resolve. The paper points out the clear computational deficiencies with current PSRO style approaches: 1) starting anew everytime, 2) slower exploration of strategy space, 3) stochastic dynamics making learning difficult.

The paper is mostly focused on two-player zero sum games.
Approach heavily dependent on efficacy of Q-mixing approach (although the idea is more general), so currently limited to discrete action problems.
Moreover, since there are no alternatives to Q-mixing, we don't actually get any understanding/intuition for why this works and how important is the performance of "TransferOracle" or "OpponentOracle".

Although section 4.4 suggests that the two stage hyperparameter selection wasn't as important, results in Fig 6b are too noisy to fully accept. It's unclear how much compute/samples were required for the hyper parameter selection. In general the results are a lot more noisier for the proposed approaches vs standard PSRO.

**Minor**
- Environment dynamics should be SxA rather than OxA
- Algorithm 4.1 does not exist?
- Fig 1. Is it because of PSRO with Nash. Would a different MSS work differently?
- "proposes MCC as a solution concept": not everyone would know what MCC refers to.

The evidence to be evaluated is:First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs. This suggests a compelling opportunity to transfer learning across epochs; however, the BR calculation step works by training anew. Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG.

In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Wright et al. (2019) extends DO's MSS to return a decay-weighted linear combination of the current and previous solutions.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution. This affords our algorithm the ability to transfer learning about previously encountered opponents.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "The first, Mixed-Oracles, transfers knowledge from previous iterations of Deep RL, requiring training only against the opponent's newest policy."

Evaluation summary:
The evidence generally supports the core mechanism of Mixed-Oracles, showing it uses Q-Mixing to transfer knowledge across epochs and focuses training on new opponent policies. However, there are some limitations not acknowledged in the claim:

1. The review points out the approach is heavily dependent on Q-mixing and limited to discrete action problems
2. The evidence shows this is specifically for two-player games
3. The reviewer notes uncertainty about the importance of "TransferOracle" performance
4. The claim doesn't acknowledge these limitations or specify the scope of applicability

The claim is slightly overstated by not mentioning these important constraints and limitations.

<score>2</score>

Justification:
- The core mechanism is accurately described and supported by evidence: "Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results" and "By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies."

- However, the claim should acknowledge:
1. The limitation to discrete action spaces (from review)
2. Specific application to two-player games (stated in evidence: "Mixed-Oracles algorithm is a variant of PSRO for two-player games")
3. Dependence on Q-mixing (from review and evidence showing Q-mixing as the specific transfer mechanism)

While the claim isn't severely overstated, these omissions make it slightly less precise than ideal for scientific communication.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The first, Mixed-Oracles, transfers knowledge from previous iterations of Deep RL, requiring training only against the opponent's newest policy.

The review comment to be evaluated is:

Summary
The paper proposes two new methods in the Policy-Space Response Oracle framework. These approaches permit to reuse past knowledge in order to reduce the amount of data required for the RL training. The first algorithm Mixed-Oracles transfers the previous iteration of Deep RL, instead of the second one, Mixed-Opponents, transfers existing strategy action-value estimates.

Strengths
The paper proposes two convincing alternatives to reuse previous knowledge in the PSRO framework. The two ideas are based on Q-mixing approach: the first one uses this method to transfer Q-values across epochs, the second one to design a new training objective.
The experiments show that the proposed methods find a good solution using less simulation than the original PSRO framework.

Weakness
The paper is not very novel, since it uses previous approaches (PSRO and Q-mixing) to transfer knowledge for the PSRO framework.
I am not aware of recent works in this framework but could be useful to compare the proposed approaches with P2RO [1].
The Mixed-Opponent section needs a better explanation of the use of Q-mixing as a training objective.

[1]  McAleer, S., Lanier, J., Fox, R., & Baldi, P. (2020). Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. arXiv preprint arXiv:2006.08555.

The evidence to be evaluated is:First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs. This suggests a compelling opportunity to transfer learning across epochs; however, the BR calculation step works by training anew. Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG.

In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Wright et al. (2019) extends DO's MSS to return a decay-weighted linear combination of the current and previous solutions.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution. This affords our algorithm the ability to transfer learning about previously encountered opponents.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence, I find that the claim is partially overstated. While the evidence does show that Mixed-Oracles transfers knowledge through Q-Mixing and trains against new opponent policies, the claim oversimplifies by stating it "requires training only against the opponent's newest policy." The evidence indicates it's more nuanced - it uses Q-Mixing to combine Q-values from previous epochs with new training. The reviewer also points out that this builds on existing approaches (PSRO and Q-mixing), suggesting less novelty than implied.

Key points from evidence:
- Uses Q-Mixing to transfer knowledge across epochs
- Focuses training on new opponent policies while leveraging previous learning
- Combines responses rather than training completely anew
- Built on established methods like PSRO and Q-mixing

<score>3</score>

Justification:
1. The claim oversimplifies the mechanism by suggesting it "only" requires training against the newest policy, when the evidence shows it actually combines previous knowledge through Q-Mixing
2. The evidence describes a more complex process of knowledge transfer and combination than suggested by the simple claim
3. The reviewer notes this builds on existing approaches, making the implied novelty somewhat overstated
4. However, the core concept that it leverages previous knowledge and focuses new training on recent opponents is supported by the evidence

The claim would be more accurate if it acknowledged both the transfer mechanism (Q-Mixing) and the combination of previous knowledge with new training.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The first, Mixed-Oracles, transfers knowledge from previous iterations of Deep RL, requiring training only against the opponent's newest policy.

The review comment to be evaluated is:

##########################################################################
Summary:

The paper provides an interesting approach to speeding up the convergence time of the Policy-Space Response Oracles framework by re-using the Q-functions of past best-responses to transfer knowledge across epochs.
##########################################################################
Reasons for score:

Overall, I am low confidence on my assessment of this paper due to the exposition in the algorithm section being relatively confusing. The experimental results are interesting which suggests that the method has value but there is key missing information on how the best response policies are constructed that make it difficult to assess the paper and lead to my not wanting to recommend its acceptance. I would highly recommend being more detailed in Sec. 3 to allow me to reassess the paper. I would certainly be willing to update my score if the paper was clearer to read.
 ##########################################################################Pros:

1. The experimental results on Leduc Poker are very speedy in terms of time-steps.

2. The idea of reusing the prior Q functions and just mixing them together rather than re-learning all of the policies is very good.

##########################################################################
Cons:

1. The algorithm boxes are so high level that I am struggling to understand how the algorithms work. I would not be able to implement it from reading the paper.

#########################################################################
Things that would improve readability:

- It would be nice in the algorithm boxes to connect Q-mixing to how the best policy is explicitly output. I was not able to understand how Q-mixing was connected to either Algorithm 2 or 3 and subsequently had difficulty following the paper.
- \lambda does not appear to be defined anywhere but appears in the Mixed Oracles algorithm box
- What is the OpponentOracle and the TransferOracle? They are defined in the algorithm boxes but are not clearly defined elsewhere.
- The specific example of RPS in section 3.2 does not provide useful intuition by going through the numerics, it may be more helpful to walk through a more high level description.
- It would probably be useful to move more of the experimental results to an appendix to leave room for the exposition of the algorithms.

The evidence to be evaluated is:First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs. This suggests a compelling opportunity to transfer learning across epochs; however, the BR calculation step works by training anew. Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG.

In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Wright et al. (2019) extends DO's MSS to return a decay-weighted linear combination of the current and previous solutions.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution. This affords our algorithm the ability to transfer learning about previously encountered opponents.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my analysis of the claim "Mixed-Oracles transfers knowledge from previous iterations of Deep RL, requiring training only against the opponent's newest policy" and the provided evidence:

The evidence shows that Mixed-Oracles uses Q-Mixing to transfer Q-values across epochs by combining Q-values learned against individual opponent strategies. While it does focus training on new opponent policies, the claim slightly overstates the simplicity by not acknowledging that it still requires maintaining and combining Q-values from previous iterations. The review comments highlight that the algorithm's implementation details are unclear. However, the core mechanism of knowledge transfer and focused training is supported by the evidence.

<score>2</score>

Justification:
1. The evidence confirms knowledge transfer through Q-Mixing: "Q-Mixing averages the Q-values learned against each opponent policy π_{-i}, weighted by their likelihood in the opponent mixture σ_{-i}"

2. The claim accurately states that training focuses on new opponents: "During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy"

3. However, some nuance is missing:
- The method requires maintaining Q-values from previous iterations
- The reviewer notes implementation details are unclear: "The algorithm boxes are so high level that I am struggling to understand how the algorithms work"
- The combination process using Q-Mixing is more complex than the claim suggests

While there is slight oversimplification, the core claim about knowledge transfer and focused training is well-supported by the evidence, warranting a score of 2 (slightly overstated).


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The second, Mixed-Opponents, constructs a pure-strategy opponent by mixing existing strategy's action-value estimates, instead of their policies.

The evidence to be evaluated is:In each epoch, each player learns a BR to a mixed profile of opponent policies. This mixture typically involves the newly added strategies (one per player) for this epoch, but may also include strategies from previous epochs.

A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions (Figure 1).

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training.

We employ Q-Mixing (Equation 1) as our Combine-Oponents. In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim against the provided evidence:

Summary:
The claim states that Mixed-Opponents constructs a pure-strategy opponent by mixing action-value estimates. The evidence strongly supports this through multiple references:
1. Clear explanation of Q-Mixing equation (1) showing how Q-values are averaged
2. Explicit statement that it "combines the strategies of the opponent mixture themselves to construct a single new opponent policy"
3. Direct contrast with Mixed-Oracles showing different applications of Q-Mixing
4. Evidence of implementation and effectiveness in reducing timesteps compared to PSRO

The claim is well-stated (1) as it accurately and precisely describes the key mechanism without exaggeration, and the evidence directly supports each component of the claim through mathematical formulation and implementation details.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The second, Mixed-Opponents, constructs a pure-strategy opponent by mixing existing strategy's action-value estimates, instead of their policies.

The review comment to be evaluated is:

The paper suggests two techniques to improve the calculation of empirically figuring out a Nash equilibrium using an iterative application of best-response dynamics. One method learns the best-response to the previously used strategy. The other uses that technique to model the opponent, and then best-responds to the modeled opponent. The experiments show a faster reaching to NE than without these changes.

The paper is well-written and explained, and is accessible even to researchers not well-versed in ML topics. While, the suggested changes are rather straightforward, they do indeed lead to the expected advance (shorter times to reach NE). I was convinced by further introspection that this is a significant enough contribution, to merit acceptance.

Of course, a more significant change to the algorithm, leading not only to a shorter time but to convergence to better equilibria (in cases where multiple exist) would be far more compelling.

The evidence to be evaluated is:In each epoch, each player learns a BR to a mixed profile of opponent policies. This mixture typically involves the newly added strategies (one per player) for this epoch, but may also include strategies from previous epochs.

A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions (Figure 1).

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training.

We employ Q-Mixing (Equation 1) as our Combine-Oponents. In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find that the claim accurately describes the Mixed-Opponents technique. The evidence clearly shows that Mixed-Opponents constructs a pure-strategy opponent by combining Q-values (action-value estimates) rather than policies, as stated in multiple sections. This is particularly evident in the description of how Q-Mixing is employed and the contrast drawn with Mixed-Oracles. The claim is precise, technically accurate, and doesn't overstate the method's capabilities or results.

<score>1</score>

Justification:
1. The evidence directly supports the claim through multiple statements:
- "To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions"
- "Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy"

2. The description of Q-Mixing (Equation 1) provides the technical foundation for how action-value estimates are combined.

3. The claim is well-grounded in specific implementation details without exaggerating the method's effectiveness or making unsupported assertions.

4. While the reviewer comment notes the straightforwardness of the changes, it doesn't contradict the technical accuracy of the claim's description.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The second, Mixed-Opponents, constructs a pure-strategy opponent by mixing existing strategy's action-value estimates, instead of their policies.

The review comment to be evaluated is:

The paper focuses on resolving the computational and sample efficiency challenges with current PSRO style approaches. To this end it proposes two different modifications to the standard PSRO setup: 1) Mixed Oracles, and 2) Mixed Opponents. These approaches allow avoiding resetting learning after each outer loop epoch and reduce the stochasticity of dynamics during training. Thee efficacy is demonstrated on relatively simple games but using Deep RL policies where the proposed approaches are at least on par with standard PSRO approach in terms of final performance while drastically improving the sample efficiency.

Although the paper has quite a few typos and unclear writing in places, it has a fantastic setup with clear description of the problems with standard PSRO it's trying to resolve. The paper points out the clear computational deficiencies with current PSRO style approaches: 1) starting anew everytime, 2) slower exploration of strategy space, 3) stochastic dynamics making learning difficult.

The paper is mostly focused on two-player zero sum games.
Approach heavily dependent on efficacy of Q-mixing approach (although the idea is more general), so currently limited to discrete action problems.
Moreover, since there are no alternatives to Q-mixing, we don't actually get any understanding/intuition for why this works and how important is the performance of "TransferOracle" or "OpponentOracle".

Although section 4.4 suggests that the two stage hyperparameter selection wasn't as important, results in Fig 6b are too noisy to fully accept. It's unclear how much compute/samples were required for the hyper parameter selection. In general the results are a lot more noisier for the proposed approaches vs standard PSRO.

**Minor**
- Environment dynamics should be SxA rather than OxA
- Algorithm 4.1 does not exist?
- Fig 1. Is it because of PSRO with Nash. Would a different MSS work differently?
- "proposes MCC as a solution concept": not everyone would know what MCC refers to.

The evidence to be evaluated is:In each epoch, each player learns a BR to a mixed profile of opponent policies. This mixture typically involves the newly added strategies (one per player) for this epoch, but may also include strategies from previous epochs.

A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions (Figure 1).

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training.

We employ Q-Mixing (Equation 1) as our Combine-Oponents. In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
1. The claim states that Mixed-Opponents constructs a pure-strategy opponent by mixing action-value estimates. The evidence strongly supports this through detailed explanation of Q-Mixing implementation and its application in Mixed-Opponents approach. The evidence clearly shows that Mixed-Opponents uses Q-Mixing to average Q-values and create a single opponent policy, rather than mixing policies directly. The reviewer comments don't dispute this technical aspect. The claim is well-stated and accurately reflects the methodology described in the evidence without exaggeration.

2. <score>1</score>

Justification:
- The claim is precise and technically accurate, directly supported by multiple pieces of evidence:
  - "mix the opponent strategies through their action-value estimates, rather than their action distributions"
  - "Q-Mixing (Equation 1) as our Combine-Oponents...apply it to average Q-values"
  - Clear distinction from Mixed-Oracles which uses Q-Mixing differently
- The claim doesn't overstate capabilities or results
- While reviewer notes limitations of Q-mixing approach, these don't affect the validity of the technical claim about how Mixed-Opponents works
- The claim sticks to describing the mechanism without making broader performance claims


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The second, Mixed-Opponents, constructs a pure-strategy opponent by mixing existing strategy's action-value estimates, instead of their policies.

The review comment to be evaluated is:

Summary
The paper proposes two new methods in the Policy-Space Response Oracle framework. These approaches permit to reuse past knowledge in order to reduce the amount of data required for the RL training. The first algorithm Mixed-Oracles transfers the previous iteration of Deep RL, instead of the second one, Mixed-Opponents, transfers existing strategy action-value estimates.

Strengths
The paper proposes two convincing alternatives to reuse previous knowledge in the PSRO framework. The two ideas are based on Q-mixing approach: the first one uses this method to transfer Q-values across epochs, the second one to design a new training objective.
The experiments show that the proposed methods find a good solution using less simulation than the original PSRO framework.

Weakness
The paper is not very novel, since it uses previous approaches (PSRO and Q-mixing) to transfer knowledge for the PSRO framework.
I am not aware of recent works in this framework but could be useful to compare the proposed approaches with P2RO [1].
The Mixed-Opponent section needs a better explanation of the use of Q-mixing as a training objective.

[1]  McAleer, S., Lanier, J., Fox, R., & Baldi, P. (2020). Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. arXiv preprint arXiv:2006.08555.

The evidence to be evaluated is:In each epoch, each player learns a BR to a mixed profile of opponent policies. This mixture typically involves the newly added strategies (one per player) for this epoch, but may also include strategies from previous epochs.

A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions (Figure 1).

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training.

We employ Q-Mixing (Equation 1) as our Combine-Oponents. In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "The second, Mixed-Opponents, constructs a pure-strategy opponent by mixing existing strategy's action-value estimates, instead of their policies."

Analysis:
The evidence strongly supports this claim. Multiple passages directly confirm that Mixed-Opponents:
- Uses Q-Mixing to average Q-values (action-value estimates)
- Constructs a single opponent policy representing an aggregate of mixed strategies
- Differs from combining policies directly by working with action-value estimates

The reviewer's comment that "The Mixed-Opponent section needs a better explanation" suggests some lack of clarity but doesn't contradict the claim's accuracy.

The claim is precise and well-supported by the evidence without overstatement. It accurately describes the technical approach while staying within the bounds of what is demonstrated.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The second, Mixed-Opponents, constructs a pure-strategy opponent by mixing existing strategy's action-value estimates, instead of their policies.

The review comment to be evaluated is:

##########################################################################
Summary:

The paper provides an interesting approach to speeding up the convergence time of the Policy-Space Response Oracles framework by re-using the Q-functions of past best-responses to transfer knowledge across epochs.
##########################################################################
Reasons for score:

Overall, I am low confidence on my assessment of this paper due to the exposition in the algorithm section being relatively confusing. The experimental results are interesting which suggests that the method has value but there is key missing information on how the best response policies are constructed that make it difficult to assess the paper and lead to my not wanting to recommend its acceptance. I would highly recommend being more detailed in Sec. 3 to allow me to reassess the paper. I would certainly be willing to update my score if the paper was clearer to read.
 ##########################################################################Pros:

1. The experimental results on Leduc Poker are very speedy in terms of time-steps.

2. The idea of reusing the prior Q functions and just mixing them together rather than re-learning all of the policies is very good.

##########################################################################
Cons:

1. The algorithm boxes are so high level that I am struggling to understand how the algorithms work. I would not be able to implement it from reading the paper.

#########################################################################
Things that would improve readability:

- It would be nice in the algorithm boxes to connect Q-mixing to how the best policy is explicitly output. I was not able to understand how Q-mixing was connected to either Algorithm 2 or 3 and subsequently had difficulty following the paper.
- \lambda does not appear to be defined anywhere but appears in the Mixed Oracles algorithm box
- What is the OpponentOracle and the TransferOracle? They are defined in the algorithm boxes but are not clearly defined elsewhere.
- The specific example of RPS in section 3.2 does not provide useful intuition by going through the numerics, it may be more helpful to walk through a more high level description.
- It would probably be useful to move more of the experimental results to an appendix to leave room for the exposition of the algorithms.

The evidence to be evaluated is:In each epoch, each player learns a BR to a mixed profile of opponent policies. This mixture typically involves the newly added strategies (one per player) for this epoch, but may also include strategies from previous epochs.

A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions (Figure 1).

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training.

We employ Q-Mixing (Equation 1) as our Combine-Oponents. In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim against the provided evidence:

The claim states that Mixed-Opponents "constructs a pure-strategy opponent by mixing existing strategy's action-value estimates, instead of their policies."

From the evidence, this appears to be a somewhat oversimplified description. The evidence shows that Mixed-Opponents:
- Uses Q-Mixing to average Q-values to define a variant training objective
- Constructs a single opponent policy representing an aggregate of the mixed strategy
- Applies Q-Mixing differently than Mixed-Oracles (which uses it for transfer across epochs)

The review comments indicate confusion about algorithm implementation details, suggesting some lack of clarity in how the mixing is actually performed. While the claim captures the basic concept, it oversimplifies the technical nuances of how the Q-mixing is applied and omits important context about the training objective.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Learning against a single policy mitigates variance in state outcomes that is induced by an unobserved distribution of opponents.

The evidence to be evaluated is:A _normal form games_ (NFG) \(\Gamma\!=\!(\Pi,\!U,\!n)\) describes a one-shot strategic interaction among \(n\) players. When there is more than one agent, we denote agent-specific components with subscripts (e.g., \(\pi_{i}\)). Negated subscripts represent the joint elements of all other agents (e.g., \(\pi_{-i}\)). Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

The sum of regrets across all players \(\text{SumRegret}(\sigma,\overline{\Pi})=\sum_{i\in n}\text{Regret}_{i}(\sigma, \overline{\Pi}_{i})\), sometimes called the Nash convergence, is a measure of how stable a solution is. At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs.

Secondly, the opponent profiles are mixtures (i.e., distribution over opponent policies). This makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult.

In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy. Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters. A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: _pure-hparams_ performed best on-average against each individual test-opponent, and _mix-hparams_ performed best against a uniform mixture of the test-opponents. These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.

We used two sets of hyperparameters: specialized for low- and high-variance outcomes of state, which are a result of facing pure- and mixed-strategy opponents respectively. This was motivated by the assumption that lower variance would require less training.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the evidence substantiates the core idea but the claim could be more precisely stated. The evidence shows that Mixed-Opponents algorithm does address variance by creating a single aggregated opponent policy rather than learning against a distribution of opponents. Figure 4 demonstrates improved performance (lower SumRegret) compared to standard PSRO. However, the claim would be more accurate if it specified that learning against a "combined/aggregated" policy (rather than "single") mitigates variance. The evidence specifically discusses combining opponent strategies into a representative policy.

<score>2</score>

Justification:
1. The evidence directly supports variance reduction through the Mixed-Opponents approach, showing both theoretical explanation and empirical results in Figure 4.
2. The text explains: "Mixed-Opponents...modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy"
3. While the core concept is accurate, the claim's wording of "single policy" could imply a simpler approach than the actual aggregation method described
4. The evidence demonstrates clear performance improvements but also acknowledges implementation complexities that the claim's simplicity doesn't capture
5. The improved performance against variance is empirically supported by Figure 4's results showing lower and more stable SumRegret values

The claim earns a 2 (mostly well-stated) because while its main point is supported, more precise wording would better reflect the technical details presented in the evidence.