Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Mixed-Oppennts applies directly over the joint opponent space, and so has the potential to scale beyond two-player games.

The review comment to be evaluated is:

The paper focuses on resolving the computational and sample efficiency challenges with current PSRO style approaches. To this end it proposes two different modifications to the standard PSRO setup: 1) Mixed Oracles, and 2) Mixed Opponents. These approaches allow avoiding resetting learning after each outer loop epoch and reduce the stochasticity of dynamics during training. Thee efficacy is demonstrated on relatively simple games but using Deep RL policies where the proposed approaches are at least on par with standard PSRO approach in terms of final performance while drastically improving the sample efficiency.

Although the paper has quite a few typos and unclear writing in places, it has a fantastic setup with clear description of the problems with standard PSRO it's trying to resolve. The paper points out the clear computational deficiencies with current PSRO style approaches: 1) starting anew everytime, 2) slower exploration of strategy space, 3) stochastic dynamics making learning difficult.

The paper is mostly focused on two-player zero sum games.
Approach heavily dependent on efficacy of Q-mixing approach (although the idea is more general), so currently limited to discrete action problems.
Moreover, since there are no alternatives to Q-mixing, we don't actually get any understanding/intuition for why this works and how important is the performance of "TransferOracle" or "OpponentOracle".

Although section 4.4 suggests that the two stage hyperparameter selection wasn't as important, results in Fig 6b are too noisy to fully accept. It's unclear how much compute/samples were required for the hyper parameter selection. In general the results are a lot more noisier for the proposed approaches vs standard PSRO.

**Minor**
- Environment dynamics should be SxA rather than OxA
- Algorithm 4.1 does not exist?
- Fig 1. Is it because of PSRO with Nash. Would a different MSS work differently?
- "proposes MCC as a solution concept": not everyone would know what MCC refers to.

The evidence to be evaluated is:A variety of other MSSs have been proposed, and assessing their relative merits is a topic of active research (Balduzzi et al., 2019). In experiments reported here, we adopt NE as the baseline MSS; however, our methods readily apply to any MSS.

In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy. The resulting policy should approximately aggregate the behavior of the policies.

The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training.

The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players.

We run both algorithms on the Gathering-Open game with three players.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent. Mixed-Opponents also trivially scales to games with more than two players.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim "Mixed-Opponents applies directly over the joint opponent space, and so has the potential to scale beyond two-player games":

While the evidence shows that Mixed-Opponents can theoretically handle more than two players (as stated in "An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players"), the empirical validation is limited to just one three-player game (Gathering-Open). The claim about "potential to scale" is carefully worded and technically correct, as it doesn't make strong assertions about proven scalability to many players, just the theoretical capability. The review comment notes that the paper is "mostly focused on two-player zero sum games" and that the approach is "heavily dependent on efficacy of Q-mixing," but these limitations don't directly contradict the careful wording of the claim about potential scalability.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Mixed-Oppennts applies directly over the joint opponent space, and so has the potential to scale beyond two-player games.

The review comment to be evaluated is:

Summary
The paper proposes two new methods in the Policy-Space Response Oracle framework. These approaches permit to reuse past knowledge in order to reduce the amount of data required for the RL training. The first algorithm Mixed-Oracles transfers the previous iteration of Deep RL, instead of the second one, Mixed-Opponents, transfers existing strategy action-value estimates.

Strengths
The paper proposes two convincing alternatives to reuse previous knowledge in the PSRO framework. The two ideas are based on Q-mixing approach: the first one uses this method to transfer Q-values across epochs, the second one to design a new training objective.
The experiments show that the proposed methods find a good solution using less simulation than the original PSRO framework.

Weakness
The paper is not very novel, since it uses previous approaches (PSRO and Q-mixing) to transfer knowledge for the PSRO framework.
I am not aware of recent works in this framework but could be useful to compare the proposed approaches with P2RO [1].
The Mixed-Opponent section needs a better explanation of the use of Q-mixing as a training objective.

[1]  McAleer, S., Lanier, J., Fox, R., & Baldi, P. (2020). Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. arXiv preprint arXiv:2006.08555.

The evidence to be evaluated is:A variety of other MSSs have been proposed, and assessing their relative merits is a topic of active research (Balduzzi et al., 2019). In experiments reported here, we adopt NE as the baseline MSS; however, our methods readily apply to any MSS.

In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy. The resulting policy should approximately aggregate the behavior of the policies.

The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training.

The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players.

We run both algorithms on the Gathering-Open game with three players.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent. Mixed-Opponents also trivially scales to games with more than two players.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim "Mixed-Opponents applies directly over the joint opponent space, and so has the potential to scale beyond two-player games."

The evidence supports this claim in several ways:
1. The text explicitly states "An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players."
2. The authors demonstrate this by running experiments on a three-player game (Gathering-Open).
3. The method uses Q-Mixing which combines opponent strategies into a single policy, naturally extending to multiple opponents.

The claim is well-stated (score 1) as it:
- Makes a modest claim about "potential to scale"
- Is directly supported by both theoretical explanation and experimental evidence
- Avoids overgeneralization or exaggeration
- Has been demonstrated in practice with a three-player game

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Mixed-Oppennts applies directly over the joint opponent space, and so has the potential to scale beyond two-player games.

The review comment to be evaluated is:

##########################################################################
Summary:

The paper provides an interesting approach to speeding up the convergence time of the Policy-Space Response Oracles framework by re-using the Q-functions of past best-responses to transfer knowledge across epochs.
##########################################################################
Reasons for score:

Overall, I am low confidence on my assessment of this paper due to the exposition in the algorithm section being relatively confusing. The experimental results are interesting which suggests that the method has value but there is key missing information on how the best response policies are constructed that make it difficult to assess the paper and lead to my not wanting to recommend its acceptance. I would highly recommend being more detailed in Sec. 3 to allow me to reassess the paper. I would certainly be willing to update my score if the paper was clearer to read.
 ##########################################################################Pros:

1. The experimental results on Leduc Poker are very speedy in terms of time-steps.

2. The idea of reusing the prior Q functions and just mixing them together rather than re-learning all of the policies is very good.

##########################################################################
Cons:

1. The algorithm boxes are so high level that I am struggling to understand how the algorithms work. I would not be able to implement it from reading the paper.

#########################################################################
Things that would improve readability:

- It would be nice in the algorithm boxes to connect Q-mixing to how the best policy is explicitly output. I was not able to understand how Q-mixing was connected to either Algorithm 2 or 3 and subsequently had difficulty following the paper.
- \lambda does not appear to be defined anywhere but appears in the Mixed Oracles algorithm box
- What is the OpponentOracle and the TransferOracle? They are defined in the algorithm boxes but are not clearly defined elsewhere.
- The specific example of RPS in section 3.2 does not provide useful intuition by going through the numerics, it may be more helpful to walk through a more high level description.
- It would probably be useful to move more of the experimental results to an appendix to leave room for the exposition of the algorithms.

The evidence to be evaluated is:A variety of other MSSs have been proposed, and assessing their relative merits is a topic of active research (Balduzzi et al., 2019). In experiments reported here, we adopt NE as the baseline MSS; however, our methods readily apply to any MSS.

In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy. The resulting policy should approximately aggregate the behavior of the policies.

The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training.

The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players.

We run both algorithms on the Gathering-Open game with three players.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent. Mixed-Opponents also trivially scales to games with more than two players.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me analyze the claim "Mixed-Opponents applies directly over the joint opponent space, and so has the potential to scale beyond two-player games" based on the provided evidence:

The evidence provides both theoretical and empirical support for this claim:

1. The paper explicitly states "An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players"

2. They demonstrate this empirically by running "both algorithms on the Gathering-Open game with three players"

3. The results show Mixed-Opponents performed well in this multi-player setting, finding "better solution than PSRO in half the simulation timesteps"

4. The algorithmic approach using Q-Mixing for combining opponent strategies appears theoretically sound for handling multiple opponents

The claim is well-stated (score of 1) as it:
- Makes a modest assertion about "potential to scale"
- Is backed by both theoretical explanation and empirical demonstration
- The actual implementation and results confirm the claimed capability

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We evaluate our methods in a series of experimental games, and find that both Mixed-Oracles and Mixed-Oppennts are able to find solutions at least as good as an unmodified PSRO, and often better, while providing a substantial reduction in the required amount of training simulation.

The evidence to be evaluated is:By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps? We begin by running both algorithms on the Gathering-Small and Leduc-Poker games. We record regrets over PSRO epochs and cumulative training timesteps. Next, we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?

An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players.

We run both algorithms on the Gathering-Open game with three players.

Figure 5(a) shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the time.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget. We report results for the Gathering-Small game in Figure 5(b).

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

In this study, we investigate PSRO's approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 5: PSRO compared against Mixed-Opponents on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 2: PSRO compared against Mixed-Oracles on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim to be generally well-supported but slightly overstated. The graphs (Figures 2, 4, and 5) demonstrate that both Mixed-Oracles and Mixed-Opponents achieve similar or better performance compared to PSRO with reduced training steps. However, the claim's certainty that they are "at least as good as" PSRO is somewhat overstated, as there is visible variance in performance (shown by shaded regions) and some periods where PSRO performs better. The reduction in training simulation is well-supported by the timesteps graphs.

<score>2</score>

Justification:
1. Evidence supporting accuracy:
- Figures show both methods generally achieve similar or better convergence with fewer timesteps
- Multiple game environments tested (Gathering-Small, Leduc-Poker, Gathering-Open)
- Clear demonstration of reduced training timesteps

2. Minor overstatement concerns:
- Performance varies and isn't consistently "at least as good" throughout all epochs
- Significant variance shown in shaded regions suggests less certainty than claimed
- Limited experimental details about statistical significance of improvements
- Some periods show PSRO outperforming the proposed methods

The claim would be better stated with slight qualifications acknowledging the variability in performance while maintaining the core finding about reduced training requirements.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We evaluate our methods in a series of experimental games, and find that both Mixed-Oracles and Mixed-Oppennts are able to find solutions at least as good as an unmodified PSRO, and often better, while providing a substantial reduction in the required amount of training simulation.

The review comment to be evaluated is:

The paper suggests two techniques to improve the calculation of empirically figuring out a Nash equilibrium using an iterative application of best-response dynamics. One method learns the best-response to the previously used strategy. The other uses that technique to model the opponent, and then best-responds to the modeled opponent. The experiments show a faster reaching to NE than without these changes.

The paper is well-written and explained, and is accessible even to researchers not well-versed in ML topics. While, the suggested changes are rather straightforward, they do indeed lead to the expected advance (shorter times to reach NE). I was convinced by further introspection that this is a significant enough contribution, to merit acceptance.

Of course, a more significant change to the algorithm, leading not only to a shorter time but to convergence to better equilibria (in cases where multiple exist) would be far more compelling.

The evidence to be evaluated is:By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps? We begin by running both algorithms on the Gathering-Small and Leduc-Poker games. We record regrets over PSRO epochs and cumulative training timesteps. Next, we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?

An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players.

We run both algorithms on the Gathering-Open game with three players.

Figure 5(a) shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the time.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget. We report results for the Gathering-Small game in Figure 5(b).

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

In this study, we investigate PSRO's approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 5: PSRO compared against Mixed-Opponents on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 2: PSRO compared against Mixed-Oracles on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find it to be mostly well-stated but slightly overstated. The graphs (Figures 2, 4, and 5) demonstrate that both Mixed-Oracles and Mixed-Opponents generally achieve similar or better performance compared to PSRO with fewer training timesteps. However, the claim could be more precise about the conditions under which improvements occur, as performance varies across games and scenarios. The review comment supports this assessment, noting that while the improvements are real, they are "rather straightforward" and primarily affect convergence speed rather than equilibrium quality.

<score>2</score>

Justification:
1. Evidence supports faster convergence with fewer timesteps (shown in right-side plots of Figures 2, 4, and 5)
2. Similar or better solution quality is demonstrated in most cases
3. The claim could benefit from:
   - Specifying the types of games where improvements are most significant
   - Acknowledging variability in performance across different scenarios
   - Quantifying the "substantial reduction" in training simulation
4. The reviewer confirms the performance improvements but characterizes them as incremental rather than transformative

The claim is mostly accurate but would be stronger with more specific qualifications about when and how the improvements manifest.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We evaluate our methods in a series of experimental games, and find that both Mixed-Oracles and Mixed-Oppennts are able to find solutions at least as good as an unmodified PSRO, and often better, while providing a substantial reduction in the required amount of training simulation.

The review comment to be evaluated is:

The paper focuses on resolving the computational and sample efficiency challenges with current PSRO style approaches. To this end it proposes two different modifications to the standard PSRO setup: 1) Mixed Oracles, and 2) Mixed Opponents. These approaches allow avoiding resetting learning after each outer loop epoch and reduce the stochasticity of dynamics during training. Thee efficacy is demonstrated on relatively simple games but using Deep RL policies where the proposed approaches are at least on par with standard PSRO approach in terms of final performance while drastically improving the sample efficiency.

Although the paper has quite a few typos and unclear writing in places, it has a fantastic setup with clear description of the problems with standard PSRO it's trying to resolve. The paper points out the clear computational deficiencies with current PSRO style approaches: 1) starting anew everytime, 2) slower exploration of strategy space, 3) stochastic dynamics making learning difficult.

The paper is mostly focused on two-player zero sum games.
Approach heavily dependent on efficacy of Q-mixing approach (although the idea is more general), so currently limited to discrete action problems.
Moreover, since there are no alternatives to Q-mixing, we don't actually get any understanding/intuition for why this works and how important is the performance of "TransferOracle" or "OpponentOracle".

Although section 4.4 suggests that the two stage hyperparameter selection wasn't as important, results in Fig 6b are too noisy to fully accept. It's unclear how much compute/samples were required for the hyper parameter selection. In general the results are a lot more noisier for the proposed approaches vs standard PSRO.

**Minor**
- Environment dynamics should be SxA rather than OxA
- Algorithm 4.1 does not exist?
- Fig 1. Is it because of PSRO with Nash. Would a different MSS work differently?
- "proposes MCC as a solution concept": not everyone would know what MCC refers to.

The evidence to be evaluated is:By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps? We begin by running both algorithms on the Gathering-Small and Leduc-Poker games. We record regrets over PSRO epochs and cumulative training timesteps. Next, we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?

An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players.

We run both algorithms on the Gathering-Open game with three players.

Figure 5(a) shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the time.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget. We report results for the Gathering-Small game in Figure 5(b).

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

In this study, we investigate PSRO's approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 5: PSRO compared against Mixed-Opponents on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 2: PSRO compared against Mixed-Oracles on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the provided evidence and review comments, I'll evaluate the claim:

Summary: The evidence from the figures (2, 4, and 5) demonstrates that both Mixed-Oracles and Mixed-Opponents achieve comparable or better performance than PSRO while using fewer training timesteps across multiple games (Gathering-Small, Leduc-Poker). However, the review points out important limitations: the results are noisier for the proposed approaches, the evaluation is limited to discrete action problems, and there's dependency on Q-mixing efficacy. The claim acknowledges these aspects by using measured language ("at least as good" and "often better") and directly ties to the evidence shown.

The claim appears well-stated (1) as it:
- Accurately reflects the experimental results shown in figures
- Uses appropriately cautious language
- Doesn't overextend beyond what the evidence supports
- Clearly states both the performance and efficiency benefits
- Aligns with the reviewer's assessment of the results

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We evaluate our methods in a series of experimental games, and find that both Mixed-Oracles and Mixed-Oppennts are able to find solutions at least as good as an unmodified PSRO, and often better, while providing a substantial reduction in the required amount of training simulation.

The review comment to be evaluated is:

Summary
The paper proposes two new methods in the Policy-Space Response Oracle framework. These approaches permit to reuse past knowledge in order to reduce the amount of data required for the RL training. The first algorithm Mixed-Oracles transfers the previous iteration of Deep RL, instead of the second one, Mixed-Opponents, transfers existing strategy action-value estimates.

Strengths
The paper proposes two convincing alternatives to reuse previous knowledge in the PSRO framework. The two ideas are based on Q-mixing approach: the first one uses this method to transfer Q-values across epochs, the second one to design a new training objective.
The experiments show that the proposed methods find a good solution using less simulation than the original PSRO framework.

Weakness
The paper is not very novel, since it uses previous approaches (PSRO and Q-mixing) to transfer knowledge for the PSRO framework.
I am not aware of recent works in this framework but could be useful to compare the proposed approaches with P2RO [1].
The Mixed-Opponent section needs a better explanation of the use of Q-mixing as a training objective.

[1]  McAleer, S., Lanier, J., Fox, R., & Baldi, P. (2020). Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. arXiv preprint arXiv:2006.08555.

The evidence to be evaluated is:By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps? We begin by running both algorithms on the Gathering-Small and Leduc-Poker games. We record regrets over PSRO epochs and cumulative training timesteps. Next, we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?

An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players.

We run both algorithms on the Gathering-Open game with three players.

Figure 5(a) shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the time.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget. We report results for the Gathering-Small game in Figure 5(b).

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

In this study, we investigate PSRO's approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 5: PSRO compared against Mixed-Opponents on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 2: PSRO compared against Mixed-Oracles on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be well-stated and accurately reflects the experimental results. The figures show that both Mixed-Oracles and Mixed-Opponents achieve similar or better performance (lower regret) compared to PSRO while using fewer training timesteps across multiple games (Gathering-Small, Leduc-Poker). The review comment supports this, noting that "the experiments show that the proposed methods find a good solution using less simulation than the original PSRO framework." The claim is conservative in stating "at least as good" rather than overstating superiority.

<score>1</score>

Justification:
1. Evidence directly supports claim through:
- Figures 2, 4, and 5 showing comparable or better regret metrics
- Results on multiple games (Gathering-Small, Leduc-Poker)
- Quantifiable reduction in training timesteps (e.g., "half the simulation timesteps")
- Conservative wording ("at least as good" rather than claiming superiority)
2. Review comment validates findings regarding reduced simulation needs
3. No instances of exaggeration or unsupported statements found
4. Multiple experimental scenarios and games tested
5. Clear graphical evidence demonstrating reduced training time while maintaining performance


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We evaluate our methods in a series of experimental games, and find that both Mixed-Oracles and Mixed-Oppennts are able to find solutions at least as good as an unmodified PSRO, and often better, while providing a substantial reduction in the required amount of training simulation.

The review comment to be evaluated is:

##########################################################################
Summary:

The paper provides an interesting approach to speeding up the convergence time of the Policy-Space Response Oracles framework by re-using the Q-functions of past best-responses to transfer knowledge across epochs.
##########################################################################
Reasons for score:

Overall, I am low confidence on my assessment of this paper due to the exposition in the algorithm section being relatively confusing. The experimental results are interesting which suggests that the method has value but there is key missing information on how the best response policies are constructed that make it difficult to assess the paper and lead to my not wanting to recommend its acceptance. I would highly recommend being more detailed in Sec. 3 to allow me to reassess the paper. I would certainly be willing to update my score if the paper was clearer to read.
 ##########################################################################Pros:

1. The experimental results on Leduc Poker are very speedy in terms of time-steps.

2. The idea of reusing the prior Q functions and just mixing them together rather than re-learning all of the policies is very good.

##########################################################################
Cons:

1. The algorithm boxes are so high level that I am struggling to understand how the algorithms work. I would not be able to implement it from reading the paper.

#########################################################################
Things that would improve readability:

- It would be nice in the algorithm boxes to connect Q-mixing to how the best policy is explicitly output. I was not able to understand how Q-mixing was connected to either Algorithm 2 or 3 and subsequently had difficulty following the paper.
- \lambda does not appear to be defined anywhere but appears in the Mixed Oracles algorithm box
- What is the OpponentOracle and the TransferOracle? They are defined in the algorithm boxes but are not clearly defined elsewhere.
- The specific example of RPS in section 3.2 does not provide useful intuition by going through the numerics, it may be more helpful to walk through a more high level description.
- It would probably be useful to move more of the experimental results to an appendix to leave room for the exposition of the algorithms.

The evidence to be evaluated is:By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps? We begin by running both algorithms on the Gathering-Small and Leduc-Poker games. We record regrets over PSRO epochs and cumulative training timesteps. Next, we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?

An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players.

We run both algorithms on the Gathering-Open game with three players.

Figure 5(a) shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the time.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget. We report results for the Gathering-Small game in Figure 5(b).

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

In this study, we investigate PSRO's approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input: Figure 5: PSRO compared against Mixed-Opponents on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input: Figure 2: PSRO compared against Mixed-Oracles on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence and reviewer comments, I'll evaluate the claim that "both Mixed-Oracles and Mixed-Opponents are able to find solutions at least as good as an unmodified PSRO, and often better, while providing a substantial reduction in the required amount of training simulation."

The graphs in Figures 2, 4, and 5 provide empirical support showing both methods achieve similar or better regret scores compared to PSRO while using fewer timesteps. However, the reviewer points out important limitations in understanding how the algorithms work due to high-level descriptions and missing details. The claim appears mostly supported by the experimental results shown, though some caution is warranted given the reviewer's concerns about implementation clarity.

<score>2</score>

Justification:
1. The visual evidence clearly shows both methods achieving similar or better regret scores compared to PSRO
2. The timesteps plots demonstrate reduced training time
3. Results are shown across multiple games (Gathering-Small, Leduc-Poker)
4. The reviewer confirms the "experimental results are interesting" and "speedy in terms of time-steps"
5. Some caution is warranted due to:
   - Reviewer's difficulty understanding implementation details
   - Limited explanation of key components (λ, OpponentOracle, TransferOracle)
   - High-level algorithm descriptions that may affect reproducibility

The claim is relatively well-stated as it aligns with the evidence shown, though it could benefit from acknowledging some implementation uncertainties highlighted by the reviewer.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Both algorithms modify how PSRO adds new policies to the empirical game, based on learned responses to a single opponent policy.

The evidence to be evaluated is:At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In PSRO, the strategy exploration problem is decomposed into two steps: solution and BR via RL.

In the BR step, PSRO generates a new policy via RL (the _response oracle_), training against the target opponent profile computed by the MSS. The choice of MSS and response oracle algorithm constitutes a strategy exploration approach and determines the convergence speed of PSRO.

In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent. In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies. The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

If, as in PSRO, player 1 then adds their BR to this meta-strategy, which is P, then their strategy set still does not contain the Nash equilibrium of the game. The problem is that there are already strategies in the empirical game that can defeat player 2's strategies, so the new strategy is not useful.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Input: Initial policy sets for all players \(\Pi^{0}\)  Simulate utilities \(\tilde{U}^{\Pi^{0}}\) for each joint \(\pi\!\in\!\Pi^{0}\)  Initialize solutions \(\sigma_{i}^{*,0}\!=\!\text{Uniform}(\Pi_{i}^{0})\) whileepoch e in \(\{1,2,...\}\)do for player \(i\!\in\![[n]]\)do for many episodesdo \(\pi_{-i}\!\sim\!\sigma_{-i}^{*-1}\)  Train \(\pi_{i}^{e}\) over \(\tau\!\sim\!(\pi_{i}^{e}\!,\!\pi_{-i})\) \(\Pi_{i}^{e}\!=\!\Pi_{i}^{-1}\!\cup\!\{\pi_{i}^{e}\}\)  Simulate missing entries in \(\tilde{U}^{\Pi^{e}}\) from \(\Pi^{e}\)  Compute a solution \(\sigma^{*,e}\) from \(\tilde{\Gamma}^{e}\) Output: Current solution \(\sigma_{i}^{*,e}\) for player \(i\)

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Their approach is still seen in PSRO today, where iterative BRs to equilibrium are used to build an empirical game.

PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

Wright et al. (2019) extends DO's MSS to return a decay-weighted linear combination of the current and previous solutions. Omidshafiei et al. (2019) proposes Markov-Conley Chains (MCC) as a solution concept that relates to their new proposed quality measure of policy called \(\alpha\)-Rank, and Muller et al. (2020) investigated its use as a MSS.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "Both algorithms modify how PSRO adds new policies to the empirical game, based on learned responses to a single opponent policy."

Analysis:
The evidence strongly supports this claim. For Mixed-Oracles, the text explicitly states "During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS." For Mixed-Opponents, it clearly indicates that it "combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training." In both cases, the algorithms modify PSRO's approach by focusing on responses to a single opponent policy, rather than the mixed opponent-profile used in standard PSRO.

<score>1</score>

Justification:
1. The claim accurately reflects how both algorithms modify PSRO's policy addition mechanism
2. The evidence directly supports that both methods deal with single opponent policies:
   - Mixed-Oracles: "focuses training exclusively on new opponent policies"
   - Mixed-Opponents: "construct a single new opponent policy as a target for training"
3. There is no exaggeration or overstatement - the claim precisely describes the key modification both algorithms make to PSRO's policy addition process


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Both algorithms modify how PSRO adds new policies to the empirical game, based on learned responses to a single opponent policy.

The review comment to be evaluated is:

The paper suggests two techniques to improve the calculation of empirically figuring out a Nash equilibrium using an iterative application of best-response dynamics. One method learns the best-response to the previously used strategy. The other uses that technique to model the opponent, and then best-responds to the modeled opponent. The experiments show a faster reaching to NE than without these changes.

The paper is well-written and explained, and is accessible even to researchers not well-versed in ML topics. While, the suggested changes are rather straightforward, they do indeed lead to the expected advance (shorter times to reach NE). I was convinced by further introspection that this is a significant enough contribution, to merit acceptance.

Of course, a more significant change to the algorithm, leading not only to a shorter time but to convergence to better equilibria (in cases where multiple exist) would be far more compelling.

The evidence to be evaluated is:At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In PSRO, the strategy exploration problem is decomposed into two steps: solution and BR via RL.

In the BR step, PSRO generates a new policy via RL (the _response oracle_), training against the target opponent profile computed by the MSS. The choice of MSS and response oracle algorithm constitutes a strategy exploration approach and determines the convergence speed of PSRO.

In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent. In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies. The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

If, as in PSRO, player 1 then adds their BR to this meta-strategy, which is P, then their strategy set still does not contain the Nash equilibrium of the game. The problem is that there are already strategies in the empirical game that can defeat player 2's strategies, so the new strategy is not useful.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Input: Initial policy sets for all players \(\Pi^{0}\)  Simulate utilities \(\tilde{U}^{\Pi^{0}}\) for each joint \(\pi\!\in\!\Pi^{0}\)  Initialize solutions \(\sigma_{i}^{*,0}\!=\!\text{Uniform}(\Pi_{i}^{0})\) whileepoch e in \(\{1,2,...\}\)do for player \(i\!\in\![[n]]\)do for many episodesdo \(\pi_{-i}\!\sim\!\sigma_{-i}^{*-1}\)  Train \(\pi_{i}^{e}\) over \(\tau\!\sim\!(\pi_{i}^{e}\!,\!\pi_{-i})\) \(\Pi_{i}^{e}\!=\!\Pi_{i}^{-1}\!\cup\!\{\pi_{i}^{e}\}\)  Simulate missing entries in \(\tilde{U}^{\Pi^{e}}\) from \(\Pi^{e}\)  Compute a solution \(\sigma^{*,e}\) from \(\tilde{\Gamma}^{e}\) Output: Current solution \(\sigma_{i}^{*,e}\) for player \(i\)

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Their approach is still seen in PSRO today, where iterative BRs to equilibrium are used to build an empirical game.

PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

Wright et al. (2019) extends DO's MSS to return a decay-weighted linear combination of the current and previous solutions. Omidshafiei et al. (2019) proposes Markov-Conley Chains (MCC) as a solution concept that relates to their new proposed quality measure of policy called \(\alpha\)-Rank, and Muller et al. (2020) investigated its use as a MSS.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me analyze the claim: "Both algorithms modify how PSRO adds new policies to the empirical game, based on learned responses to a single opponent policy."

Analysis:
The evidence strongly supports this claim as:

1. Mixed-Oracles "learns a BR for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS" and reuses learned behaviors from previous epochs.

2. Mixed-Opponents "combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training."

Both algorithms indeed modify PSRO's policy addition method by working with single opponent policies, either by learning responses to individual policies (Mixed-Oracles) or by creating a combined single opponent policy (Mixed-Opponents). The claim accurately reflects the technical details without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Both algorithms modify how PSRO adds new policies to the empirical game, based on learned responses to a single opponent policy.

The review comment to be evaluated is:

The paper focuses on resolving the computational and sample efficiency challenges with current PSRO style approaches. To this end it proposes two different modifications to the standard PSRO setup: 1) Mixed Oracles, and 2) Mixed Opponents. These approaches allow avoiding resetting learning after each outer loop epoch and reduce the stochasticity of dynamics during training. Thee efficacy is demonstrated on relatively simple games but using Deep RL policies where the proposed approaches are at least on par with standard PSRO approach in terms of final performance while drastically improving the sample efficiency.

Although the paper has quite a few typos and unclear writing in places, it has a fantastic setup with clear description of the problems with standard PSRO it's trying to resolve. The paper points out the clear computational deficiencies with current PSRO style approaches: 1) starting anew everytime, 2) slower exploration of strategy space, 3) stochastic dynamics making learning difficult.

The paper is mostly focused on two-player zero sum games.
Approach heavily dependent on efficacy of Q-mixing approach (although the idea is more general), so currently limited to discrete action problems.
Moreover, since there are no alternatives to Q-mixing, we don't actually get any understanding/intuition for why this works and how important is the performance of "TransferOracle" or "OpponentOracle".

Although section 4.4 suggests that the two stage hyperparameter selection wasn't as important, results in Fig 6b are too noisy to fully accept. It's unclear how much compute/samples were required for the hyper parameter selection. In general the results are a lot more noisier for the proposed approaches vs standard PSRO.

**Minor**
- Environment dynamics should be SxA rather than OxA
- Algorithm 4.1 does not exist?
- Fig 1. Is it because of PSRO with Nash. Would a different MSS work differently?
- "proposes MCC as a solution concept": not everyone would know what MCC refers to.

The evidence to be evaluated is:At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In PSRO, the strategy exploration problem is decomposed into two steps: solution and BR via RL.

In the BR step, PSRO generates a new policy via RL (the _response oracle_), training against the target opponent profile computed by the MSS. The choice of MSS and response oracle algorithm constitutes a strategy exploration approach and determines the convergence speed of PSRO.

In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent. In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies. The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

If, as in PSRO, player 1 then adds their BR to this meta-strategy, which is P, then their strategy set still does not contain the Nash equilibrium of the game. The problem is that there are already strategies in the empirical game that can defeat player 2's strategies, so the new strategy is not useful.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Input: Initial policy sets for all players \(\Pi^{0}\)  Simulate utilities \(\tilde{U}^{\Pi^{0}}\) for each joint \(\pi\!\in\!\Pi^{0}\)  Initialize solutions \(\sigma_{i}^{*,0}\!=\!\text{Uniform}(\Pi_{i}^{0})\) whileepoch e in \(\{1,2,...\}\)do for player \(i\!\in\![[n]]\)do for many episodesdo \(\pi_{-i}\!\sim\!\sigma_{-i}^{*-1}\)  Train \(\pi_{i}^{e}\) over \(\tau\!\sim\!(\pi_{i}^{e}\!,\!\pi_{-i})\) \(\Pi_{i}^{e}\!=\!\Pi_{i}^{-1}\!\cup\!\{\pi_{i}^{e}\}\)  Simulate missing entries in \(\tilde{U}^{\Pi^{e}}\) from \(\Pi^{e}\)  Compute a solution \(\sigma^{*,e}\) from \(\tilde{\Gamma}^{e}\) Output: Current solution \(\sigma_{i}^{*,e}\) for player \(i\)

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Their approach is still seen in PSRO today, where iterative BRs to equilibrium are used to build an empirical game.

PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

Wright et al. (2019) extends DO's MSS to return a decay-weighted linear combination of the current and previous solutions. Omidshafiei et al. (2019) proposes Markov-Conley Chains (MCC) as a solution concept that relates to their new proposed quality measure of policy called \(\alpha\)-Rank, and Muller et al. (2020) investigated its use as a MSS.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim: "Both algorithms modify how PSRO adds new policies to the empirical game, based on learned responses to a single opponent policy."

Analysis:
The evidence clearly supports that both algorithms (Mixed-Oracles and Mixed-Opponents) modify PSRO's policy addition process, but the claim is slightly imprecise about how they work:

1. Mixed-Oracles: Learns BR against single new opponent policy but combines knowledge from previous BRs using Q-Mixing.

2. Mixed-Opponents: Creates a single aggregate opponent policy from the mixture but uses Q-Mixing for averaging Q-values during training.

While both methods do involve "single opponent" aspects, the claim oversimplifies their mechanisms and doesn't capture the full complexity of how they work with Q-Mixing and knowledge transfer.

The reviewer's comments support this assessment, noting the heavy dependence on Q-mixing and suggesting more clarity is needed about how the approaches work.

<score>2</score>

The claim gets the basic concept right but slightly oversimplifies the mechanisms. While not severely overstated, it could be more precise in describing how each algorithm handles opponent policies and combines responses.