Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We address this challenge through variants of PSRO in which all RL is applied to environments where opponents play pure strategies.

The review comment to be evaluated is:

##########################################################################
Summary:

The paper provides an interesting approach to speeding up the convergence time of the Policy-Space Response Oracles framework by re-using the Q-functions of past best-responses to transfer knowledge across epochs.
##########################################################################
Reasons for score:

Overall, I am low confidence on my assessment of this paper due to the exposition in the algorithm section being relatively confusing. The experimental results are interesting which suggests that the method has value but there is key missing information on how the best response policies are constructed that make it difficult to assess the paper and lead to my not wanting to recommend its acceptance. I would highly recommend being more detailed in Sec. 3 to allow me to reassess the paper. I would certainly be willing to update my score if the paper was clearer to read.
 ##########################################################################Pros:

1. The experimental results on Leduc Poker are very speedy in terms of time-steps.

2. The idea of reusing the prior Q functions and just mixing them together rather than re-learning all of the policies is very good.

##########################################################################
Cons:

1. The algorithm boxes are so high level that I am struggling to understand how the algorithms work. I would not be able to implement it from reading the paper.

#########################################################################
Things that would improve readability:

- It would be nice in the algorithm boxes to connect Q-mixing to how the best policy is explicitly output. I was not able to understand how Q-mixing was connected to either Algorithm 2 or 3 and subsequently had difficulty following the paper.
- \lambda does not appear to be defined anywhere but appears in the Mixed Oracles algorithm box
- What is the OpponentOracle and the TransferOracle? They are defined in the algorithm boxes but are not clearly defined elsewhere.
- The specific example of RPS in section 3.2 does not provide useful intuition by going through the numerics, it may be more helpful to walk through a more high level description.
- It would probably be useful to move more of the experimental results to an appendix to leave room for the exposition of the algorithms.

The evidence to be evaluated is:A _normal form games_ (NFG) \(\Gamma\!=\!(\Pi,\!U,\!n)\) describes a one-shot strategic interaction among \(n\) players.

Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In PSRO, the strategy exploration problem is decomposed into two steps: solution and BR via RL.

In the BR step, PSRO generates a new policy via RL (the _response oracle_), training against the target opponent profile computed by the MSS.

In experiments reported here, we adopt NE as the baseline MSS; however, our methods readily apply to any MSS. We observe and address two main problems with the standard version of PSRO (Algorithm 1). First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs. This suggests a compelling opportunity to transfer learning across epochs; however, the BR calculation step works by training anew. Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG. Secondly, the opponent profiles are mixtures (i.e., distribution over opponent policies). This makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

Consider the Rock-Paper-Scissors (R-P-S) game in figure 1. Player 1's initial strategy \(\pi_{1}^{1}\) mostly plays S, which induces a best response of R. Player 1's second strategy \(\pi_{1}^{2}\) is an approximate best response to R, scoring \(80\%\) against it. The best response to \(\pi_{1}^{2}\) is P. This leads to an empirical game similar to matching pennies, with player 2 playing an even mixture of Rock and Paper; in particular, player 2's meta-strategy never plays S, despite it being good against both player 1 strategies so far. If, as in PSRO, player 1 then adds their BR to this meta-strategy, which is P, then their strategy set still does not contain the Nash equilibrium of the game.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents.

Input: Initial policy sets for all players \(\Pi^{0}\)  Simulate utilities \(\tilde{U}^{\Pi^{0}}\) for each joint \(\pi\!\in\!\Pi^{0}\)  Initialize solutions \(\sigma_{i}^{*,0}\!=\!\text{Uniform}(\Pi_{i}^{0})\) whileepoch e in \(\{1,2,...\}\)do for player \(i\!\in\![[n]]\)do for many episodesdo \(\pi_{-i}\!\sim\!\sigma_{-i}^{*-1}\)  Train \(\pi_{i}^{e}\) over \(\tau\!\sim\!(\pi_{i}^{e}\!,\!\pi_{-i})\) \(\Pi_{i}^{e}\!=\!\Pi_{i}^{-1}\!\cup\!\{\pi_{i}^{e}\}\)  Simulate missing entries in \(\tilde{U}^{\Pi^{e}}\) from \(\Pi^{e}\)  Compute a solution \(\sigma^{*,e}\) from \(\tilde{\Gamma}^{e}\) Output: Current solution \(\sigma_{i}^{*,e}\) for player \(i\) ```**Algorithm 1**Policy-Space Response

First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy. Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters. A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: _pure-hparams_ performed best on-average against each individual test-opponent, and _mix-hparams_ performed best against a uniform mixture of the test-opponents. These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.

We used two sets of hyperparameters: specialized for low- and high-variance outcomes of state, which are a result of facing pure- and mixed-strategy opponents respectively.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget.

The trends observed in the previous experiment reoccur: Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Schvartzman and Wellman (2009b) followed this up by introducing reinforcement learning as the approximate oracle method. These algorithms, alongside a trend of deep reinforcement learning being used to solve difficult games, set the stage for the PSRO framework. PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

PSRO's strategy exploration happens in two phases: first, a MSS produces an opponent-profile as a learning objective, then reinforcement learning is used as an approximate oracle to best-respond to the objective.

Similarly, self-play's resulting profile is a pure strategy containing only the newest policy.

PSRO's strategy exploration occurs in two steps: solution and BR via RL.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input: Figure 4: PSRO compared against Mixed-Opponents on the Gathering-Small game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim that "all RL is applied to environments where opponents play pure strategies" appears to be overstated. While the paper does present variants of PSRO that work with pure strategies, the evidence shows that:

1. Mixed-Oracles algorithm maintains a collection of best responses to each opponent policy and uses Q-Mixing to handle mixed opponent strategies
2. Mixed-Opponents algorithm explicitly combines mixed strategies into a single opponent policy
3. The experiments use both "pure-hparams" and "mix-hparams" settings, indicating that both pure and mixed strategies are considered

The claim oversimplifies the actual approach, which involves more nuanced handling of both pure and mixed strategies.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_Mixed-Oracles_ learns separate best-responses (BR) to each pure strategy in a mixture and combines the results from learning to approximate a BR to the mixture.

The evidence to be evaluated is:Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies. The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies. We provide a general description of Mixed-Oracles, where the Combine-Respones method is abstract, as Algorithm 2. Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

Player 1's initial strategy \(\pi_{1}^{1}\) mostly plays S, which induces a best response of R. Player 1's second strategy \(\pi_{1}^{2}\) is an approximate best response to R, scoring \(80\%\) against it. The best response to \(\pi_{1}^{2}\) is P.

To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions (Figure 1).

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution. This affords our algorithm the ability to transfer learning about previously encountered opponents.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim against the provided evidence:

The evidence strongly supports the claim that Mixed-Oracles learns separate best-responses to pure strategies and combines them. The algorithmic description clearly shows that:
1. It maintains a collection of BRs to each opponent policy (Λᵢᵉ)
2. During each epoch, it learns a BR for only the new opponent policy
3. It uses Combine-Responses (specifically Q-Mixing) to construct a BR to mixtures by combining individual BRs
4. The Q-Mixing formula (1) shows exactly how responses are combined using weights from the opponent mixture

The claim is precise and accurately reflects the mechanics described in the evidence without exaggeration or oversimplification.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_Mixed-Oracles_ learns separate best-responses (BR) to each pure strategy in a mixture and combines the results from learning to approximate a BR to the mixture.

The review comment to be evaluated is:

The paper suggests two techniques to improve the calculation of empirically figuring out a Nash equilibrium using an iterative application of best-response dynamics. One method learns the best-response to the previously used strategy. The other uses that technique to model the opponent, and then best-responds to the modeled opponent. The experiments show a faster reaching to NE than without these changes.

The paper is well-written and explained, and is accessible even to researchers not well-versed in ML topics. While, the suggested changes are rather straightforward, they do indeed lead to the expected advance (shorter times to reach NE). I was convinced by further introspection that this is a significant enough contribution, to merit acceptance.

Of course, a more significant change to the algorithm, leading not only to a shorter time but to convergence to better equilibria (in cases where multiple exist) would be far more compelling.

The evidence to be evaluated is:Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies. The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies. We provide a general description of Mixed-Oracles, where the Combine-Respones method is abstract, as Algorithm 2. Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

Player 1's initial strategy \(\pi_{1}^{1}\) mostly plays S, which induces a best response of R. Player 1's second strategy \(\pi_{1}^{2}\) is an approximate best response to R, scoring \(80\%\) against it. The best response to \(\pi_{1}^{2}\) is P.

To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions (Figure 1).

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution. This affords our algorithm the ability to transfer learning about previously encountered opponents.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim about Mixed-Oracles based on the provided evidence:

1. Summary: The claim accurately reflects the core mechanism of Mixed-Oracles as described in the evidence. The evidence clearly shows that Mixed-Oracles learns separate best responses to individual opponent policies (λᵢᵉ for each πᵉ⁻¹) and combines them using a Combine-Responses function (specifically Q-Mixing in this implementation). The mathematical formulation in equation (1) provides concrete evidence of how responses are combined weighted by their likelihood. The review comment supports this by acknowledging the straightforward but effective nature of the approach. The claim is well-stated without exaggeration.

<score>1</score>

Justification:
1. The claim directly aligns with the evidence that states "Mixed-Oracles works by learning and maintaining a collection of BRs to each opponent policy" and combines them through "Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy"
2. The mathematical formulation (equation 1) provides explicit evidence of how responses are combined
3. The mechanism is experimentally validated, as noted in "Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps"
4. The review comment confirms the straightforward but effective nature of the approach
5. The claim avoids overstating capabilities and accurately represents the method's core functionality


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_Mixed-Oracles_ learns separate best-responses (BR) to each pure strategy in a mixture and combines the results from learning to approximate a BR to the mixture.

The review comment to be evaluated is:

The paper focuses on resolving the computational and sample efficiency challenges with current PSRO style approaches. To this end it proposes two different modifications to the standard PSRO setup: 1) Mixed Oracles, and 2) Mixed Opponents. These approaches allow avoiding resetting learning after each outer loop epoch and reduce the stochasticity of dynamics during training. Thee efficacy is demonstrated on relatively simple games but using Deep RL policies where the proposed approaches are at least on par with standard PSRO approach in terms of final performance while drastically improving the sample efficiency.

Although the paper has quite a few typos and unclear writing in places, it has a fantastic setup with clear description of the problems with standard PSRO it's trying to resolve. The paper points out the clear computational deficiencies with current PSRO style approaches: 1) starting anew everytime, 2) slower exploration of strategy space, 3) stochastic dynamics making learning difficult.

The paper is mostly focused on two-player zero sum games.
Approach heavily dependent on efficacy of Q-mixing approach (although the idea is more general), so currently limited to discrete action problems.
Moreover, since there are no alternatives to Q-mixing, we don't actually get any understanding/intuition for why this works and how important is the performance of "TransferOracle" or "OpponentOracle".

Although section 4.4 suggests that the two stage hyperparameter selection wasn't as important, results in Fig 6b are too noisy to fully accept. It's unclear how much compute/samples were required for the hyper parameter selection. In general the results are a lot more noisier for the proposed approaches vs standard PSRO.

**Minor**
- Environment dynamics should be SxA rather than OxA
- Algorithm 4.1 does not exist?
- Fig 1. Is it because of PSRO with Nash. Would a different MSS work differently?
- "proposes MCC as a solution concept": not everyone would know what MCC refers to.

The evidence to be evaluated is:Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies. The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies. We provide a general description of Mixed-Oracles, where the Combine-Respones method is abstract, as Algorithm 2. Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

Player 1's initial strategy \(\pi_{1}^{1}\) mostly plays S, which induces a best response of R. Player 1's second strategy \(\pi_{1}^{2}\) is an approximate best response to R, scoring \(80\%\) against it. The best response to \(\pi_{1}^{2}\) is P.

To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions (Figure 1).

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution. This affords our algorithm the ability to transfer learning about previously encountered opponents.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim based on the provided evidence and review comments:

The claim that "Mixed-Oracles learns separate best-responses (BR) to each pure strategy in a mixture and combines the results from learning to approximate a BR to the mixture" appears to be well-stated and accurately reflects the evidence without exaggeration.

Key supporting evidence:
1. The mathematical description clearly shows that Mixed-Oracles maintains "a collection of BRs to each opponent policy" (Λᵢᵉ)
2. The evidence explicitly states it works by "learning and maintaining a collection of BRs to each opponent policy"
3. The combination method is clearly defined through Q-Mixing equation (1)
4. The review comments acknowledge the clear description of the approach, though noting some limitations regarding Q-mixing dependency

The claim directly describes the core mechanism without overstating capabilities or making unsupported generalizations. It aligns precisely with the technical details provided in the evidence.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_Mixed-Oracles_ learns separate best-responses (BR) to each pure strategy in a mixture and combines the results from learning to approximate a BR to the mixture.

The review comment to be evaluated is:

Summary
The paper proposes two new methods in the Policy-Space Response Oracle framework. These approaches permit to reuse past knowledge in order to reduce the amount of data required for the RL training. The first algorithm Mixed-Oracles transfers the previous iteration of Deep RL, instead of the second one, Mixed-Opponents, transfers existing strategy action-value estimates.

Strengths
The paper proposes two convincing alternatives to reuse previous knowledge in the PSRO framework. The two ideas are based on Q-mixing approach: the first one uses this method to transfer Q-values across epochs, the second one to design a new training objective.
The experiments show that the proposed methods find a good solution using less simulation than the original PSRO framework.

Weakness
The paper is not very novel, since it uses previous approaches (PSRO and Q-mixing) to transfer knowledge for the PSRO framework.
I am not aware of recent works in this framework but could be useful to compare the proposed approaches with P2RO [1].
The Mixed-Opponent section needs a better explanation of the use of Q-mixing as a training objective.

[1]  McAleer, S., Lanier, J., Fox, R., & Baldi, P. (2020). Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. arXiv preprint arXiv:2006.08555.

The evidence to be evaluated is:Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies. The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies. We provide a general description of Mixed-Oracles, where the Combine-Respones method is abstract, as Algorithm 2. Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

Player 1's initial strategy \(\pi_{1}^{1}\) mostly plays S, which induces a best response of R. Player 1's second strategy \(\pi_{1}^{2}\) is an approximate best response to R, scoring \(80\%\) against it. The best response to \(\pi_{1}^{2}\) is P.

To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions (Figure 1).

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution. This affords our algorithm the ability to transfer learning about previously encountered opponents.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate this claim carefully:

1. Summary of evaluation:
The evidence strongly supports the core mechanism described in the claim - Mixed-Oracles does learn separate best responses to pure strategies and combines them. The evidence explicitly states "During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy" and describes combining these responses through Q-Mixing. The claim accurately represents the key aspects without overstating capabilities or results. The review comment acknowledges the method's effectiveness while noting it builds on existing approaches, which aligns with the modest scope of the original claim.

2. <score>1</score>

Justification:
- The claim precisely matches the mechanism described in the evidence without exaggeration
- Evidence directly supports both parts of the claim:
  - Learning separate BRs: "maintaining a collection of BRs to each opponent policy"
  - Combining results: Uses "Q-Mixing as our Combine-Respones" to approximate BR to mixtures
- The claim doesn't overstate performance benefits or make unsupported generalizations
- The review comment, while noting limited novelty, confirms the method works as claimed
- Experimental results validate the approach works but aren't overstated in the claim


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_Mixed-Oracles_ learns separate best-responses (BR) to each pure strategy in a mixture and combines the results from learning to approximate a BR to the mixture.

The review comment to be evaluated is:

##########################################################################
Summary:

The paper provides an interesting approach to speeding up the convergence time of the Policy-Space Response Oracles framework by re-using the Q-functions of past best-responses to transfer knowledge across epochs.
##########################################################################
Reasons for score:

Overall, I am low confidence on my assessment of this paper due to the exposition in the algorithm section being relatively confusing. The experimental results are interesting which suggests that the method has value but there is key missing information on how the best response policies are constructed that make it difficult to assess the paper and lead to my not wanting to recommend its acceptance. I would highly recommend being more detailed in Sec. 3 to allow me to reassess the paper. I would certainly be willing to update my score if the paper was clearer to read.
 ##########################################################################Pros:

1. The experimental results on Leduc Poker are very speedy in terms of time-steps.

2. The idea of reusing the prior Q functions and just mixing them together rather than re-learning all of the policies is very good.

##########################################################################
Cons:

1. The algorithm boxes are so high level that I am struggling to understand how the algorithms work. I would not be able to implement it from reading the paper.

#########################################################################
Things that would improve readability:

- It would be nice in the algorithm boxes to connect Q-mixing to how the best policy is explicitly output. I was not able to understand how Q-mixing was connected to either Algorithm 2 or 3 and subsequently had difficulty following the paper.
- \lambda does not appear to be defined anywhere but appears in the Mixed Oracles algorithm box
- What is the OpponentOracle and the TransferOracle? They are defined in the algorithm boxes but are not clearly defined elsewhere.
- The specific example of RPS in section 3.2 does not provide useful intuition by going through the numerics, it may be more helpful to walk through a more high level description.
- It would probably be useful to move more of the experimental results to an appendix to leave room for the exposition of the algorithms.

The evidence to be evaluated is:Each of the players has a set of policies \(\Pi_{i}\!=\!\left\{\pi_{i}^{0},\ldots,\pi_{i}^{k}\right\}\) from which it may choose. Player \(i\) may select a _pure strategy_\(\pi_{i}\!\in\!\Pi_{i}\), or may randomize play by sampling from a _mixed strategy_\(\sigma_{i}\!\in\!\Delta(\Pi_{i})\).

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs. This method works by learning and maintaining a collection of BRs to each opponent policy \(\Lambda_{i}^{e}\!=\!\left\{\lambda_{1}^{1},\!...,\!\lambda_{i}^{e}\right\}\), where \(\lambda_{i}^{e}\) is the BR to \(\pi_{-i}^{e-1}\). During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies. The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies. We provide a general description of Mixed-Oracles, where the Combine-Respones method is abstract, as Algorithm 2. Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

Player 1's initial strategy \(\pi_{1}^{1}\) mostly plays S, which induces a best response of R. Player 1's second strategy \(\pi_{1}^{2}\) is an approximate best response to R, scoring \(80\%\) against it. The best response to \(\pi_{1}^{2}\) is P.

To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions (Figure 1).

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution. This affords our algorithm the ability to transfer learning about previously encountered opponents.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "_Mixed-Oracles_ learns separate best-responses (BR) to each pure strategy in a mixture and combines the results from learning to approximate a BR to the mixture."

This claim appears to be well-stated based on the evidence provided. The evidence explicitly shows that:
1. Mixed-Oracles maintains "a collection of BRs to each opponent policy"
2. It learns "a BR for the single new opponent policy" in each epoch
3. Uses Combine-Responses function to construct "a BR to the MSS-generated target mixture"
4. Specifically implements Q-Mixing to combine Q-values weighted by opponent likelihood

The reviewer's comments don't contradict the claim's technical accuracy, though they note implementation details are unclear. The claim itself accurately represents the core mechanism without overstating capabilities.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_Mixed-Oppennts_ constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy.

The evidence to be evaluated is:At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In each epoch, each player learns a BR to a mixed profile of opponent policies. This mixture typically involves the newly added strategies (one per player) for this epoch, but may also include strategies from previous epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy. The resulting policy should approximately aggregate the behavior of the policies.

The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

We employ Q-Mixing (Equation 1) as our Combine-Oponents.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Their approach is still seen in PSRO today, where iterative BRs to equilibrium are used to build an empirical game.

PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim and evidence:

The claim states that Mixed-Opponents "constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy."

This claim appears well-stated based on several pieces of evidence:

1. The evidence explicitly states: "it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training"

2. The method is clearly described using Q-Mixing as the Combine-Opponents function to aggregate opponent policies

3. The description matches precisely with what the claim states about constructing a single policy representing an aggregate of the mixed strategy

4. The empirical results support the method's effectiveness

The claim accurately reflects what is described in the evidence without exaggeration or overstatement of capabilities.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_Mixed-Oppennts_ constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy.

The review comment to be evaluated is:

The paper suggests two techniques to improve the calculation of empirically figuring out a Nash equilibrium using an iterative application of best-response dynamics. One method learns the best-response to the previously used strategy. The other uses that technique to model the opponent, and then best-responds to the modeled opponent. The experiments show a faster reaching to NE than without these changes.

The paper is well-written and explained, and is accessible even to researchers not well-versed in ML topics. While, the suggested changes are rather straightforward, they do indeed lead to the expected advance (shorter times to reach NE). I was convinced by further introspection that this is a significant enough contribution, to merit acceptance.

Of course, a more significant change to the algorithm, leading not only to a shorter time but to convergence to better equilibria (in cases where multiple exist) would be far more compelling.

The evidence to be evaluated is:At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In each epoch, each player learns a BR to a mixed profile of opponent policies. This mixture typically involves the newly added strategies (one per player) for this epoch, but may also include strategies from previous epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy. The resulting policy should approximately aggregate the behavior of the policies.

The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

We employ Q-Mixing (Equation 1) as our Combine-Oponents.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Their approach is still seen in PSRO today, where iterative BRs to equilibrium are used to build an empirical game.

PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim based on the provided evidence:

1. The claim states that "Mixed-Opponents constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy."

The evidence strongly supports this claim by:
- Explicitly describing that Mixed-Opponents "combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training"
- Specifying that it uses Q-Mixing as the Combine-Opponents function
- Demonstrating that this method performs effectively, finding better solutions than PSRO in half the simulation timesteps
- Explaining the mathematical framework (Equation 1) for how the combination is performed

The claim is precise and accurately reflects what's described in the evidence without exaggeration. It captures the key aspects of the Mixed-Opponents approach while staying within the bounds of what's demonstrated in the evidence.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_Mixed-Oppennts_ constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy.

The review comment to be evaluated is:

The paper focuses on resolving the computational and sample efficiency challenges with current PSRO style approaches. To this end it proposes two different modifications to the standard PSRO setup: 1) Mixed Oracles, and 2) Mixed Opponents. These approaches allow avoiding resetting learning after each outer loop epoch and reduce the stochasticity of dynamics during training. Thee efficacy is demonstrated on relatively simple games but using Deep RL policies where the proposed approaches are at least on par with standard PSRO approach in terms of final performance while drastically improving the sample efficiency.

Although the paper has quite a few typos and unclear writing in places, it has a fantastic setup with clear description of the problems with standard PSRO it's trying to resolve. The paper points out the clear computational deficiencies with current PSRO style approaches: 1) starting anew everytime, 2) slower exploration of strategy space, 3) stochastic dynamics making learning difficult.

The paper is mostly focused on two-player zero sum games.
Approach heavily dependent on efficacy of Q-mixing approach (although the idea is more general), so currently limited to discrete action problems.
Moreover, since there are no alternatives to Q-mixing, we don't actually get any understanding/intuition for why this works and how important is the performance of "TransferOracle" or "OpponentOracle".

Although section 4.4 suggests that the two stage hyperparameter selection wasn't as important, results in Fig 6b are too noisy to fully accept. It's unclear how much compute/samples were required for the hyper parameter selection. In general the results are a lot more noisier for the proposed approaches vs standard PSRO.

**Minor**
- Environment dynamics should be SxA rather than OxA
- Algorithm 4.1 does not exist?
- Fig 1. Is it because of PSRO with Nash. Would a different MSS work differently?
- "proposes MCC as a solution concept": not everyone would know what MCC refers to.

The evidence to be evaluated is:At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In each epoch, each player learns a BR to a mixed profile of opponent policies. This mixture typically involves the newly added strategies (one per player) for this epoch, but may also include strategies from previous epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy. The resulting policy should approximately aggregate the behavior of the policies.

The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

We employ Q-Mixing (Equation 1) as our Combine-Oponents.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Their approach is still seen in PSRO today, where iterative BRs to equilibrium are used to build an empirical game.

PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim: "Mixed-Opponents constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy."

1. Summary of evaluation:
The claim is well-stated and accurately reflects what the evidence shows. The evidence clearly states that Mixed-Opponents "combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training." It specifically mentions using Q-Mixing as the Combine-Opponents method to aggregate the mixed strategy into a single policy. The evidence fully supports the technical details presented in the claim without exaggeration or misrepresentation. The review comments don't contradict this specific claim, though they note some limitations of Q-mixing in general.

<score>1</score>

Justification:
- The claim directly matches the evidence's description of the Mixed-Opponents approach
- The evidence explicitly states "Mixed-Opponents...modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy"
- The mechanism (Q-Mixing) for combining opponent strategies is clearly specified in the evidence
- No exaggeration of capabilities or results
- The claim stays within the bounds of what is directly supported by the evidence
- While the reviewer notes limitations of Q-mixing, these don't affect the accuracy of this specific claim about what Mixed-Opponents does


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_Mixed-Oppennts_ constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy.

The review comment to be evaluated is:

Summary
The paper proposes two new methods in the Policy-Space Response Oracle framework. These approaches permit to reuse past knowledge in order to reduce the amount of data required for the RL training. The first algorithm Mixed-Oracles transfers the previous iteration of Deep RL, instead of the second one, Mixed-Opponents, transfers existing strategy action-value estimates.

Strengths
The paper proposes two convincing alternatives to reuse previous knowledge in the PSRO framework. The two ideas are based on Q-mixing approach: the first one uses this method to transfer Q-values across epochs, the second one to design a new training objective.
The experiments show that the proposed methods find a good solution using less simulation than the original PSRO framework.

Weakness
The paper is not very novel, since it uses previous approaches (PSRO and Q-mixing) to transfer knowledge for the PSRO framework.
I am not aware of recent works in this framework but could be useful to compare the proposed approaches with P2RO [1].
The Mixed-Opponent section needs a better explanation of the use of Q-mixing as a training objective.

[1]  McAleer, S., Lanier, J., Fox, R., & Baldi, P. (2020). Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. arXiv preprint arXiv:2006.08555.

The evidence to be evaluated is:At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In each epoch, each player learns a BR to a mixed profile of opponent policies. This mixture typically involves the newly added strategies (one per player) for this epoch, but may also include strategies from previous epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy. The resulting policy should approximately aggregate the behavior of the policies.

The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

We employ Q-Mixing (Equation 1) as our Combine-Oponents.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Their approach is still seen in PSRO today, where iterative BRs to equilibrium are used to build an empirical game.

PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim against the provided evidence:

The claim states that Mixed-Opponents "constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy."

Based on the evidence:
- The text explicitly confirms that Mixed-Opponents "combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training"
- It clearly states they use Q-Mixing as their Combine-Opponents function to aggregate the opponent strategies
- The review comment notes this approach uses Q-mixing as a training objective, though requests better explanation
- The experimental results validate that this approach is effective, finding better solutions than PSRO in fewer timesteps

The claim appears to be well-stated (score of 1) as it accurately describes the core mechanism of Mixed-Opponents without exaggeration and is directly supported by multiple pieces of evidence. The description matches precisely what the method does according to the technical details provided.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_Mixed-Oppennts_ constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy.

The review comment to be evaluated is:

##########################################################################
Summary:

The paper provides an interesting approach to speeding up the convergence time of the Policy-Space Response Oracles framework by re-using the Q-functions of past best-responses to transfer knowledge across epochs.
##########################################################################
Reasons for score:

Overall, I am low confidence on my assessment of this paper due to the exposition in the algorithm section being relatively confusing. The experimental results are interesting which suggests that the method has value but there is key missing information on how the best response policies are constructed that make it difficult to assess the paper and lead to my not wanting to recommend its acceptance. I would highly recommend being more detailed in Sec. 3 to allow me to reassess the paper. I would certainly be willing to update my score if the paper was clearer to read.
 ##########################################################################Pros:

1. The experimental results on Leduc Poker are very speedy in terms of time-steps.

2. The idea of reusing the prior Q functions and just mixing them together rather than re-learning all of the policies is very good.

##########################################################################
Cons:

1. The algorithm boxes are so high level that I am struggling to understand how the algorithms work. I would not be able to implement it from reading the paper.

#########################################################################
Things that would improve readability:

- It would be nice in the algorithm boxes to connect Q-mixing to how the best policy is explicitly output. I was not able to understand how Q-mixing was connected to either Algorithm 2 or 3 and subsequently had difficulty following the paper.
- \lambda does not appear to be defined anywhere but appears in the Mixed Oracles algorithm box
- What is the OpponentOracle and the TransferOracle? They are defined in the algorithm boxes but are not clearly defined elsewhere.
- The specific example of RPS in section 3.2 does not provide useful intuition by going through the numerics, it may be more helpful to walk through a more high level description.
- It would probably be useful to move more of the experimental results to an appendix to leave room for the exposition of the algorithms.

The evidence to be evaluated is:At each epoch \(e\) of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile \(\sigma_{-i}^{*,e-1}\) from the currently constructed ENFG: \(\pi_{i}^{e}\in\text{BR}(\sigma_{-i}^{*,e-1})\). These policies are then added to each player's strategy set, \(\tilde{\Pi}_{i}^{e}\!\leftarrow\!\tilde{\Pi}_{i}^{e-1}\!\cup\!\left\{\pi_{i}^{ e}\right\}\), and the new profiles are simulated to expand the ENFG.

In each epoch, each player learns a BR to a mixed profile of opponent policies. This mixture typically involves the newly added strategies (one per player) for this epoch, but may also include strategies from previous epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture. Constructing the new policy is done through a general Combine-Respones function that maps a set of policies and a distribution over the policies into a single policy. The resulting policy should approximately aggregate the behavior of the policies.

The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses.

We employ Q-Mixing (Equation 1) as our Combine-Oponents.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

Their approach is still seen in PSRO today, where iterative BRs to equilibrium are used to build an empirical game.

PSRO highlighted a subclass of EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
1. Analysis Summary:
The claim regarding Mixed-Opponents appears to be well-stated and accurately reflects what is presented in the evidence. The evidence directly states "Mixed-Opponents...modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent." The evidence details how Q-Mixing is used as the Combine-Opponents function to aggregate opponent strategies into a single policy. The reviewer comments mainly focus on clarity issues rather than questioning the accuracy of this specific claim.

2. <score>1</score>

Justification:
- The claim precisely matches the implementation described in the evidence without any exaggeration
- The evidence provides specific technical details about how the aggregation is performed using Q-Mixing
- The claim is supported by multiple sections of the evidence that consistently describe this aspect of Mixed-Opponents
- The reviewer's concerns are primarily about clarity of exposition rather than accuracy of this specific claim
- The evidence demonstrates clear experimental validation showing that this approach is effective, leading to "better solution than PSRO in half the simulation timesteps"
- The claim makes no overreaching statements about performance or capabilities beyond what is directly supported by the evidence


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We introduce two variations of PSRO designed to reduce the amount of simulation required during Deep RL training.

The evidence to be evaluated is:We observe and address two main problems with the standard version of PSRO (Algorithm 1). First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile \(\sigma_{-i}\); the transition dynamics remain the same, and the mixed profile \(\sigma_{-i}\) itself likely contains other-agent policies encountered in previous epochs. This suggests a compelling opportunity to transfer learning across epochs; however, the BR calculation step works by training anew. Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG.

In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent. In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

The _Mixed-Oracles_ algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.

During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS. A BR to the MSS-generated target mixture is then _constructed_ from the collection of BR results for constituent policies in the mixture.

By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Smith et al. (2020) propose _Q-Mixing_ as an approach for constructing policies against any mixture of opponent strategies. The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here. Specifically, Q-Mixing averages the Q-values learned against each opponent policy \(\pi_{-i}\), weighted by their likelihood in the opponent mixture \(\sigma_{-i}\): \[Q_{i}(o_{i},\!a_{i}|\,\sigma_{-i})\!=\!\sum_{\pi_{-i}}\!\psi_{i}(\pi_{-i}|\,o_ {i},\!\sigma_{-i})Q_{i}(o_{i},\!a_{i}|\,\pi_{-i}), \tag{1}\] where \(\psi\) determines the relative likelihood of playing an opponent \(\psi_{i}\colon\mathcal{O}_{i}\rightarrow\Delta(\Pi_{-i})\). In this study, we use Q-Mixing as our Combine-Respones, where \(\psi\) is the prior over the opponent distribution as given by an MSS.

This example motivates our second algorithm: Mixed-Opponents. This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training. We refer to the method for generating a new opponent policy from a mixture of opponents the Combine-Oponents, and it has the same functional form as the Combine-Responses. The generalized Mixed-Opponent algorithm is shown in Algorithm 3. We employ Q-Mixing (Equation 1) as our Combine-Oponents. In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Input: Initial policy sets for all players \(\Pi^{0}\)  Simulate utilities \(\tilde{U}^{\Pi^{0}}\) for each joint \(\pi\!\in\!\Pi^{0}\)  Initialize solutions \(\sigma_{i}^{*,0}\!=\!\text{Uniform}(\Pi_{i}^{0})\) whileepoch e in \(\{1,2,...\}\)do for player \(i\!\in\![[n]]\)do for many episodesdo \(\pi_{-i}\!\sim\!\sigma_{-i}^{*-1}\)  Train \(\pi_{i}^{e}\) over \(\tau\!\sim\!(\pi_{i}^{e}\!,\!\pi_{-i})\) \(\Pi_{i}^{e}\!=\!\Pi_{i}^{-1}\!\cup\!\{\pi_{i}^{e}\}\)  Simulate missing entries in \(\tilde{U}^{\Pi^{e}}\) from \(\Pi^{e}\)  Compute a solution \(\sigma^{*,e}\) from \(\tilde{\Gamma}^{e}\) Output: Current solution \(\sigma_{i}^{*,e}\) for player \(i\)

To address these issues we take a two-step approach (detailed in Section B). First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy. Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters. A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: _pure-hparams_ performed best on-average against each individual test-opponent, and _mix-hparams_ performed best against a uniform mixture of the test-opponents. These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.

The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps? We begin by running both algorithms on the Gathering-Small and Leduc-Poker games. We record regrets over PSRO epochs and cumulative training timesteps. Next, we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?

An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players.

Figure 5(a) shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the time.

To answer this question we run all three algorithms with the same set of hyperparameters, forcing all to adopt the same simulation budget.

Mixed-Oracles and Mixed-Opponents find solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis. In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half the simulation timesteps.

In this study, we investigate PSRO's approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.

We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Next, we introduce Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Input: Figure 5: PSRO compared against Mixed-Opponents on the Leduc-Poker game. SumRegret over ΠPSRO∪ΠEVAL is compared over PSRO epochs (left) and training timesteps (right).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears well-stated. The evidence demonstrates that both PSRO variations (Mixed-Oracles and Mixed-Opponents) successfully reduce simulation requirements during Deep RL training:

1. Mixed-Oracles reduces simulation by reusing learned behaviors from previous epochs and focusing training only on new opponent policies.

2. Mixed-Opponents achieves similar quality solutions in half the time, as shown in Figure 5 where it matches PSRO's performance with significantly fewer timesteps.

The experimental results directly support the claim's scope and magnitude, without overreaching. The paper acknowledges limitations and provides clear empirical evidence for the reduced simulation requirements.

<score>1</score>