$Q_i^j(a,b)$ represents agent $i$’s Q-value when agent $i$ and $j$ select action $a$ and $b$ respectively. To determine the relative values of their individual actions, agent $i$ assumes that each other agent $j$ will choose actions in accordance with agent $i$’s empirical distribution over agent $j$’s action. In general, agent $i$ assesses the expected value of each action $a$ as follows: $$EV(a) = \sum_{m \in A_j} Q_i^j(a,m) \ p_i^j(m),$$ where $p_i^j(m)$ is agent $i$’s estimation for the probability of agent $j$ choosing action $m$.

-   $\emph{Joint-Action WoLF-PHC (JA-WoLF)}$. WoLF [@bowling2001rational] extends Q-learning and learns mixed strategies with the idea of quickly adapting when losing but being cautious when winning. Specifically we modify WoLF to a Joint-Action version as agents can observe others’ actions in our environments. We replace $Q(s,a)$ with $Q(s,\vec{a})$ and each agent updates its Q-values for each joint-action. In addition, to determine ’win’ or ’lose’, we need to keep the frequency distribution over opponents’ actions., and agent $i$ adjust its learning rate $\delta$ against agent $j$ as follows: $$\delta =
        \left\{
                     \begin{array}{lr}
                     \delta_w & \text{if} \ \sum_{a \in A_i}\pi_i^j(a) \sum_{m \in A_j}Q_i^j(a,m)p_i^j(m) \\
                 & \ge \sum_{a \in A_i}\bar \pi_i^j(a) \sum_{m \in A_j}Q_i^j(a,m)p_i^j(m), \\
                     \delta_l
                 & \text{otherwise.}
                     \end{array}
        \right.$$ 