When agents are randomly choosing their behaviors, it represents the fact that agents have only limited rationality. Hence in the learning process, we apply $\epsilon$-greedy as a way to incorporate exploration for faster convergence as well as to take into account limited rationality of agents.

-   Data efficiency and parameter sharing.

    Keeping separated Q functions for individual agents is not data efficient. An agent may not be able to collect enough samples to properly learn the desired Q function. Due to the homogeneity assumptions we made earlier about agents’ cost functions, it is more data efficient to share the Q function for all agents. Its societal implication is that agents are sharing information and knowledge with each other. Hence, we apply parameter sharing[@gupta2017cooperative] as a way to improve data efficiency as well as to consider information sharing among agents during the learning process.

With the above modifications, the multi-agent Q learning algorithm[@hu2003nash] is summarized below.

-   For every time step $k$, agents choose their actions as: $$u_{i,k} = \left\{\begin{array}{ll}
    \arg\min_u Q(x_{i,k}, m_k, u) & \text{probability }1-\epsilon\\
    \text{random} &\text{probability }\epsilon \end{array}\right. \forall i.$$

-   At the next time step $k+1$, agents observe the new states $x_{i,k+1}$ and receive rewards $l_{i,k}$ for 