To gain high rewards in muti-agent scenes, it is sometimes necessary to understand other agents and make corresponding optimal decisions. We can solve these tasks by first building models for other agents and then finding the optimal policy with these models. To get an accurate model, many observations are needed and this can be sample-inefficient. What's more, the learned model and policy can overfit to current agents and cannot generalize if the other agents are replaced by new agents. In many practical situations, each agent we face can be considered as a sample from a population with a fixed but unknown distribution. Thus we can treat the task against some specific agents as a task sampled from a task distribution. We apply meta-learning method to build models and learn policies. Therefore when new agents come, we can adapt to them efficiently. Experiments on grid games show that our method can quickly get high rewards. Applying Reinforcement Learning (RL) to multi-agent scenes requires carefully consideration about the influence of other agents. We cannot simply treat other agents as part of the environment and apply independent RL methods BID8 if the actions of them has impact on the payoff of the agent to be trained. For example, consider the two-person ultimatum bargaining game, where two players take part in. One player propose a deal to split a fixed amount of money for them two and the other player decides to accept it or not. If the second player accepts the proposal, they split the money, but if the proposal is refused, they both get zero. Experimental results BID4 show that in actual life, the second player makes the decision according to whether he or she judge the final result fair, rather than makes the obvious rational decision. Thus, the first player needs to predict how the second player will react so as to make the proposal acceptable.In order to exploit the other agents and find the corresponding optimal policy, we need to understand these agents. Here in this paper, we call all the other agents "opponents" to distinguish our agent from them, even if they may have cooperative relationship with our agent. For simplicity, we only consider tasks with only one opponent. Extension to tasks with more opponents is straightforward. A general way to exploit an opponent is to build a model for it from observations. This model can characterize any needed feature of the opponent, such as next action or the final goal. Such a model can make predictions for the opponent and thus turns the two-agent task into a simple-agent decision making problem. Then we can apply various RL methods to solve this problem.It is necessary that we need to have an accurate model for the opponent to help make decision. Previous works BID6 BID12 propose some methods to model the opponent. Generally, it requires many observations to get a precise model for the opponent. This may cost many iterations to act with the opponent. What's more, even if we can precisely model the opponent, there exists a main drawback of above process that the performance of the learned policy has no guarantee for any other opponent. Things are even worse if opponents have their private types which are unknown for us. New opponents with different types can have different policies or even different payoffs. Therefore, it seems that when a new opponent came, we have to learn a policy from the beginning. In some practical situations, the whole opponents follow a distributions over all these possible types. Let's come back to the ultimatum bargaining game. BID0 shows that people with different ethnicity may have different standards for fairness. Thus if we assume the type for player 2 to be its judgment for fairness, there can be a distribution for types dependent on the ethnic distribution. Given that opponents follows a distribution, it is possible that we can employ some given opponents to help us speed up the process of opponent modeling and policy improving for the current opponent.If we consider the policy learning against a specific opponent as a task, our goal can be considered as training a policy on various tasks so that it can efficiently adapt to a good policy on a new task with few training samples. This is exactly a meta-learning problem. We employ Model-Agnostic MetaLearning (MAML) BID1 to conduct meta-learning. BID11 applied meta-learning to understand opponents, but this work doesn't address the policy improvement for the agent to be trained. We apply meta-learning to opponent modeling and policy learning separately while training the two meta-learners jointly. Then we use the meta-learners to initialize the model and policy for the new opponent. Experimental results show that the agent can adapt to the new opponent with a small number of interactions with the opponent. In the face of other agents, it is beneficial to build models for opponents and find a corresponding good policy. This method can be sample-inefficient since it costs many observations to build models and learn a policy then. We propose a method that can employ the information learned from experiences with other opponents to speed up the learning process for the current opponents. This method is suitable for many practical situations where the opponent population has a relative stable distribution over their policies. We apply meta-learning to jointly train the opponent modeling and policy improving process. Experimental results show that our method can be sample-efficient.
Our work applies meta-learning to multi-agent Reinforcement Learning to help our agent efficiently adapted to new coming opponents. This paper focuses on fast adaptation to new behaviour of the other agents of the environment using a method based on MAML The paper presents an approach to multi-agent learning based on the framework of model-agnostic meta learning for the task of opponent modeling for multi-agent RL.