We describe a simple scheme that allows an agent to learn about its environment in an unsupervised manner. Our scheme pits two versions of the same agent, Alice and Bob, against one another. Alice proposes a task for Bob to complete; and then Bob attempts to complete the task.   In this work we will focus on two kinds of environments: (nearly) reversible environments and environments that can be reset. Alice will "propose" the task by doing a sequence of actions and then Bob must undo or repeat them, respectively.   Via an appropriate reward structure, Alice and Bob automatically generate a curriculum of exploration, enabling unsupervised training of the agent. When Bob is deployed on an RL task within the environment, this unsupervised training reduces the number of supervised episodes needed to learn, and in some cases converges to a higher reward. Model-free approaches to reinforcement learning are sample inefficient, typically requiring a huge number of episodes to learn a satisfactory policy. The lack of an explicit environment model means the agent must learn the rules of the environment from scratch at the same time as it tries to understand which trajectories lead to rewards. In environments where reward is sparse, only a small fraction of the agents' experience is directly used to update the policy, contributing to the inefficiency.In this paper we introduce a novel form of unsupervised training for an agent that enables exploration and learning about the environment without any external reward that incentivizes the agents to learn how to transition between states as efficiently as possible. We demonstrate that this unsupervised training allows the agent to learn new tasks within the environment quickly. In this work we described a novel method for intrinsically motivated learning which we call asymmetric self-play. Despite the method's conceptual simplicity, we have seen that it can be effective in both discrete and continuous input settings with function approximation, for encouraging ex- ploration and automatically generating curriculums. On the challenging benchmarks we consider, our approach is at least as good as state-of-the-art RL methods that incorporate an incentive for exploration, despite being based on very different principles. Furthermore, it is possible show theoretically that in simple environments, using asymmetric self-play with reward functions from FORMULA0 and FORMULA1 , optimal agents can transit between any pair of reachable states as efficiently as possible. Code for our approach can be found at (link removed for anonymity). A PSEUDO CODE Algorithm 1 and 2 are the pseudo codes for training an agent on self-play and target task episodes.Algorithm 1 Pseudo code for training an agent on a self-play episode DISPLAYFORM0
Unsupervised learning for reinforcement learning using an automatic curriculum of self-play A new formulation for exploring the environment in an unsupervised way to aid a specific task later, where one agent proposes increasingly difficult tasks and the learning agent tries to accomplish them. A self-play model where one agent learns to propose tasks that are easy for them but difficult for an opponent, creating a moving target of self-play objectives and learning curriculum. 