We study the role of intrinsic motivation as an exploration bias for reinforcement learning in sparse-reward synergistic tasks, which are tasks where multiple agents must work together to achieve a goal they could not individually. Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own. Thus, we propose to incentivize agents to take (joint) actions whose effects cannot be predicted via a composition of the predicted effect for each individual agent. We study two instantiations of this idea, one based on the true states encountered, and another based on a dynamics model trained concurrently with the policy. While the former is simpler, the latter has the benefit of being analytically differentiable with respect to the action taken. We validate our approach in robotic bimanual manipulation tasks with sparse rewards; we find that our approach yields more efficient learning than both 1) training with only the sparse reward and 2) using the typical surprise-based formulation of intrinsic motivation, which does not bias toward synergistic behavior. Videos are available on the project webpage: https://sites.google.com/view/iclr2020-synergistic. Consider a multi-agent environment such as a team of robots working together to play soccer. It is critical for a joint policy within such an environment to produce synergistic behavior, allowing multiple agents to work together to achieve a goal which they could not achieve individually. How should agents learn such synergistic behavior efficiently? A naive strategy would be to learn policies jointly and hope that synergistic behavior emerges. However, learning policies from sparse, binary rewards is very challenging -exploration is a huge bottleneck when positive reinforcement is infrequent and rare. In sparse-reward multi-agent environments where synergistic behavior is critical, exploration is an even bigger issue due to the much larger action space. A common approach for handling the exploration bottleneck in reinforcement learning is to shape the reward using intrinsic motivation, as was first proposed by Schmidhuber (1991) . This has been shown to yield improved performance across a variety of domains, such as robotic control tasks (Oudeyer et al., 2007) and Atari games (Bellemare et al., 2016; Pathak et al., 2017) . Typically, intrinsic motivation is formulated as the agent's prediction error regarding some aspects of the world; shaping the reward with such an error term incentivizes the agent to take actions that "surprise it," and is intuitively a useful heuristic for exploration. But is this a good strategy for encouraging synergistic behavior in multi-agent settings? Although synergistic behavior may be difficult to predict, it could be equally difficult to predict the effects of certain single-agent behaviors; this formulation of intrinsic motivation as "surprise" does not specifically favor the emergence of synergy. In this paper, we study an alternative strategy for employing intrinsic motivation to encourage synergistic behavior in multi-agent tasks. Our method is based on the simple insight that synergistic behavior leads to effects which would not be achieved if the individual agents were acting alone. So, we propose to reward agents for joint actions that lead to different results compared to if those same actions were done by the agents individually, in a sequential composition. For instance, consider the task of twisting open a water bottle, which requires two hands (agents): one to hold the base in place, and another to twist the cap. Only holding the base in place would not effect any change in Figure 1 : An overview of our approach to incentivizing synergistic behavior via intrinsic motivation. A heavy red bar (requiring two arms to lift) rests on a table, and the policy π θ suggests for arms A and B to lift the bar from opposite ends. A composition of pretrained single-agent forward models, f A and f B , predicts the resulting state to be one where the bar is only partially lifted, since neither f A nor f B has ever encountered states where the bar is lifted during training. A forward model trained on the complete two-agent environment, f joint , correctly predicts that the bar is fully lifted, very different from the compositional prediction. We train π θ to prefer actions such as these, as a way to bias toward synergistic behavior. Note that differentiating this intrinsic reward with respect to the action taken does not require differentiating through the environment. the bottle's pose, while twisting the cap without holding the bottle in place would cause the entire bottle to twist, rather than just the cap. Here, holding with one hand and subsequently twisting with the other would not open the bottle, but holding and twisting concurrently would. Based on this intuition, we propose a formulation for intrinsic motivation that leverages the difference between the true effect of an action and the composition of individual-agent predicted effects. We then present a second formulation that instead uses the discrepancy of predictions between a joint and a compositional prediction model. While the latter formulation requires training a forward model alongside learning the control strategy, it has the benefit of being analytically differentiable with respect to the action taken. We later show that this can be leveraged within the policy gradient framework, in order to obtain improved sample complexity over using the policy gradient as-is. As our experimental point of focus, we study five simulated robotic tasks: four bimanual manipulation (bottle opening, ball pickup, corkscrew rotating, and bar pickup) and multi-agent locomotion (ant push). All tasks have sparse rewards: 1 if the goal is achieved and 0 otherwise. These tasks were chosen both because they require synergistic behavior, and because they represent challenging control problems for modern state-of-the-art deep reinforcement learning algorithms (Levine et al., 2016; Lillicrap et al., 2015; Gu et al., 2017; Mnih et al., 2016; . Across all tasks, we find that shaping the reward via our formulation of intrinsic motivation yields more efficient learning than both 1) training with only the sparse reward signal and 2) shaping the reward via the more standard single-agent formulation of intrinsic motivation as "surprise," which does not explicitly encourage synergistic behavior. We view this work as a step toward general-purpose synergistic multi-agent reinforcement learning. 2) Synergistic intrinsic rewards perform better than non-synergistic intrinsic rewards. Policies that use our synergistic intrinsic rewards also work better than the Non-synergistic surprise baseline. This is primarily because the baseline policies learn to exploit the joint model rather than to behave synergistically. This also explains why Non-synergistic surprise used together with extrinsic reward hurts task performance (green vs. red curve in Figure 3 ). Past experiments with such surprise models have largely been limited to games, where progress is correlated with continued exploration (Burda et al., 2018) ; solving robotic tasks often involves more than just surprise-driven exploration. Figure 4 (top) gives additional results showing that our method's competitive advantage over this baseline persists even if we allow the baseline additional interactions to pretrain the joint prediction model f joint without using any extrinsic reward (similar to our method's pretraining for f composed ). 3) Analytical gradients boost sample efficiency. In going from r intrinsic 1 (compositional prediction error) to r intrinsic 2 (prediction disparity), we changed two things: 1) the reward function and 2) how it is optimized (we used Equation 1 to leverage the partial differentiability of r intrinsic 2 ). We conduct an ablation to disentangle the impact of these two changes. Figure 4 (bottom) presents learning curves for using r intrinsic 2 without analytical gradients, situated in comparison to the previously shown results. When we factor out the difference due to optimization and compare r requires training an extra model f joint concurrently with the policy, which at best could match the trues env . Leveraging the analytical gradients, though, affords r intrinsic 2 more sample-efficient optimization (brown vs. purple curve), making it a better overall choice. We have also tried using our formulation of intrinsic motivation without extrinsic reward (λ = 0); qualitatively, the agents learn to act synergistically, but in ways that do not solve the "task," which is sensible since the task is unknown to the agents. See the project webpage for videos of these results. Furthermore, in Appendix D we provide a plot of policy performance versus various settings of λ. In this work, we presented a formulation of intrinsic motivation that encourages synergistic behavior, and allows efficiently learning sparse-reward tasks such as bimanual manipulation and multi-agent locomotion. We observed significant benefits compared to non-synergistic forms of intrinsic motivation. Our formulation relied on encouraging actions whose effects would not be achieved by individual agents acting in isolation. It would be beneficial to extend this notion further, and explicitly encourage action sequences, not just individual actions, whose effects would not be achieved by individual agents. Furthermore, while our intrinsic reward encouraged synergistic behavior in the single policy being learned, it would be interesting to extend it to learn a diverse set of policies, and thereby discover a broad set of synergistic skills over the course of training. Finally, it would be good to extend the domains to involve more complicated object types, such as asymmetric or deformable ones; especially for deformable objects, it may be important to engineer better state representations since these objects do not have a natural notion of 6D pose.
We propose a formulation of intrinsic motivation that is suitable as an exploration bias in multi-agent sparse-reward synergistic tasks, by encouraging agents to affect the world in ways that would not be achieved if they were acting individually. The paper focuses on using intrinsic motivation to improve the exploration process of reinforcement learning agents in tasks that require multi-agent to achieve.