Multi-agent reinforcement learning offers a way to study how communication could emerge in communities of agents needing to solve specific problems. In this paper, we study the emergence of communication in the negotiation environment, a semi-cooperative model of agent interaction. We introduce two communication protocols - one grounded in the semantics of the game, and one which is a priori ungrounded.   We show that self-interested agents can use the pre-grounded communication channel to negotiate fairly, but are unable to effectively use the ungrounded, cheap talk channel to do the same.   However, prosocial agents do learn to use cheap talk to find an optimal negotiating strategy, suggesting that cooperation is necessary for language to emerge. We also study communication behaviour in a setting where one agent interacts with agents in a community with different levels of prosociality and show how agent identifiability can aid negotiation. How can communication emerge? A necessary prerequisite is a task that requires coordination between multiple agents to solve, and some communication protocol for the agents to exchange messages through (see a review by BID41 on earlier work on emergent communication as well as recent deep reinforcement learning methods by BID8 and BID39 ). Given these basic requirements, an interesting question to ask is what task structures aid the emergence of communication and how different communication protocols affect task success.In the context of linguistic communication, previous work on this subject has mainly studied the emergence of communication in co-operative games like referential games, variants of the Lewis signaling game BID19 , where messages are used to disambiguate between different possible referents BID11 BID17 BID6 . Human language, though, is not merely a referential tool. Amongst other things, we communicate private information and thoughts, discuss plans, ask questions and tell jokes. Moreover, many human interactions are not fully cooperative, yet we can still successfully use language to communicate in these situations.In this paper, we study communication in the negotiation game (see Figure 1 ), an established model of non-cooperative games in classical game theory BID25 BID26 BID24 BID23 BID35 BID1 BID29 . In this game, agents are asked to establish a mutually acceptable division of a common pool of items while having their own hidden utilities for each of them. Effective communication is crucial in this game, as the agents need to exchange strategic information about their desires, infer their opponent's desires from communication, and balance between the two.Work in classical game theory on negotiation typically uses simple forms of offer / counter-offer bargaining games that do not explicitly address the question of emergent communication (Rubin- Figure 1 : High-level overview of the negotiation environment that we implement. Agent A consistently refers to the agent who goes first. stein, 1982) . Recent work on deep multi-agent reinforcement learning (MARL) has shown great success in teaching agents complex behaviour without a complex environment simulator or demonstration data BID28 BID2 BID40 BID18 . By repeatedly interacting with other agents learning at the same time, agents can gradually bootstrap complex behaviour, including motor skills BID0 and linguistic communication BID17 BID13 .We apply techniques from the MARL literature and train agents to negotiate using task success as the only supervision signal. 1 We show that, when communicating via a task-specific communication channel with inherent semantics, selfish agents can learn to negotiate fairly and divide up the item pool to the agents' mutual satisfaction. However , when communicating via cheap talk BID4 BID7 , a task-independent communication channel consisting of sequences of arbitrary symbols similar to language, selfish agents fail to exhibit negotiating behaviour at all. On the other hand, we show that cheap talk can facilitate effective negotiation in prosocial agents, which take the other agent's reward into consideration, providing experimental evidence that cooperation is necessary for language emergence BID27 .The above results are obtained from paired agents interacting exclusively with each other. In more realistic multi-agent scenarios, agents may interact with many other agents within a society. In these cases, cheap talk can have a significant effect on the evolutionary dynamics of the population BID32 ) as well as the equilibria, stability, and basins of attractions BID38 . Furthermore, it is well-known that, unless trained in a diverse environment, agents overfit to the their specific opponent or teammate . Inspired by these considerations , we perform experiments where agents interact with many agents having different prosociality levels, and find that being able to identify and model other agents' beliefs aids the negotiation success. This is consistent with experiments using models based on Theory of Mind: boundedly rational agents can collectively benefit by making inferences about the sophistication levels and beliefs of their opponents, and there is evidence that this occurs in human behavior BID44 2 GAME SETTING We showed that by communicating through a verifiable and binding communication channel, selfinterested agents can learn to negotiate fairly by reinforcement learning, using only task success as the reward signal. Moreover, cheap talk facilitated negotiation in prosocial but not in self-interested agents, corroborating theoretical results from the game theory literature BID4 ). An interesting future direction of research would be to investigate whether cheap talk can be made to emerge out of self-interested agents interacting. Recent encouraging results by BID3 show that communication can help agents cooperate. However, their signalling mechanism is heavily engineered: the speech acts are predefined and the consequences of the speech acts on the observed behaviour are deterministically hard-coded. It would be interesting to see whether a learning algorithm, such as BID9 , can discover the same result.A related paper from BID20 takes a top-down approach to learning to negotiate by leveraging dialogue data. We demonstrated a bottom up alternative towards learning communicative behaviours directly from interaction with peers. This opens up the exciting possibility of learning domain-specific reasoning capabilities from interaction, while having a general-purpose language layer at the top producing natural language.A ADDITIONAL FIGURES AND TABLES Table 7 : Joint reward success and average number of turns taken for paired agents negotiating when allowed the full 10 turns, varying the agent reward scheme and communication channel. The results are averaged across 20 seeds, with 128 games per seed. We also report the standard deviation as the ± number and the quartiles.
We teach agents to negotiate using only reinforcement learning; selfish agents can do so, but only using a trustworthy communication channel, and prosocial agents can negotiate using cheap talk. The authors describe a variant of the negotation game with the consideration of a secondary communication channel for cheap talk, finding that the secondary channel improves negotation outcomes. This paper explores how agents can learn to communicate to solve a negotiation task and find that prosocial agents are able to learn to ground symbols using RL, but self-interested agents are not. Examines problems of how agents can use communication to maximise their rewards in a simple negotiation game.