Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games. We identify three key challenges that any algorithm needs to master in order to perform well on all games:  processing diverse reward distributions, reasoning over long time horizons, and exploring efficiently.   In this paper, we propose an algorithm that addresses each of these challenges and is able to learn human-level policies on nearly all Atari games. A new transformed Bellman operator allows our algorithm to process rewards of varying densities and scales; an auxiliary temporal consistency loss allows us to train stably using a discount factor of 0.999 (instead of 0.99) extending the effective planning horizon by an order of magnitude; and we ease the exploration problem by using human demonstrations that guide the agent towards rewarding states. When tested on a set of 42 Atari games, our algorithm exceeds the performance  of an average human on 40 games using a common set of hyper parameters. In recent years, significant advances in the field of deep Reinforcement Learning (RL) have led to artificial agents that are able to reach human-level control on a wide array of tasks such as some Atari 2600 games . In many of the Atari games, these agents learn control policies that far exceed the capabilities of an average human player BID4 . However, learning human-level policies consistently across the entire set of games remains an open problem.We argue that an algorithm needs to overcome three key challenges in order to perform well on all Atari games. The first challenge is processing diverse reward distributions. An algorithm must learn stably regardless of reward density and scale. BID12 showed that clipping rewards to the canonical interval [−1, 1] is one way to achieve stability. However, this clipping operation may change the set of optimal policies. For example, the agent no longer differentiates between striking a single pin or all ten pins in BOWLING. Hence, optimizing the unaltered reward signal in a stable manner is crucial to achieving consistent performance across games. The second challenge is reasoning over long time horizons, which means the algorithm should be able to choose actions in anticipation of rewards that might be far away. For example, in MONTEZUMA'S REVENGE, individual rewards might be separated by several hundred time steps. In the standard γ-discounted RL setting, this means the algorithm should be able to handle discount factors close to 1. The third and final challenge is efficient exploration of the MDP. An algorithm that explores efficiently is able to discover long trajectories with a high cumulative reward in a reasonable amount of time even if individual rewards are very sparse. While each problem has been partially addressed in the literature, none of the existing deep RL algorithms have been able to address these three challenges at once.In this paper, we propose a new Deep Q-Network (DQN) BID12 style algorithm that specifically addresses these three challenges. In order to learn stably independent of the reward distribution, we use a transformed Bellman operator that reduces the variance of the action-value function. Learning with the transformed operator allows us to process the unaltered environment rewards regardless of scale and density. We prove that the optimal policy does not change in deterministic MDPs and show that under certain assumptions the operator is a contraction in stochastic MDPs (i.e., the algorithm converges to a fixed point) (see Sec. 3.2) . Our algorithm learns stably even at high discount factors due to an auxiliary temporal consistency (TC) loss. This loss prevents the network from prematurely generalizing to unseen states (Sec. 3.3) allowing us to use a discount factor as high as γ = 0.999 in practice. This extends the effective planning horizon of our algorithm by one order of magnitude when compared to other deep RL approaches on Atari. Finally, we improve the efficiency of DQN's default exploration scheme by combining the distributed experience replay approach of with the Deep Q-learning from Demonstrations (DQfD) algorithm of BID6 . The resulting architecture is a distributed actor-learner system that combines offline expert demonstrations with online agent experiences (Sec. 3.4).We experimentally evaluate our algorithm on a set of 42 games for which we have demonstrations from an expert human player (see Table 6 ). Using the same hyper parameters on all games, our algorithm exceeds the performance of an average human player on 40 games, the expert player on 34 games, and state-of-the-art agents on at least 28 games. Furthermore , we significantly advance the state-of-the-art on sparse reward games. Our algorithm completes the first level of MONTEZUMA'S REVENGE and it achieves a score of 3997 points on PITFALL! without compromising performance on dense reward games and while only using 5 demonstration trajectories. In this paper, we presented a deep Reinforcement Learning (RL) algorithm that achieves human-level performance on a wide variety of MDPs on the Atari 2600 benchmark. It does so by addressing three challenges: handling diverse reward distributions, acting over longer time horizons, and efficiently exploring on sparse reward tasks. We introduce novel approaches for each of these challenges: a transformed Bellman operator, a temporal consistency loss, and a distributed RLED framework for learning from human demonstrations and task reward. Our algorithm exceeds the performance of an average human on 40 out of 42 Atari 2600 games.
Ape-X DQfD = Distributed (many actors + one learner + prioritized replay) DQN with demonstrations optimizing the unclipped 0.999-discounted return on Atari. The paper proposes three extensions (Bellman update, temporal consistency loss, and expert demonstration) to DQN to improve the learning performance on Atari games, achieving outperformance over the state-of-the-art results for Atari games.  This paper proposes a transformed Bellman operator that aims to solve sensitivity to unclipped reward, robustness to the value of the discount factor, and the exploration problem.