We propose Episodic Backward Update - a new algorithm to boost the performance of a deep reinforcement learning agent by fast reward propagation. In contrast to the conventional use of the replay memory with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state into its previous states. Our computationally efficient recursive algorithm allows sparse and delayed rewards to propagate effectively throughout the sampled episode. We evaluate our algorithm on 2D MNIST Maze Environment and 49 games of the Atari 2600 Environment and show that our agent improves sample efficiency with a competitive computational cost.
We propose Episodic Backward Update, a novel deep reinforcement learning algorithm which samples transitions episode by episode and updates values recursively in a backward manner to achieve fast and stable learning. Proposes a new DQN where the targets are computed on a full episode by a backward update (end to start) for faster propagation of rewards by the episode end. The authors propose to modify the DQN algorithm by applying the max Bellman operator recursively on a trajectory with some decay to prevent accumulating errors with the nested max. In deep-Q networks, update Q values starting from the end of the episode in order to facilitate quick propagation of rewards back along the episode.