Beyond The Rainbow: High Performance Deep Reinforcement Learning On A
  Desktop PC
Rainbow Deep Q-Network (DQN) demonstrated combining multiple independent
enhancements could significantly boost a reinforcement learning (RL) agent's
performance. In this paper, we present "Beyond The Rainbow" (BTR), a novel
algorithm that integrates six improvements from across the RL literature to
Rainbow DQN, establishing a new state-of-the-art for RL using a desktop PC,
with a human-normalized interquartile mean (IQM) of 7.4 on atari-60. Beyond
Atari, we demonstrate BTR's capability to handle complex 3D games, successfully
training agents to play Super Mario Galaxy, Mario Kart, and Mortal Kombat with
minimal algorithmic changes. Designing BTR with computational efficiency in
mind, agents can be trained using a desktop PC on 200 million Atari frames
within 12 hours. Additionally, we conduct detailed ablation studies of each
component, analzying the performance and impact using numerous measures.
Tyler Clark Mark Towers Christine Evers Jonathon Hare