Generalizing soft actor-critic algorithms to discrete action spaces
ATARI is a suite of video games used by reinforcement learning (RL)
researchers to test the effectiveness of the learning algorithm. Receiving only
the raw pixels and the game score, the agent learns to develop sophisticated
strategies, even to the comparable level of a professional human games tester.
Ideally, we also want an agent requiring very few interactions with the
environment. Previous competitive model-free algorithms for the task use the
valued-based Rainbow algorithm without any policy head. In this paper, we
change it by proposing a practical discrete variant of the soft actor-critic
(SAC) algorithm. The new variant enables off-policy learning using policy heads
for discrete domains. By incorporating it into the advanced Rainbow variant,
i.e., the ``bigger, better, faster'' (BBF), the resulting SAC-BBF improves the
previous state-of-the-art interquartile mean (IQM) from 1.045 to 1.088, and it
achieves these results using only replay ratio (RR) 2. By using lower RR 2, the
training time of SAC-BBF is strictly one-third of the time required for BBF to
achieve an IQM of 1.045 using RR 8. As a value of IQM greater than one
indicates super-human performance, SAC-BBF is also the only model-free
algorithm with a super-human level using only RR 2. The code is publicly
available on GitHub at https://github.com/lezhang-thu/bigger-better-faster-SAC.
Le Zhang Yong Gu Xin Zhao Yanshuo Zhang Shu Zhao Yifei Jin Xinxin Wu