Title: Deep Reinforcement learning on Adaptive Pairwise Critic and Asymptotic Actor. Abstract: Maximum entropy deep reinforcement learning has displayed great potential on a range of challenging continuous tasks. The maximum entropy is able to encourage policy exploration, however, it has a tradeoff between the efficiency and stability, especially when employed on large-scale tasks with high state and action dimensionality.
Sometimes the temperature hyperparameter of maximum entropy term is limited to remain stable at the cost of slower and lower convergence.
Besides, the function approximation errors existing in actor-critic learning are known to induce estimation errors and suboptimal policies.
In this paper, we propose an algorithm based on adaptive pairwise critics, and adaptive asymptotic maximum entropy combined.
Specifically, we add a trainable state-dependent weight factor to build an adaptive pairwise target Q-value to serve as the surrogate policy objective.
Then we adopt a state-dependent adaptive temperature to smooth the entropy policy exploration, which introduces an asymptotic maximum entropy.
The adaptive pairwise critics can effectively improve the value estimation, preventing overestimation or underestimation errors. Meanwhile, the adaptive asymptotic entropy can adapt to the tradeoff between efficiency and stability, which provides more exploration and flexibility.
We evaluate our method on a set of Gym tasks, and the results show that the proposed algorithms have better performance than several baselines on continuous control.