Title: Impulse Control Arbitration for A Dual System of Exploitation and Exploration. Abstract: Efficient reinforcement learning (RL) involves a trade-off between "exploitative" actions that maximise expected reward and ``explorative" ones that lead to the visitation of "novel" states. To encourage exploration, existing methods proposed methods such as injecting stochasticity into action selection, implicit regularisation, and additive synthetic reward. However, these techniques do not necessarily offer entirely systematic approaches making this trade-off. Here we introduce SElective Reinforcement EXploration (SEREX), a plug-and-play framework that casts the exploration-exploitation trade-off as a game between an RL agent--- exploiter, which purely exploits task-dependent rewards, and another RL agent--- switcher, which chooses at which states to activate a pure exploration policy that is trained to minimise system uncertainty and override Exploiter. Using a form of policies known as impulse control, switcher is able to determine the best set of states to switch to the exploration policy while Exploiter is free to execute its actions everywhere else. We prove that SEREX converges quickly and induces a natural schedule towards pure exploitation. Through extensive empirical studies in both discrete and continuous control benchmarks, we show that with minimal modification, SEREX can be readily combined with existing RL algorithms and yields significant improvement in performance.