highest average reward in 28 games, Ensemble Voting in 10 games, and A3C+ in 10 games. Our approach outperforms A3C+.

Finally to understand why and when the proposed methods are superior, we aggregate the comparison results according to four categories: Human Optimal, Score Explicit, Dense Reward, and Sparse Reward. These categories follow the taxonomy in Table 1 of @ostrovski2017count. Out of all games evaluated, 23 games are Human Optimal, 8 are Score Explicit, 8 are Dense Reward, and 5 are Sparse Reward. The comparison results are tabulated in Table \[tab:compare\_each\_category\], where we see $\texttt{ucb exploration}$ achieves top performance in more games than Ensemble Voting, Double DQN, and Bootstrapped DQN in the categories of Human Optimal, Score Explicit, and Dense Reward. In Sparse Reward, both $\texttt{ucb exploration}$ and Ensemble Voting achieve best performance in 2 games out of total of 5. Thus, we conclude that $\texttt{ucb exploration}$ improves prior methods consistently across different game categories within the Arcade Learning Environment.

Conclusion
==========

We proposed a $Q$-ensemble approach to deep $Q$-learning, a computationally practical algorithm inspired by Bayesian reinforcement learning that outperforms Double DQN and bootstrapped DQN, as evaluated on Atari. The key ingredient is the UCB exploration strategy, inspired by bandit 