curves of all algorithms across Atari games. Overall, Ensemble Voting, Algorithm \[algo:approx-bayes-q\], outperforms both Double DQN and bootstrapped DQN. With exploration, $\texttt{ucb exploration}$ improves further by outperforming Ensemble Voting.

In Appendix \[app:table\], we tabulate detailed results that compare our algorithms, Ensemble Voting and $\texttt{ucb exploration}$, against prior methods. In Table \[tab:my\_label\], we tabulate the maximal mean reward in $100$ consecutive episodes for Ensemble Voting, $\texttt{ucb exploration}$, bootstrapped DQN and Double DQN. Without exploration, Ensemble Voting already achieves higher maximal mean reward than both Double DQN and bootstrapped DQN in a majority of Atari games. $\texttt{ucb exploration}$ achieves the highest maximal mean reward among these four algorithms in 30 games out of the total 49 games evaluated. Figure \[fig:all\] displays the learning curves of these five algorithms on a set of six Atari games. Ensemble Voting outperforms Double DQN and bootstrapped DQN. $\texttt{ucb exploration}$ outperforms Ensemble Voting.

In Table \[tab:compare\_with\_bellemare\], we compare our proposed methods with the count-based exploration method A3C+ of @bellemare2016unifying based on their published results of A3C+ trained with 200 million frames. We point out that even though our methods were trained with only 40 million frames, much less than A3C+’s 200 million frames, UCB exploration achieves the 