exploration strategy of Algorithm \[algo:second-improv\] effective in improving learning compared to Algorithm \[algo:approx-bayes-q\]?

3.  how does UCB exploration compare with prior exploration methods such as the count-based exploration method of @bellemare2016unifying?

We evaluate the algorithms on each Atari game of the Arcade Learning Environment (@bellemare2013arcade). We use the multi-head neural net architecture of @osband2016deep. We fix the common hyperparameters of all algorithms based on a well-tuned double DQN implementation, which uses the Adam optimizer (@kingma2014adam), different learning rate and exploration schedules compared to @mnih2015human. Appendix \[app:param\] tabulates the hyperparameters. The number of $\{Q_k\}$ functions is $K=10$. Experiments are conducted on the OpenAI Gym platform (@brockman2016openai) and trained with $40$ million frames and $2$ trials on each game.

We take the following directions to evaluate the performance of our algorithms:

1.  we compare Algorithm \[algo:approx-bayes-q\] against Double DQN and bootstrapped DQN,

2.  we isolate the impact of UCB exploration by comparing Algorithm \[algo:second-improv\] with $\lambda=0.1$, denoted as $\texttt{ucb exploration}$, against Algorithm \[algo:approx-bayes-q\].

3.  we compare Algorithm \[algo:approx-bayes-q\] and Algorithm \[algo:second-improv\] with the count-based exploration method of @bellemare2016unifying.

4.  we aggregate the comparison according to different categories of games, to understand when our methods are suprior.

Figure \[fig:normalized\] compares the normalized learning 