outputs of the $\{Q_k\}$ functions, we construct a UCB by adding the empirical standard deviation $\tilde{\sigma}(s_t, a)$ of $\{Q_k(s_t, a)\}_{k=1}^K$ to the empirical mean $\tilde{\mu}(s_t, a)$ of $\{Q_k(s_t, a)\}_{k=1}^K$. The agent chooses the action that maximizes this UCB $$\label{eqn:ucb-action}
a_t \in \operatorname*{argmax}_a \big\{ \tilde{\mu}(s_t, a)+ \lambda \cdot \tilde{\sigma}(s_t, a)\big\},$$ where $\lambda \in \mathbb{R}_+$ is a hyperparameter.

We present Algorithm \[algo:second-improv\], which incorporates the UCB exploration. The hyperparemeter $\lambda$ controls the degrees of exploration. In Section \[section:exp\], we compare the performance of our algorithms on Atari games using a consistent set of parameters.

Value function networks $Q$ with $K$ outputs $\{Q_k\}_{k=1}^K$. Hyperparameter $\lambda$. Let $B$ be a replay buffer storing experience for training. Obtain initial state from environment $s_0$ Pick an action according to $a_t \in \operatorname*{argmax}_a \big\{ \tilde{\mu}(s_t, a)+ \lambda \cdot \tilde{\sigma}(s_t, a)\big\}$ Receive state $s_{t+1}$ and reward $r_t$ from environment, having taken action $a_t$ Add $(s_t, a_t, r_t, s_{t+1})$ to replay buffer $B$ At learning interval, sample random minibatch and update $\{Q_k\}$ according to 

Experiment {#section:exp}
==========

In this section, we conduct experiments to answer the following questions:

1.  does Ensemble Voting, Algorithm \[algo:approx-bayes-q\], improve upon existing algorithms including Double DQN and bootstrapped DQN?

2.  is the proposed UCB 