and reward $r_t$ from the environment Add $(s_t, a_t, r_t, s_{t+1})$ to replay buffer $B$ At learning interval, sample random minibatch and update $\{Q_k\}$

UCB Exploration Strategy Using $\boldsymbol{Q}$-Ensembles
=========================================================

In this section, we propose optimism-based exploration by adapting the UCB algorithms (@auer2002finite [@audibert2009exploration]) from the bandit setting. The UCB algorithms maintain an upper-confidence bound for each arm, such that the expected reward from pulling each arm is smaller than this bound with high probability. At every time step, the agent optimistically chooses the arm with the highest UCB. @auer2002finite constructed the UCB based on empirical reward and the number of times each arm is chosen. @audibert2009exploration incorporated the empirical variance of each arm’s reward into the UCB, such that at time step $t$, an arm $A_t$ is pulled according to $$A_t = \operatorname*{argmax}_{i}\Big\{ \hat{r}_{i, t} + c_1 \cdot \sqrt{\frac{\hat{V}_{i,t} \log(t) }{n_{i,t}}} + c_2 \cdot \frac{\log(t)}{n_{i, t}} \Big\}$$ where $\hat{r}_{i, t}$ and $\hat{V}_{i,t}$ are the empirical reward and variance of arm $i$ at time $t$, $n_{i, t}$ is the number of times arm $i$ has been pulled up to time $t$, and $c_1, c_2$ are positive constants.

We extend the intuition of UCB algorithms to the RL setting. Using the 