outputs $\{Q_k\}_{k=1}^K$. Hyperparameters $\mathsf{T}, \lambda$, and $\rho$. Let $B$ be a replay buffer storing experience for training. Obtain initial state from environment $s_0$ Pick an action according to $a_t \in \operatorname*{argmax}_a \big\{ \tilde{\mu}(s_t, a)+ \lambda \cdot \tilde{\sigma}(s_t, a)\big\}$ Receive state $s_{t+1}$ and reward $r_t$ from environment, having taken action $a_t$ Calculate exploration bonus $b_{\mathsf{T}}(s_t)$ according to Add $(s_t, a_t, r_t + \rho \cdot b_{\mathsf{T}}(s_t), s_{t+1})$ to replay buffer $B$ At learning interval, sample random minibatch and update $\{Q_k\}$

Performance of UCB+InfoGain exploration
---------------------------------------

We demonstrate the performance of the combined UCB+InfoGain exploration in Figure \[fig:normalized\_with\_infogain\] and Figure \[fig:normalized\_with\_infogain\]. We augment the previous figures in Section \[section:exp\] with the performance of $\texttt{ucb+infogain exploration}$, where we set $\lambda=0.1, \rho=1$, and $\mathsf{T}=1$ in Algorithm \[algo:infogain\].

Figure \[fig:normalized\_with\_infogain\] shows that combining UCB and InfoGain exploration does not lead to uniform improvement in the normalized learning curve.

At the individual game level, Figure \[fig:normalized\_with\_infogain\] shows that the impact of InfoGain exploration varies. UCB exploration achieves sufficient exploration in games including Demon Attack and Kangaroo and Riverraid, while InfoGain exploration further improves learning on Enduro, Seaquest, and Up N Down. The effect of InfoGain exploration depends on the choice of the temperature $\mathsf{T}$. The optimal temperature 