discrepancy among $\{Q_k\}$. Thus, the exploration bonus is monotonous with respect to the residual information in the posterior $\mathsf{H}(\tilde{p}(Q^*|\tau_1, \dots, \tau_N))$. We first compute the Boltzmann distribution for each $Q_k$ $$P_{\mathsf{T}, k} (a|s) = \frac{\exp\big(Q_k(s, a)/\mathsf{T}\big)}{\sum\nolimits_{a'} \exp\big(Q_k(s, a')/\mathsf{T}\big)},$$ where $\mathsf{T} > 0$ is a temperature parameter. Next, calculate the average Boltzmann distribution $$P_{\mathsf{T}, \mathrm{avg}} = \frac{1}{K} \cdot \sum\nolimits_{k=1}^K P_{\mathsf{T}, k} (a|s).$$ The InfoGain exploration bonus is the average KL-divergence from $\{P_{\mathsf{T}, k}\}_{k=1}^K$ to $P_{\mathsf{T}, \mathrm{avg}}$ $$\label{eqn:bonus}
b_{\mathsf{T}}(s) = \frac{1}{K} \cdot \sum\nolimits_{k=1}^K \mathrm{D}_{KL}[ P_{\mathsf{T}, k} || P_{\mathsf{T}, \mathrm{avg}}].$$ The modified reward is $$\label{eqn:modified-reward}
\hat{r}(s, a, s') = r(s, a) + \rho \cdot b_{\mathsf{T}}(s),$$ where $\rho \in \mathbb{R}_+$ is a hyperparameter that controls the degree of exploration.

The exploration bonus $b_{\mathsf{T}}(s_t)$ encourages the agent to explore where $\{Q_k\}$ disagree. The temperature parameter $\mathsf{T}$ controls the sensitivity to discrepancies among $\{Q_k\}$. When $\mathsf{T} \rightarrow +\infty$, $\{P_{\mathsf{T}, k}\}$ converge to the uniform distribution on the action space and $b_{\mathsf{T}}(s) \rightarrow 0$. When $\mathsf{T}$ is small, the differences among $\{Q_k\}$ are magnified and $b_{\mathsf{T}}(s)$ is large.

We display Algorithrim \[algo:infogain\], which incorporates our InfoGain exploration bonus into Algorithm \[algo:second-improv\]. The hyperparameters $\lambda$, $\mathsf{T}$ and $\rho$ vary for each game.

Value function networks $Q$ with $K$ 