In this section, we also studied an “InfoGain” exploration bonus, which encourages agents to gain information about the $Q^*$-function and examine its effectiveness. We found it had some benefits on top of Ensemble Voting, but no uniform additional benefits once already using Q-ensembles on top of Double DQN. We describe the approach and our experimental findings here.

Similar to @sun2011planning, we define the information gain from observing an additional transition $\tau_n$ as $$H_{\tau_t| \tau_1, \dots, \tau_{n-1}} = D_{KL}(\tilde{p}(Q^*|\tau_1, \dots, \tau_n) || \tilde{p}(Q^*|\tau_1, \dots, \tau_{n-1}))$$ where $\tilde{p}(Q^* |\tau_1, \dots, \tau_n)$ is the posterior distribution of $Q^*$ after observing a sequence of transitions $(\tau_1, \dots, \tau_n)$. The total information gain is $$\label{eqn:info-gain}
H_{\tau_1, \dots, \tau_N} = \sum\nolimits_{n=1}^N H_{\tau_n| \tau_1, \dots, \tau_{n-1}}.$$ Our Ensemble Voting, Algorithm \[algo:approx-bayes-q\], does not maintain the posterior $\tilde{p}$, thus we cannot calculate  explicitly. Instead, inspired by @lakshminarayanan2016simple, we define an InfoGain exploration bonus that measures the disagreement among $\{Q_k\}$. Note that $$H_{\tau_1, \dots, \tau_N} + \mathsf{H}(\tilde{p}(Q^*|\tau_1, \dots, \tau_N)) = \mathsf{H}(p(Q^*)),$$ where $\mathsf{H}(\cdot)$ is the entropy. If $H_{\tau_1, \dots, \tau_N}$ is small, then the posterior distribution has high entropy and high residual information. Since $\{Q_k\}$ are approximate samples from the posterior, high entropy of the posterior leads to large 