$Q_k(\cdot; \theta^-)$ estimates the value $$y_{\tau}^{Q_k}  = r + \gamma \max_a Q^k(s', \operatorname*{argmax}_a Q_k(s', a; \theta_t); \theta^-).$$ In their experiments on Atari games, @osband2016deep set the mask $m_{\tau}=(1, \dots, 1)$ such that all $\{Q_k\}$ are trained with the same samples and their only difference is initialization. Bootstrapped DQN picks one $Q_k$ uniformly at random at the start of an episode and follows the greedy action $a_t = \operatorname*{argmax}_a Q_k(s_t, a)$ for the whole episode.

Approximating Bayesian $\boldsymbol{Q}$-learning with $\boldsymbol{Q}$-Ensembles
================================================================================

Ignoring computational costs, the ideal Bayesian approach to reinforcement learning is to maintain a posterior over the MDP. However, with limited computation and model capacity, it is more tractable to maintain a posterior of the $Q^*$-function. In this section, we first derive a posterior update formula for the $Q^*$-function under full exploration assumption and this formula turns out to depend on the transition Markov chain (). The Bellman equation emerges as an approximation of the log-likelihood. This motivates using a $Q$-ensemble as a particle-based approach to approximate the posterior over $Q^*$-function and an Ensemble Voting algorithm ().

Bayesian update for $\boldsymbol{Q}^*$ {#section:bayes-Q}
--------------------------------------

An MDP is specified by the transition probability $T$ and the reward function $R$. Unlike prior works 