challenge, we approximate the posterior update by reusing the one-sample next state $s'$ from $\tau$ such that $$Q_{k, i+1} \leftarrow \operatorname*{argmin}_{Q_{k, i+1}} \big[ Q_{k, i+1}(s, a) -\big( r + \gamma \cdot \max_{a'} Q_{k, i+1}(s', a') \big) \big]^2.$$ Instead of updating the posterior after each transition, we use an experience replay buffer $B$ to store observed transitions and sample a minibatch $B_{\mathrm{mini}}$ of transitions $(s, a, r, s')$ for each update. In this case, the batched update of each $Q_{k, i}$ to $Q_{k, i+1}$ becomes a standard Bellman update $$\label{eqn:approx-Q-k-update}
Q_{k, i+1} \leftarrow \operatorname*{argmin}_{Q_{k, i+1}} \mathbb{E}_{(s, a, r, s') \in B_{\mathrm{mini}}}  \big[ Q_{k, i+1}(s, a) -\big( r + \gamma \cdot  \max_{a'} Q_{k, i+1}(s', a') \big) \big]^2.$$ For stability, Algorithm \[algo:approx-bayes-q\] also uses a target network for each $Q_{k}$ as in Double DQN in the batched update. We point out that the action choice of Algorithm \[algo:approx-bayes-q\] is exploitation only. In the next section, we propose two exploration strategies.

: $K \in \mathbb{N}_+$ copies of independently initialized $Q^*$-functions $\{Q_k\}_{k=1}^K$. Let $B$ be a replay buffer storing transitions for training do Obtain initial state from environment $s_0$ Pick an action according to $a_t = \mathrm{Majority Vote}(\{\operatorname*{argmax}_a Q_k(s_t, a)\}_{k=1}^K)$ Execute $a_t$. Receive state $s_{t+1}$ 