drawn from the posterior. The agent chooses the action by taking a majority vote from the actions determined by each $Q_k$. We display our method, Ensemble Voting, in Algorithm \[algo:approx-bayes-q\].

We derive the update rule for $\{Q_k\}$ after observing a new transition $\tau=(s, a, r, s')$. At iteration $i$, given $Q^*=Q_{k, i}$ the joint probability of $(Q^*, T)$ factors into $$\label{eqn:prior-factor}
p(Q_{k,i}, T) = p(Q^*, T| Q^*=Q_{k,i} ) = p(T| Q_{k, i}).$$ Substitute into and we obtain the corresponding posterior for each $Q_{k, i+1}$ at iteration $i+1$ as $$\begin{aligned}
 \label{eqn:new-posterior}
\tilde{p}(Q_{k, i+1}, T|\tau) &= \frac{p(T|Q_{k, i}) \cdot T(s'|s, a) \cdot p(s, a) }{Z(\tau)} \cdot \mathbbm{1}(Q_{k, i+1}, T).\\
\tilde{p}(Q_{k, i+1} | \tau) &= \int_{T} \tilde{p}(Q_{k, i+1}, T|\tau)  \mathrm{d} T =  p(s, a) \cdot \int_{T} \tilde{p}(T|Q_{k,i}, \tau) \cdot \mathbbm{1}(Q_{k, i+1}, T) \mathrm{d} T.\end{aligned}$$ We update $Q_{k, i}$ to $Q_{k, i+1}$ according to $$\label{eqn:update-Q-k}
Q_{k, i+1} \leftarrow \operatorname*{argmax}_{Q_{k, i+1}} \tilde{p}(Q_{k, i+1}|\tau).$$ We first derive a lower bound of the the posterior $\tilde{p}(Q_{k,i+1}|\tau)$: $$\begin{aligned}
& \tilde{p}(Q_{k, i+1}|\tau)  = p(s, a)\cdot {\operatorname{\mathbb{E}}}_{T \sim \tilde{p}(T|Q_{k,i}, \tau)} \mathbbm{1}(Q_{k, i+1}, T)  \nonumber \\
& = p(s, a) \cdot {\operatorname{\mathbb{E}}}_{T \sim \tilde{p}(T|Q_{k,i}, \tau)} \lim_{c\rightarrow + \infty}  \exp\big(-c [Q_{k, i+1}(s,a) - r - \gamma {\operatorname{\mathbb{E}}}_{s'' \sim T(\cdot|s, a)} \max_{a'}Q_{k,i+1}(s'', a')]^2\big) 