probabilities in . First, $$\begin{aligned}
 \label{eqn:cond-1}
p( s'|Q^*, T, (s, a)) = p(s'| T, (s, a)) = T(s'|s, a),\end{aligned}$$ where the first equality is because given $T$, $Q^*$ does not influence the transition. Second, $$\begin{aligned}
p(r |Q^*, T, (s, a, s')) & = p(r | Q^*, T, (s, a)) \nonumber\\
& = \mathbbm{1}_{\{ Q^*(s, a) = r + \gamma \cdot \mathbb{E}_{s'' \sim  T(\cdot|s, a)} \max_{a'} Q^*(s'', a')  \}} \nonumber\\
&:= \mathbbm{1}(Q^*, T), \label{eqn:cond-2}\end{aligned}$$ where $\mathbbm{1}_{\{\cdot\}}$ is the indicator function and in the last equation we abbreviate it as $\mathbbm{1}(Q^*, T)$. Substituting  and into , we obtain the joint posterior of $Q^*$ and $T$ after observing an additional randomly sampled transition $\tau$ $$\label{eqn:bayes-update}
\tilde{p}(Q^*, T | \tau ) = \frac{p(Q^*, T) \cdot T(s'|s, a)  \cdot p(s,a)}{Z(\tau)} \cdot \mathbbm{1}(Q^*, T).$$ We point out that the exact $Q^*$-posterior update is intractable in high-dimensional RL due to the large space of $(Q^*, T)$.

$\boldsymbol{Q}$-learning with $\boldsymbol{Q}$-ensembles {#section:approximation}
---------------------------------------------------------

In this section, we make several approximations to the $Q^*$-posterior update and derive a tractable algorithm. First, we approximate the prior of $Q^*$ by sampling $K \in \mathbb{N}_+$ independently initialized $Q^*$-functions $\{Q_k\}_{k=1}^K$. Next, we update them as more transitions are sampled. The resulting $\{Q_k\}$ approximate samples 