outlined in Section \[section:bayesian\] which learned the posterior of the MDP, we will consider the joint distribution over $(Q^*, T)$. Note that $R$ can be recovered from $Q^*$ given $T$. So $(Q^*, T)$ determines a unique MDP. In this section, we assume that the agent samples $(s, a)$ according to a fixed distribution. The corresponding reward $r$ and next state $s'$ given by the MDP append to $(s, a)$ to form a transition $\tau = (s, a, r, s’)$, for updating the posterior of $(Q^*, T)$. Recall that the $Q^*$-function satisfies the Bellman equation $$Q(s, a) = r + \mathbb{E}_{s' \sim T(\cdot |s, a)} \left[ \gamma \max_{a'}Q(s', a') \right].$$ Denote the joint prior distribution as $p(Q^*, T)$ and the posterior as $\tilde{p}$. We apply Bayes’ formula to expand the posterior: $$\begin{aligned}
\tilde{p}(Q^*, T | \tau ) & = \frac{ p(\tau | Q^*, T) \cdot p(Q^*, T) }{Z(\tau)}  \nonumber \\
& = \frac{p(Q^*, T) \cdot  p(s' | Q^*, T, (s, a)) \cdot p(r | Q^*, T, (s, a, s')) \cdot p(s,a) }{Z(\tau)} \label{eqn:bayes},\end{aligned}$$ where $Z(\tau)$ is a normalizing constant and the second equality is because $s$ and $a$ are sampled randomly from $\mathcal{S}$ and $\mathcal{A}$. Next, we calculate the two conditional 