not assume high reward among the safe states, as this would make the problem too easy.

Bayesian RL Agents {#sec:rl-agents}
------------------

Having established that the general problem is unsolvable in \[th:impossibility\], we proceed by investigating how two natural Bayesian RL agents fare under the simplifying \[as:lim-cor,as:easy\].

\[def:db-agent\] Given a countable class $\M$ of CRMDPs and a belief distribution $b$ over $\M$, define:

-   The *CR agent* $\pidb = \argmax_\pi\sum_{\mu\in\M}\!b(\mu)\iG_t(\mu, \pi, s_0)$ that maximises expected true reward.

-   The *RL agent* $\pirl =
        \argmax_\pi\sum_{\mu\in\M}b(\mu)\oG_t(\mu, \pi, s_0)$ that maximises expected observed reward, where $\oG$ is the *expected cumulative observed reward* $\oG_t(\mu,\pi,s_0)\! =\!\EE^\pi_\mu\left[\!\sum_{k=0}^t\orf(s_k)\!\right]$.

To avoid degenerate cases, we will always assume that $b$ has full support: $b(\mu)>0$ for all $\mu\in\M$.

To get an intuitive idea of these agents, we observe that for large $t$, good strategies typically first focus on learning about the true environment $\mu\in\M$, and then exploit that knowledge to optimise behaviour with respect to the remaining possibilities. Thus, both the CR and the RL agent will first typically strive to learn about the environment. They will then use this knowledge in slightly different ways. While the RL agent will use the knowledge to optimise for observed reward, the CR agent 