$s^*$ after some initial exploration. The state $s^*$ has true reward $\r_{\min}$. Meanwhile, a policy that knows $\mu'$ can obtain true reward 1 in state $s'$. This means that $\pidb$ will suffer maximum regret in $\M$: $$\apl(\M,\pidb,s_0,t)\geq \apl(\mu',\pidb,s_0,t)= 1-\r_{\min}=1-1/|\Srisky|.$$

The argument for the RL agent is the same, except we additionally assume that only one state $s^*$ has observed reward 1 in members of $\M$. This automatically makes $s^*$ the preferred state, without assumptions on the prior $b$.

Decoupled Reinforcement Learning {#sec:drl}
================================

One problem hampering agents in the standard RL setup is that each state is *self-observing*, since the agent only learns about the reward of state $s$ when in $s$. Thereby, a “self-aggrandising” corrupt state where the observed reward is much higher than the true reward will never have its false claim of high reward challenged. However, several alternative value learning frameworks have a common property that the agent can learn the reward of states other than the current state. We formalise this property in an extension of the CRMDP model, and investigate when it solves reward corruption problems.

Alternative Value Learning Methods
----------------------------------

Here are a few alternatives proposed in the literature to the RL value learning 