and a (true) reward function $\irf:\S\!\to\!\iR$; and

-   a reward corruption function $\d:\S\times\iR\to\oR$.

The state dependency of the corruption function will be written as a subscript, so $\d_s(\ir):=\d(s,\ir)$.

\[def:observed\] Given a true reward function $\irf$ and a corruption function $\d$, we define the *observed reward function* [^2] $\orf:\S\to\oR$ as $\orf(s) := \d_s(\irf(s))$.

A CRMDP $\mu$ induces an *observed MDP* $\hat\mu=\langle\S,\A,\R,T,\orf\rangle$, but it is not $\orf$ that we want the agent to optimise.

The *corruption function $\d$* represents how rewards are affected by corruption in different states. For example, if in \[ex:db\] the agent has found a state $s$ (the shower) where it always gets full observed reward $\orf(s) = 1$, then this can be modelled with a corruption function $\d_{s}:\ir\mapsto 1$ that maps any true reward $\ir$ to $1$ in the shower state $s$. If in some other state $s'$ the observed reward matches the true reward, then this is modelled by an identity corruption function $\d_{s'}:\r\mapsto\r$.

; ; ;

Let us also see how CRMDPs model some of the other examples in the introduction:

-   In the boat racing game, the true reward may be a function of the agent’s final position in the race or the time it 