higher or equal than the states following $a'$. Similarly, stories describe the value of states other than the current one, as does the supervisor in SSRL. We therefore argue that CIRL, LVFS, and SSRL all share the same abstract feature, which we call *decoupled reinforcement learning*:

A *CRMDP with decoupled feedback*, is a tuple $\drmdp$, where $\S,\A,\R,T,\irf$ have the same definition and interpretation as in \[def:crmdp\], and $\{\orf_s\}_{s\in\S}$ is a collection of observed reward functions $\orf_s:\S\to\R\bigcup\{\#\}$. When the agent is in state $s$, it sees a pair $\langle s',\orf_s(s')\rangle$, where $s'$ is a randomly sampled state that may differ from $s$, and $\orf_s(s')$ is the reward observation for $s'$ from $s$. If the reward of $s'$ is not observable from $s$, then $\orf_s(s')=\#$.

The pair $\langle s',\orf_s(s')\rangle$ is observed in $s$ instead of $\orf(s)$ in standard CRMDPs. The possibility for the agent to observe the reward of a state $s'$ different from its current state $s$ is the key feature of CRMDPs with decoupled feedback. Since $\orf_s(s')$ may be blank $(\#)$, all states need not be observable from all other states. Reward corruption is modelled by a mismatch between $\orf_s(s')$ and $\irf(s')$.

For example, in RL only the reward of 