the structure of their feedback loops. We develop an abstract framework called *decoupled RL* that generalises all of these alternative frameworks.

We also show that an agent based on quantilisation [@Taylor2016a] may be more robust to reward corruption when high reward states are much more numerous than corrupt states (\[sec:quant\]). Finally, the results are illustrated with some simple experiments (\[sec:experiments\]). concludes with takeaways and open questions.

Formalisation {#sec:formal}
=============

We begin by defining a natural extension of the MDP framework [@Sutton1998] that models the possibility of reward corruption. To clearly distinguish between true and corrupted signals, we introduce the following notation.

We will let a dot indicate the *true* signal, and let a hat indicate the *observed* (possibly corrupt) counterpart. The reward sets are represented with $\iR=\oR=\R$. For clarity, we use $\iR$ when referring to true rewards and $\oR$ when referring to possibly corrupt, observed rewards. Similarly, we use $\ir$ for true reward, and $\dr$ for (possibly corrupt) observed reward.

\[def:crmdp\] A *corrupt reward MDP* (CRMDP) is a tuple $\mu=\crmdp$ with

-   $\langle\S,\A,\R,T,\irf\rangle$ an MDP with [^1] a finite set of states $\S$, a finite set of actions $\A$, a finite set of rewards $\R=\iR=\oR\subset[0,1]$, a transition function $T(s'| s,a)$, 