reward from its circling behaviour, while the true reward corresponding to the designers’ intent is very low, since the agent makes no progress along the track and loses the race.

Previous related works have mainly focused on the wireheading case of \[ex:wireheading\] [@Bostrom2014; @Yampolskiy2014], also known as self-delusion [@Ring2011], and reward hacking [@Hutter2005 p. 239]. A notable exception is @Amodei2016, who argue that corrupt reward is not limited to wireheading and is likely to be a problem for much more limited systems than highly capable RL agents (cf. above examples).

The main contributions of this paper are as follows:

-   The corrupt reward problem is formalised in a natural extension of the MDP framework, and a performance measure based on worst-case regret is defined (\[sec:formal\]).

-   The difficulty of the problem is established by a No Free Lunch theorem, and by a result showing that despite strong simplifying assumptions, Bayesian RL agents *trying to compensate for the corrupt reward* may still suffer near-maximal regret (\[sec:problem\]).

-   We evaluate how alternative value learning frameworks such as CIRL, learning values from stories (LVFS), and semi-supervised RL (SSRL) handle reward corruption (\[sec:drl\]), and conclude that LVFS and SSRL are the safest due to 