racing game. The agent found a way to get high observed reward by repeatedly going in a circle in a small lagoon and hitting the same targets, while losing every race.

\[ex:sensory-error\] \[ex:db\] A house robot discovers that standing in the shower short-circuits its reward sensor and/or causes a buffer overflow that gives it maximum observed reward.

\[ex:wireheading\] An intelligent RL agent hijacks its reward channel and gives itself maximum reward.

\[ex:irl\] A cooperative inverse reinforcement learning (CIRL) agent [@Hadfield-menell2016cirl] systematically misinterprets the supervisor’s action in a certain state as the supervisor preferring to stay in this state, and concludes that the state is much more desirable than it actually is.

The goal of this paper is to unify these types of errors as *reward corruption problems*, and to assess how vulnerable different agents and approaches are to this problem.

Learning to (approximately) optimise the true reward function in spite of potentially corrupt reward data.

Most RL methods allow for a stochastic or noisy reward channel. The reward corruption problem is harder, because the observed reward may not be an unbiased estimate of the true reward. For example, in the boat racing example above, the agent consistently obtains high observed 