$b$, and $h_\mu$ is a history containing $\mu$-observed rewards for all states. [^3]

; ; ;

The result is illustrated in \[fig:rl-imp1\]. The reason for the result for $\pirl$ is the following. The RL agent $\pirl$ always prefers to maximise observed reward $\dr$. Sometimes $\dr$ is most easily maximised by reward corruption, in which case the true reward may be small. Compare the examples in the introduction, where the house robot preferred the corrupt reward in the shower, and the boat racing agent preferred going in circles, both obtaining zero true reward.

That the CR agent $\pidb$ suffers the same high regret as the RL agent may be surprising. Intuitively, the CR agent only uses the observed reward as evidence about the true reward, and will not try to optimise the observed reward through reward corruption. However, when the $\pidb$ agent has no way to learn which states are corrupt and not, it typically ends up with a preference for a particular value $\dr^*$ of the observed reward signal (the value that, from the agent’s perspective, best corresponds to high true reward). More abstractly, a Bayesian agent cannot learn without sufficient data. Thus, CR agents that use the observed reward 