takes to complete the race, depending on the designers’ intentions. The reward corruption function $\d$ increases the observed reward on the loop the agent found. has a schematic illustration.

-   In the wireheading example, the agent finds a way to hijack the reward channel. This corresponds to some set of states where the observed reward is (very) different from the true reward, as given by the corruption function $\d$.

The CIRL example will be explored in further detail in \[sec:drl\].

#### CRMDP classes

Typically, $T$, $\irf$, and $\d$ will be fixed but unknown to the agent. To make this formal, we introduce classes of CRMDPs. Agent uncertainty can then be modelled by letting the agent know only which class of CRMDPs it may encounter, but not which element in the class.

For given sets $\Tf$, $\iRf$, and $\D$ of transition, reward, and corruption functions, let $\M=\crmdpclass$ be the class of CRMDPs containing $\crmdp$ for $(T,\irf,\d)\in \Tf\times\iRf\times\D$.

#### Agents

Following the POMDP [@Kaelbling1998] and general reinforcement learning [@Hutter2005] literature, we define an agent as a (possibly stochastic) policy $\pi:\H\leadsto\A$ that selects a next action based on the *observed history* $\oh_n=s_0\dr_0a_1s_1\dr_1\dots a_ns_n\dr_n$. Here $X^*$ denotes the set of finite sequences that can 