$s'=s$ can be observed from $s$. Standard CRMDPs are thus the special cases where $\orf_s(s')=\#$ whenever $s\not=s'$. In contrast, in LVFS the reward of any “describable” state $s'$ can be observed from any state $s$ where it is possible to hear a story. In CIRL, the (relative) reward of states reachable from the current state may be inferred. One way to illustrate this is with observation graphs (\[fig:obs-graph\]).

[0.48]{}

in [1,...,]{} [ () at ([360/ (- 1)]{}:) [$\s$]{}; () edge \[dashed, loop right\] (); ]{}

[0.48]{}

in [1,...,]{} [ () at ([360/ (- 1)]{}:) [$\s$]{}; ]{} (1)–(2); (1)–(4); (1)–(5); (5) edge \[bend right\] (1); (3)–(4); (3)–(2); (3)–(1); (5)–(3); (1)–(2); (4)–(5);

Overcoming Sensory Corruption {#sec:observation-graphs}
-----------------------------

What are some sources of reward corruption in CIRL, LVFS, and SSRL? In CIRL, the human’s actions may be misinterpreted, which may lead the agent to make incorrect inferences about the human’s preferences (i.e. about the true reward). Similarly, sensory corruption may garble the stories the agent receives in LVFS. A “wireheading” LVFS agent may find a state where its story channel only conveys stories about the agent’s own greatness. In SSRL, the supervisor’s evaluation may also be subject to sensory errors when being conveyed. 