scheme:

-   Cooperative inverse reinforcement learning (CIRL) [@Hadfield-menell2016cirl]. In every state, the agent observes the actions of an expert or supervisor who knows the true reward function $\irf$. From the supervisor’s actions the agent may infer $\irf$ to the extent that different reward functions endorse different actions.

-   Learning values from stories (LVFS) [@Riedl2016]. Stories in many different forms (including news stories, fairy tales, novels, movies) convey cultural values in their description of events, actions, and outcomes. If $\irf$ is meant to represent human values (in some sense), stories may be a good source of evidence.

-   In (one version of) semi-supervised RL (SSRL) [@Amodei2016], the agent will from time to time receive a careful human evaluation of a given situation.

These alternatives to RL have one thing in common: they let the agent learn something about the value of some states $s'$ different from the current state $s$. For example, in CIRL the supervisor’s action informs the agent not so much about the value of the current state $s$, as of the relative value of states reachable from $s$. If the supervisor chooses an action $a$ rather than $a'$ in $s$, then the states following $a$ must have value 