agent wants to avoid catastrophic forgetting of dangerous states. They proposed to learn a classifier that detects hazardous states and then use it to shape the reward of a DQN agent. Fulda et al. ([-@fulda2017can]) studied affordances, the set of behaviors enabled by a situation, and presented a method for affordance extraction via inner products of pre-trained word embeddings.

Action Elimination {#action_elimination}
==================

We now describe a learning algorithm for MDPs with an elimination signal. Our approach builds on the standard RL formulation [@sutton1998reinforcement]. At each time step $t$, the agent observes a state $s_t$ and chooses a discrete action $a_t \in \left \{1,..,|A| \right \}$. After executing the action, the agent obtains a reward $r_t(s_t,a_t)$ and observes the next state $s_{t+1}$ according to a transition kernel $P(s_{t+1}|s_t,a_t)$. The goal of the algorithm is to learn a policy $\pi(a|s)$ that maximizes the discounted cumulative return $V^\pi(s)=\mathbb{E}^\pi\brs*{\sum_{t=0}^\infty\gamma^tr(s_t,a_t)\vert s_0=s}$ where $0<\gamma<1$ is the discount factor and $V$ is the value function. The optimal value function is given by $V^*(s)=\max_\pi V^\pi(s)$ and the optimal policy by $\pi^*(s)=\arg\max_\pi V^\pi(s)$. The Q-function $Q^{\pi}(s,a)=\mathbb{E}^{\pi}\brs*{\sum_{t=0}^\infty\gamma^tr(s_t,a_t)\vert s_0=s,a_0=a}$ corresponds to the value of taking action $a$ in state $s$ and continuing according to policy $\pi$. The optimal Q-function $Q^*(s,a)=Q^{\pi^*}(s,a)$ 