$\pi^*$ that maximizes the expected cumulative discounted reward $E_{\pi^*}[\sum_t \gamma^t r_t]$ obtainable by following that policy.

Options
-------

The options framework, defined in the context of MDPs [@Sutton1999], consists of a set of options $O$ where each option $\omega \in O$ is a tuple $\langle \pi_\omega, I_\omega, \beta_\omega \rangle$, with $\pi_\omega (s_t, a_t) \in [0, 1]$ the memoryless option policy, $\beta_\omega(s_t) \in [0, 1]$ the termination function that gives the probability for the option $\omega$ to terminate in state $s_t$, and $I_\omega \subseteq S$ the initiation set that defines in which states $\omega$ can be started [@Sutton1999].

The memoryless top-level policy $\mu(s_t, \omega_t) \in [0, 1]$ maps states to a distribution over options and allows to choose which option to start in a given state. When an option $\omega$ is started, it executes until termination (due to $\beta_\omega$), at which point $\mu$ selects a new option based on the now current state.

Partially Observable MDPs
-------------------------

Most real-world problems are not completely captured by MDPs, and exhibit at least some degree of partial observability. A Partially Observable MDP (POMDP) $\langle \Omega, S, A, R, T, W, \gamma \rangle$ is an MDP extended with two components: the possibly-infinite set $\Omega$ of 