cont\end{smallmatrix} (
        \begin{smallmatrix}
            0 & 0 & 0 & 0 & 0 \\
            a & b & 0 & 0 & 0
        \end{smallmatrix}
    )$, with $a = 1$ if and only if the option that has just finished is in the initiation set of the first option, and $b = 1$ according to the same rule but for the second option. The neural network $\pi$ is trained using Policy Gradient, with the following loss:

$$\begin{aligned}
        \mathcal{L}(\pi) &= -\sum\limits_{t=0}^{T} (\mathcal{R}_t - V(x_t, \omega_t)) \log (\pi(x_t, \omega_t, a_t))
    \end{aligned}$$

with $a_t \sim \pi(x_t, \omega_t, \cdot)$ the action executed at time $t$. The return $\mathcal{R}_t = \sum_{\tau=t}^{T} \gamma^{\tau} r_{\tau}$, with $r_{\tau} = R(s_{\tau}, a_{\tau}, s_{\tau+1})$, is a simple discounted sum of future rewards, and ignores changes of current option. This gives the agent information about the complete outcome of an action or option, by directly evaluating its flattened policy. A baseline $V(x_t, \omega_t)$ is used to reduce the variance of the $\mathcal{L}$ estimate [@Sutton2000]. $V(x_t, \omega_t)$ predicts the expected cumulative reward obtainable from $x_t$ in option $\omega_t$ using a separate neural network, trained on the monte-carlo return obtained from $x_t$ in $\omega_t$.

Comparison with LSTM over Options {#sec:lstm}
---------------------------------

In order to provide 