a complete evaluation of [OOIs]{}, a variant of the $\pi$ and $V$ networks of Section \[sec:nnet\], where the hidden layer is replaced with a layer of 20 LSTM units [@Hochreiter1997; @Sridharan2010], is also evaluated on every task. We use 20 units as this leads to the best results in our experiments, which ensures a fair comparison of LSTM against OOIs. In all experiments, the LSTM agents are provided the same set of options as the agent with [OOIs]{}. Not providing any option, or less options, leads to worse results. Options allow the LSTM network to focus on important observations, and reduces the time horizon to be considered. Shorter time horizons have been shown to be beneficial to LSTM [@Bakker2001].

Despite our efforts, LSTM over options only manages to learn good policies in our robotic experiment (see Section \[sec:khepera\]), and requires more than twice the amount of episodes as [OOIs]{}to do so. In our repetitive task, dozens of repetitions seem to confuse the network, that quickly diverges from any good policy it may learn (see Section \[sec:duplicatedinput\]). On TreeMaze, a much more complex version of the T-maze task, originally used to benchmark reinforcement learning LSTM agents [@Bakker2001], the LSTM agent learns 