leverage random [OOIs]{}, thereby removing the need for designing them. In Section \[sec:treemaze\], we progressively reduce the amount of options available to the agent, and demonstrate how [OOIs]{}still allow good memory-based policies to emerge when a sub-optimal amount of options are used.

All our results are averaged over 20 runs, with standard deviation represented by the light regions in the figures. The source code, raw experimental data, run scripts, and plotting scripts of our experiments, along with a detailed description of our robotic setup, are available as supplementary material. A video detailing our robotic experiment is available at <http://steckdenis.be/oois_demo.mp4>.

Learning Algorithm {#sec:nnet}
------------------

All our agents learn their top-level and option policies (if not provided) using a single feed-forward neural network, with one hidden layer of 100 neurons, trained using Policy Gradient [@Sutton2000] and the Adam optimizer [@kingma2014adam]. Our neural network $\pi$ takes three inputs and produces one output. The inputs are problem-specific observation features $\mathbf{x}$, the one-hot encoded current option $\boldsymbol{\omega}$ ($\boldsymbol{\omega} = \mathbf{0}$ when executing the top-level policy), and a mask, $\mathbf{mask}$. The output $\mathbf{y}$ is the joint probability distribution over selecting actions or options (so that the same network can be used for the top-level and option 