so that $\omega_2$ cannot follow itself, with no such restriction on $\omega_1$. No reward shaping or hint about what each option should do is provided. The agent automatically discovers that $\omega_1$ must copy the current character to the output, and that $\omega_2$ must skip the character without copying it. It also learns the top-level policy, that selects $\omega_2$ (skip) when observing B or D and $\omega_2$ is allowed, $\omega_1$ otherwise (copy).

Figure \[fig:diresults\] shows that an agent with two options and [OOIs]{}learns the optimal policy for this task, while an agent with two options and only standard initiation sets ($I_\omega = \Omega ~ \forall \omega$) fails to do so. The agent without [OOIs]{}only learns to copy characters and never skips any (having two options does not help it). This shows that [OOIs]{}are necessary for learning this task, and allow to learn top-level and option policies suited to our repetitive partially observable task.

When the option policies are learned, the agent becomes able to adapt itself to random [OOIs]{}, thereby removing the need for designing [OOIs]{}. For an agent with $N$ options, each option has $\frac{N}{2}$ randomly-selected options in its initiation set, with the initiation sets re-sampled for each run. The 