its accuracy using two 1000-episodes runs on the actual robot. The agent learns to properly select options at the terminals, depending on the QR-code, and to output a proper distribution over options at the root, thereby matching our expert policy. The LSTM agent learns the policy too, but requires more than twice the amount of episodes to do so. The high variance displayed in Figure \[fig:kresults\] comes from the varying amounts of objects in the terminals, and the random selection of how many times they have to be emptied.

Because fixed option policies are not always available, we now show that [OOIs]{}allow them to be learned at the same time as the top-level policy.

Modified DuplicatedInput {#sec:duplicatedinput}
------------------------

In some cases, a hierarchical reinforcement learning agent may not have been provided policies for several or any of its options. In this case, [OOIs]{}allow the agent to learn its top-level policy, the option policies and their termination functions. In this experiment, the agent has to learn its top-level and option policies to copy characters from an input tape to an output tape, removing duplicate B’s and D’s (mapping ABBCCEDD to ABCCED for instance; B’s and D’s always appear in pairs). The 