the particular instance of Hierarchical RL we focus on, addresses the first challenge by factoring a complex task into simpler sub-tasks [@Barto2003; @Roy2006; @Tessler2016]. Instead of learning what action to perform depending on an observation, the agent learns a top-level policy that repeatedly selects options, that in turn execute sequences of actions before returning [@Sutton1999]. The second challenge, partial observability, is addressed by maintaining a belief of what the agent thinks the full state is [@Kaelbling1998; @Cassandra1994], reasoning about possible future observations [@Littman2001; @Boots2009], storing information in an external memory for later reuse [@Peshkin2001; @Zaremba2015; @Graves2016], or using recurrent neural networks (RNNs) to allow information to flow between time-steps [@Bakker2001; @Mnih2016].

Combined solutions to the above two challenges have recently been designed for planning [@He2011], but solutions for learning algorithms are not yet ideal. HQ-Learning decomposes a task into a sequence of fully-observable subtasks [@Wiering1997], which precludes cyclic tasks from being solved. Using recurrent neural networks in options and for the top-level policy [@Sridharan2010] addresses both challenges, but brings in the design complexity of RNNs [@Jozefowicz15; @Angeline94; @Mikolov14]. RNNs also have limitations regarding long time horizons, as their memory decays over time [@Hochreiter1997].

In her PhD thesis, Precup ([-@Precup2000], page 