<EMAIL_ADDRESS>
# Hierarchical Reinforcement Learning in Complex 3D Environments

Bernardo Avila Pires  Equal contributions  DeepMind  Feryal Behbahani  Equal
contributions  DeepMind  Huber Soyer  Equal contributions  DeepMind  Kyriacos
Nikiforou  Equal contributions  DeepMind  Thomas Keck  Equal contributions
DeepMind  Satinder Singh  Equal contributions  DeepMind

###### Abstract

Hierarchical Reinforcement Learning (HRL) agents have the potential to
demonstrate appealing capabilities such as planning and exploration with
abstraction, transfer, and skill reuse. Recent successes with HRL across
different domains provide evidence that practical, effective HRL agents are
possible, even if existing agents do not yet fully realize the potential of
HRL. Despite these successes, visually complex partially observable 3D
environments remained a challenge for HRL agents. We address this issue with
Hierarchical Hybrid Offline-Online (H2O2), a hierarchical deep reinforcement
learning agent that discovers and learns to use options from scratch using its
own experience. We show that H2O2 is competitive with a strong non-
hierarchical Muesli baseline in the DeepMind Hard Eight tasks and we shed new
light on the problem of learning hierarchical agents in complex environments.
Our empirical study of H2O2 reveals previously unnoticed practical challenges
and brings new perspective to the current understanding of hierarchical agents
in complex domains.

###### keywords:

Hierarchical Reinforcement Learning, Partially Observable Markov Decision
Processes, Deep Reinforcement Learning

##  1 Introduction

Hierarchical Reinforcement Learning (HRL; Barto and Mahadevan, 2003; Sutton
and Barto, 2018; Pateria et al., 2021; Hutsebaut-Buysse et al., 2022) is a
framework that could provide us with general and reusable agent
representations and behaviors that can exhibit improved exploration and
temporal abstraction (Nachum et al., 2019). The inspiration comes from humans’
ability to break down novel tasks into a sequence of simpler sub-tasks they
know how to solve (Solway et al., 2014). This hierarchical approach enables us
to transfer our knowledge and reuse our skills to solve new problems.

#### Contributions.

In this work we introduce Hierarchical Hybrid Offline-Online (H2O2), a
hierarchical deep reinforcement learning agent that discovers and learns to
use options from scratch using its own experience. We show that H2O2 is
competitive with a strong non-hierarchical Muesli baseline (Hessel et al.,
2021) in the Hard Eight task suite Gulcehre et al. (2019); Ward et al. (2020).
These are challenging sparse-reward tasks in a complex partially observable,
first-person 3D environment. H2O2 employs a combination of primitive actions
and temporally-extended options selected from a continuous option space111We
also provide videos of the agent (see Section C.4 for details). To the best of
our knowledge, this is the first hierarchical agent that can be competitive
with a strong flat baseline in tasks as complex as the Hard Eight suite, while
demonstrably using options to solve tasks.

Our work also sheds new light on the problem of learning hierarchical agents
and learning options in complex environments. We use H2O2 to test a number of
hypotheses about its learning and performance in response to changes in its
hierarchical design, and our results reveal previously undetected practical
challenges. While some of our experiments support conclusions in line with the
conventional understanding of HRL, others challenged our understanding of
hierarchical agents. For instance, we observed that seemingly beneficial
actions such as increasing the agent’s option space or allowing it to learn
longer options, can actually hurt its performance.

##  2 Background and Related Work

A common HRL approach is to add options to the MDP, turning it into a semi-MDP
(SMDP; Sutton et al., 1999), and then use a general-purpose RL algorithm to
solve the SMDP (Dayan and Hinton, 1992; Barto and Mahadevan, 2003). _SMDP
agents_ decompose into a low-level controller (LLC, which executes the options
in the original MDP) and a high-level controller (HLC, which learns to solve
the SMDP). The options may include the actions of the MDP (the _primitive
actions_) in addition to temporally extended behaviors _per se_ , so that no
generality is lost when solving the SMDP instead of solving the MDP
“directly”.

The SMDP strategy effectively changes the problem for the general-purpose RL
algorithm. It is possible to add various capabilities to the general-purpose
RL algorithm by augmenting the SMDP with expressive, diverse options (Barreto
et al., 2019), additional/alternative state representations (Dayan and Hinton,
1992; Shah et al., 2022), and even actions to exert fine-grained control over
the options (Precup, 2000; Barto and Mahadevan, 2003).

Options can be learned from experience, which can be generated by the
hierarchical agent itself (“learned from scratch”; Bacon et al., 2017;
Eysenbach et al., 2018; Harutyunyan et al., 2019; Wulfmeier et al., 2021; Ajay
et al., 2020; Hafner et al., 2022), or by another agent (for example, an
expert; Merel et al., 2019; Lynch et al., 2020; Lynch and Sermanet, 2020). The
agent can be learned as one unit (Bacon et al., 2017; Merel et al., 2019;
Wulfmeier et al., 2021; Ajay et al., 2020), but one can also decouple the
LLC’s option-learning and the HLC’s general-purpose RL algorithm (Dayan and
Hinton, 1992; Vezhnevets et al., 2017; Hafner et al., 2022).

There have been recent successes in learning _goal-conditioned_ options
(Machado and Bowling, 2016; Lynch et al., 2020; Lynch and Sermanet, 2020; Ajay
et al., 2020; Khazatsky et al., 2021; Mendonca et al., 2021). These behaviors
are trained to produce specific outcomes (observations or states) in the
environment, and they are commonly learned in hindsight from trajectories of
agent (Andrychowicz et al., 2017). The idea is to identify goals achieved in
each trajectory, and use trajectories as demonstrations of behavior that
achieves the goal. The policy can be trained, for example, using behavior
cloning (BC; Pomerleau, 1989) or offline RL (Lange et al., 2012; Nachum et
al., 2018; Fujimoto et al., 2019; Levine et al., 2020; Fu et al., 2020;
Gulcehre et al., 2020). The effectiveness of the learned options is largely
affected by the choice of policy-learning algorithm, the data available, and
how goals are discovered. For example, BC has been shown to yield effective
goal-conditioned policies when used on data generated by experts (Lynch et
al., 2020), but not on non-expert data (Ajay et al., 2020), whereas offline RL
has shown more promise in the latter case.

Discovering both which sub-behaviors to learn and how to combine them can be
tackled by pre-learning skills/behaviors with a variety of signals such as
expert demonstrations (Gupta et al., 2020), pseudo-rewards (Barreto et al.,
2019), state-space coverage (Eysenbach et al., 2018; Lee et al., 2019; Islam
et al., 2019; Pong et al., 2020), empowerment (Gregor et al., 2017), among
many others. Alternatively, the agent can learn its sub-behaviors from its own
data, that is, “from scratch” (Wulfmeier et al., 2021; Hafner et al., 2022).
This approach has the appeal of being end-to-end, and is philosophically
aligned with mainstream deep RL, where agents learn on data that is relevant
to the task they are expected to perform (Mnih et al., 2015; Silver et al.,
2018; Gulcehre et al., 2020; Fu et al., 2020). It is justified on the grounds
that a learning agent will ultimately have to collect novel experience and
learn novel sub-behaviors on that experience.

The set of options added to the SMDP can also vary. If there are only a few
options, they can be learnt as separate entities (Bacon et al., 2017;
Wulfmeier et al., 2021). A much larger set of options (and, more specifically,
goal conditioned policies), on the other hand, can be learned implicitly by
encoding the options (goals) in latent space, and treating any element of the
latent space as a valid option (goal) (Merel et al., 2019; Lynch et al., 2020;
Lynch and Sermanet, 2020; Ajay et al., 2020; Hafner et al., 2022). In this
case the whole latent space is part of the action space for the SMDP, and the
HLC needs to learn to select elements of this latent space. The complexity of
the set of options can be, to a certain extent, limited, by regularizing or
constraining the latent output of the option encoders (Merel et al., 2019;
Lynch et al., 2020; Lynch and Sermanet, 2020; Ajay et al., 2020; Hafner et
al., 2022). We are not aware of any successful HRL approaches that encode
goals in latent space but do not constrain the latent output of the option
encoders in some way. This suggests that some manner of latent space
regularization is essential for deep RL SMDP agents, and this hypothesis is
consistent with the empirical findings we present in this work. We will see in
our experiments that not constraining the latent output of the encoder is
detrimental H2O2’s performance.

We are primarily interested in partially observable environments, and we adopt
the typical deep RL setup for this type of domain: The agent, at each timestep
$t$, must select the action $a_{t}$ according to a stochastic policy that can
depend on the _history_ (the sequence of past observations
$o_{1},\ldots,o_{t}$ and actions $a_{1},\ldots,a_{t-1}$). This reduces the
POMDP to an MDP whose states are the histories of past observations and
actions (Cassandra et al., 1994). This design choice burdens the deep RL agent
with learning to represent histories effectively, but it allows us to use
general-purpose deep RL algorithms on both MDPs and POMDPs. For SMDP agents,
both the MDP and the SMDP are treated as above: The LLC acts in a partially
observable environment reduced to an MDP over histories, and the HLC acts in a
partially observable environment reduced to an SMDP over histories.

##  3 Agent Design

H2O2 is an SMDP agent with options learned from scratch, in offline fashion,
that is _decoupled_ from the HLC. The options are goal-conditioned: The goals
are selected in hindsight from experience and encoded into a latent space;
then we use offline RL to train an LLC to attain these goals. The general-
purpose deep RL algorithm used for the HLC is Muesli (Hessel et al., 2021),
which has state-of-the-art performance in the Atari benchmark (Bellemare et
al., 2013). We train the HLC online, through interaction with the environment,
as usual for deep RL agents. Due to our agent’s hierarchical design and how
its components are trained, we call it Hierarchical Hybrid Offline-Online
(H2O2).

Figure 1 gives an overview of H2O2, and how its components interact with each
other and the environment.

Figure 1: H2O2 component diagram. The dotted boxes indicate how components (in
blue) are trained. The arrows indicate information (inputs, outputs and
gradients) passed between components.

We outline H2O2’s main components in the rest of this section, and we give
details in Appendix B.

###  3.1 Low-Level Controller Design

#### Training data.

The experience generated by H2O2 is inserted into a replay buffer (Horgan et
al., 2018; Cassirer et al., 2021) as the agent interacts with the environment.
The LLC learner processes minibatches of trajectories sampled (uniformly at
random) from the replay (Horgan et al., 2018), where each trajectory has the
form $(o_{1},a_{1},\ldots,o_{n},a_{n})$.

We sample _start_ and _end_ timesteps $t_{s},t_{e}$ from the set
$\\{(i,j):1\leq i<j\leq n\\}$, encode $o_{t_{e}}$ with the Goal Encoder (see
Fig. 1) to obtain a _latent goal_ $g\in[-1,1]^{d}$. The sampled goal is fixed
for the sampled subtrajectory ($g_{t_{s}}=g_{t_{s}+1}=\cdots=g_{t_{e}}=g$) and
each subtrajectory is treated as a separate episodic task terminating on
timestep $t_{e}$ (when the goal $g$ is attained). The reward is
$r_{t}\doteq\mathbb{I}\\{t=t_{e}\\}$ ($\mathbb{I}$ denotes the indicator
function), that is, one for attaining the goal, and zero otherwise. During
training we sample multiple pairs $t_{s},t_{e}$ per trajectory in the
minibatch, so we train the LLC on multiple tasks (goals) at once.

The LLC’s policy and value function are conditioned on the latent goal $g_{t}$
and on the agent’s _representation_ $b_{t}$ (the “agent state” Sutton and
Barto, 2018). To compute $b_{t}$, we process the observations and actions
using a recurrent IMPALA-like network (Espeholt et al., 2018) (the Observtion
Encoder and the RNN Core in Fig. 1). For each $t$, the representation depends
only on the previous observations and actions, and on the recurrent state of
the neural network before $o_{1}$.

#### Goal Sampling during Training.

During goal sampling, we reject subtrajectories that are too short or too long
(determined by hyperparameters), as well as goals that are too “similar” to
other goals (details in Section B.1). We also increase the relative frequency
of pairs $(t_{s},t_{e})$ where the reward from the environment is non-zero for
some timestep $t_{s}\leq t\leq t_{e}$. Increasing the frequency of reward-
related goals allowed us to direct the behavior of the LLC to be meaningful
for the RL problem, without introducing too much of a dependency on task
rewards, rather than learning options that directly optimize for the
environment reward (Hafner et al., 2022). Controlling the goal sampling
distribution also provides a direct way to study the option discovery problem
using H2O2.

#### Offline RL.

To learn the LLC policy $\pi_{\mathrm{LLC}}$, we introduce a _regularized_
offline V-Trace (Espeholt et al., 2018; Mathieu et al., 2021).
$\pi_{\mathrm{LLC}}$ is a distribution over primitive actions that we optimize
by following a V-Trace policy gradient. We also regularize
$\pi_{\mathrm{LLC}}$ to stay close to an estimate of the behavior policy
$\widehat{\mu}$, trained with behavior cloning (Pomerleau, 1989). Similar to
other offline RL works (Fujimoto et al., 2019; Gulcehre et al., 2021), we
found this regularizer to be essential for training an effective LLC, as
removing it or annealing it out made the offline RL ineffective.

The gradient step of regularized offline V-Trace is:

$\begin{split}&\frac{\pi_{\mathrm{LLC}}(a_{t}|b_{t},g)}{\widehat{\mu}(a_{t}|b_{t},g)}\cdot\operatorname{\mathrm{Adv}}_{t}\cdot\nabla\log\pi_{\mathrm{LLC}}(a_{t}|b_{t},g)\\\
&-\alpha\nabla
KL(\pi_{\mathrm{LLC}}(b_{t},g)|\widehat{\mu}(b_{t},g)),\end{split}$ (1)

where $\operatorname{\mathrm{Adv}}_{t}$ is the advantage estimate at time $t$
(computed using V-Trace returns and the value estimate
$\widehat{V}^{\pi_{\mathrm{LLC}}}$, Espeholt et al., 2018), $\alpha$ is a
fixed hyperparameter for the KL regularizer, and the gradient is taken only
with respect to the parameters of $\pi_{\mathrm{LLC}}$ (without
differentiating through $\widehat{\mu}$). Differently from the original
V-Trace, Eq. 1 uses an estimate of the behavior policy instead of the behavior
policy itself.

Following Espeholt et al. (2018), we add a weighted neg-entropy regularizer
($-H(\pi_{\mathrm{LLC}})$) to the objective for $\pi_{\mathrm{LLC}}$. We train
the value function estimate $\widehat{V}^{\pi_{\mathrm{LLC}}}$ through
regression (as done by Espeholt et al., 2018, and akin to Fitted Q Iteration,
Ernst et al., 2005). The representation $b_{t}$ and the latent goal $g_{t}$
are shared between $\pi_{\mathrm{LLC}}$, $\widehat{V}^{\pi_{\mathrm{LLC}}}$
and $\widehat{\mu}$, and all three learning tasks (regularized policy
gradient, value learning and behavior cloning) flow gradients into the
representation and the latent goal.

#### Variational Goal Encoder.

Our Goal Encoder is inspired by variational autoencoders (Kingma and Welling,
2013; Rezende et al., 2014) and it outputs a distribtuion over the goal space
from which we can sample goals $g$. Concretely, Goal Encoder outputs the
parameters of a multivariate normal distribution with diagonal covariance that
is used to sample the latent goal $g$. Differently from VAEs, we do not
attempt to autoencode the input of the Goal Encoder from the sampled latent
goals, but instead use the sampled latent goals for the Offline RL and
auxiliary tasks.

We use a KL regularizer term with weight $\beta$ to encourage the multivariate
normal to stay close to a standard normal distribution. A weight $\beta=0$
will allow the goal space to be as expressive as afforded by the Goal Encoder,
whereas a large enough $\beta$ will effectively cause the goals $g$ to be
sampled from a standard normal distribution (ignoring the observation input to
the encoder). The KL regularization is primarily for the benefit of the HLC,
as our empirical results will show. We believe that this regularization makes
the goal space smoother, and make it easier for the HLC to explore and choose
goals.

#### Auxiliary Tasks.

In addition to the learning objectives outlined above, we employed three
auxiliary prediction tasks to improve the quality of our LLC.

The first one is training the Goal Model in Fig. 1. via maximum likelihood
estimation of latent goals $g_{t}$ conditioned on $c_{t}$ (the output of the
agent’s Observation Encoder). This is an auxiliary only flow gradients into
the observation encoder (not $g$). We observed that, without this auxiliary
task, the LLC would frequently ignore the goal conditioning.

The second auxiliary task is to estimate the state value of the behavior
policy, with respect to the environment rewards. The value estimate is a
function of $b_{t}$ and $g$ and flows gradient into both. We found this
auxiliary task to be beneficial, and we believe it helps by shaping $b_{t}$
and $g$ to encode information about rewarding states in the environment.

The third auxiliary task is to predict, from each timestep $t$, how far in
future the goal is. We frame the prediction task as a multiclass logistic
classification task. During training, if the goal is on step $t_{e}$, then the
classification label for each step $t$ is $t_{e}-t$, out of $n$ possible
classes, where $n$ is an upper-bound on how far into the future goals can be.

###  3.2 SMDP and High-Level Controller Design

The HLC can instruct the LLC to execute either primitive actions or goals (and
the decision of which one to choose at each step is part of the HLC’s policy),
and the LLC executes them in the call-and-return model (Dayan and Hinton,
1992).

#### Option Termination and Initiation.

We improved our agent’s sample efficiency by composing simple fixed rules and
learned termination/initiation functions. We used a hard limit on option
duration (timeout; Sutton et al., 1999) as the fixed rule, and an “attainment
classifier” as the learned termination function. We built the attainment
classifier from the LLC’s auxiliary task that predicts (via classification)
how far in the future the goal is. The option terminates when class $0$ (“the
goal is attained now”) is the most likely class predicted by the time-to-goal
classifier.

The fixed initiation criterion is to allow any goal in any situation (Bacon et
al., 2017). However, this is problematic with learned goal-conditioned
behavior because it is possible to task the LLC with attaining goals that
cannot be attained—either because the goals are implausible, or because the
LLC is incapable. When the HLC requests an unattainable goal, the LLC will
likely run until timeout, which has a significant cost in environment
interactions. We observed this to be very problematic in early training, as
the HLC can frequently select unattainable goals, but is oblivious of the
sample cost of doing so.

We addressed this issue by terminating options after one step if the goal was
unattainable. A goal was deemed unattainable if the value estimate
$\widehat{V}^{\pi_{\mathrm{LLC}}}(b_{t},g)$ was below a certain threshold. For
a high enough threshold, this is a conservative criterion because the value
estimates $\widehat{V}^{\pi_{\mathrm{LLC}}}(b_{t},g)$ will often only be high
for goals that the LLC can achieve.

#### HLC Observations.

The LLC is responsible for what the HLC observes—it may forward environment
observations, and it may also process and combine what has been observed
during the execution of an option. In this work, the LLC simply forwards the
environment observation to the HLC on the steps where the HLC gets to observe
the SMDP and take action.

#### High-Level Controller.

We used Muesli (Hessel et al., 2021) as the general-purpose RL algorithm for
the HLC. Muesli is a strong RL agent, and it is among the strongest RL agents
in the Atari benchmark (Bellemare et al., 2013). Moreover, it admits policies
over continuous and discrete actions, and this allows us to parameterize the
policies we need for interacting with the LLC. For additional implementation
details see Appendix B.

##  4 Experiments

###  4.1 H2O2 is competitive with a strong flat baseline

We evaluated H2O2 in the DeepMind Hard Eight suite (Gulcehre et al., 2019).
These tasks are 3D, partially observable, procedurally generated, and they
require exploration and complex behavior to solve (see Gulcehre et al. (2019)
for detailed task descriptions). The agents are trained in a multi-task
setting in all eight tasks. The flat baseline is a Muesli agent, and it has
only the minimal, necessary differences from the HLC’s Muesli agent—for
example, the action spaces differ between H2O2 and the flat baseline, so the
policies need to be changed accordingly. Unless otherwise stated, all
quantities reported in this work are binned over number of frames, and
averaged across all tasks and over five independent runs. Bands show standard
error over independent runs.

Figure 2 shows the average return per episode of H2O2 and the flat baseline as
a function of the number of frames generated by interacting with the
environment (i.e., _throughout training_).

Figure 2: Average episode return for H2O2 and the Muesli baseline.

The plot shows that the two agents are competitive, with H2O2 attaining
slightly higher performance more frequently. We report per-task performance in
Fig. 9 in Section C.1, where we can see different variations of sample
efficiency and final performance between the two agents across tasks.

H2O2’s improved performance is a demonstration of the effectiveness of our
hierarchical agent, but the variations between H2O2 and the flat baseline
performances in each task suggest that H2O2 is indeed learning differently
from the flat baseline. How is the hierarchical design influencing H2O2’s
learning and final performance? What HRL capabilities is H2O2 demonstrating?

###  4.2 Is H2O2 using temporally extended behaviors?

Yes, but we did not observe that “the more temporal extension, the better”. We
found that parameters that control temporal extension have to be carefully
selected in order to obtain better performance and even, “paradoxically”,
temporally extended behavior. The apparent paradox stems from considering the
benefits of increasing temporal extension, without accounting for how it
impacts the learning problem. That is, an effective hierarchical agent with
more temporally extended behavior is expected to perform at least as well as
one with less temporally extended behavior, but giving an untrained agent
access to more temporally extended behavior may make the learning problem
harder. The problem may be so hard that even after significant training the
learning agent may have subpar performance and it may fails to display any
meaningful hierarchical behavior.

To substantiate our claim, we measured the average number of environment (LLC)
steps per SMDP (HLC) step, as well as task performance, of different variants
of H2O2. Higher values of “average LLC steps” (per HLC step) mean H2O2 spent
more timesteps in temporally extended behavior. An agent that exclusively
executed primitive actions would have a ratio of $1$. This ratio allows us to
infer the fraction of steps spent executing options excluding the first step,
which for the purpose of our discussion is how much temporally extended
behavior an agent displays.

The typical range for the the average LLC steps is between $1.0$ and $2.5$. An
average LLC steps ratio of $1.5$ means the agent is in temporally extended
behavior for about $\frac{1}{3}$ of its interaction with the base environment.
A ratio of $1.25$ corresponds to temporal extension in at least $20\%$ of the
interaction, and a ratio of $2.5$ is corresponds to at least $60\%$.

#### H2O2 with different option timeouts.

We considered variants of H2O2 with different timeouts: $7$, $16$ and $32$
steps (the timeout used for H2O2 in Fig. 2 was $7$). Options terminate when
either the goal is attained (according to the LLC’s classifier) or at timeout.
Because the options have a termination function, we would expect that
increasing timeout should increase the effectiveness and frequency of the
agent’s temporally extended behavior. Figure 3(a) shows, however, that this is
not the case.

(a)

(b)

Figure 3: Evaluation of H2O2 with different option timeouts. (3(a)) Average
number of LLC steps per HLC steps throughout training All curves are at
approximately $1.0$ in the initial training steps; the apparent start of some
curves at higher values is an artifact of binning. (3(b)) Average return of
H2O2 for different option timeouts.

Surprisingly, attempting to increase the amount of temporal abstraction by
increasing timeouts eventually _harms_ H2O2’s ability to employ temporally
extended behavior. Moreover, H2O2’s performance is surprisingly sensitive to
the amount of temporal abstraction: even with a timeout of $16$ (which has
roughly the same amount of temporally extended behavior (Fig. 3(a)) as the
timeout of $7$), H2O2’s performance is worse than with a timeout of $7$ (see
Fig. 3(b)). The performance with the timeout of $32$ is worst, so this setting
leads to poor behavior both in terms of temporal abstraction and task
performance.

Our data suggests that H2O2 with a timeout of $32$ breaks down because the
learning problem is too hard. We measured why and how often options terminate
throughout training (see Fig. 10), and we saw that in this setting our agent
spends about half of the first $2\cdot 10^{9}$ frames issuing invalid goals
(and thus generating transitions with no-ops). So we suspect that the HLC
failed to generate “good” data for the LLC to learn effective goal-following
behavior, which in turn led to an unnecessarily challenging SMDP for the HLC
to solve, that is, one filled with useless options that waste several frames
of environment interaction.

#### H2O2 with different discounts.

We also considered three variants of H2O2 with different discounts $\gamma$,
in $\\{0.9,0.99,0.997\\}$ (the value used for H2O2 in Fig. 2 was $0.997$).
Since the discounting only incides on HLC timesteps, the rewards $n$ timesteps
in the future will only be discounted by $\gamma^{n}$, even if the amount of
primitive actions required to get to that state is significantly larger. For
example, if options take an average of $1.5$ steps, $\gamma=0.9$ and $n=10$,
the reward for HLC would be discounted with $0.59$, whereas a flat agent
executing the same actions would see the reward discounted with $0.21$. This
“horizon shortening” is expected to encourage the agent to use options, so we
expect to see the variants with smaller $\gamma$ using more options. Figure
4(a) shows that this is indeed the case.

(a) Average number of LLC steps per option.

(b) Average return.

Figure 4: Option use (left) and performance (right) for H2O2 with different
discount factors $\gamma$.

Figure 4(b) shows that as we decrease $\gamma$ the agent spends more time in
temporally extended behavior. HRL folklore suggests that the agent with more
temporally extended behavior will perform better, because it will be able to
assign credit at longer timescales. The results in Fig. 4(b) are evidence
against this claim: The plot shows that H2O2 with the largest $\gamma$
performs best, and that decreasing $\gamma$ worsens perfomance (even though it
increases option use, as shown in Fig. 4(a)). The issue is that changing
$\gamma$ also affects the objective of the HLC, and that changing the
objective can change both the final solution and how the agent explores. Our
hypothesis is that H2O2 with lower $\gamma$ explores worse, possibly in three
ways: 1) The HLC fails to generate behavior with higher rewards because it is
optimizing for short-term cumulative reward; 2) The options learned from the
poor-performing HLC are also poor (with respect to the task), and the agent is
incentivized to choose poor-performing options over exploring further with
primitive actions; 3) The longer options also reduce the amount of training
data the HLC generates for itself; that is, the HLC is incentivized to
generate less data for itself by using options. This third point makes the
agent sample inefficient!

Section C.3 shows that H2O2 outperforms a simpler idea of adding temporal
abstraction to a flat baseline by increasing the number of repeated actions.

###  4.3 Does H2O2 benefit from more options?

Sometimes, and we argue that it depends on whether the options _simplify the
problem_.

HRL folklore suggests that making more skills available to the HLC empowers
the agent and leads to better solutions. We claim that this is not necessarily
the case, and that in H2O2, for learned options to be beneficial, they must
_simplify_ the problem for the HLC. That is, it does not make sense to solve
an SMDP that is harder to solve than the original MDP—in that case we are
better off using the flat agent. We considered two ways to offer more options
to H2O2: Increasing the dimension of the latent goals, and reducing the amount
of regularization on the goal space.

Figure 5(a) shows the performance of H2O2 where we varied the dimension of the
latent goals. We considered dimensions in $\\{16,32,48\\}$, and H2O2 from Fig.
2 uses $32$.

(a) Average return for H2O2 with different latent goal dimensions.

(b) Average return for H2O2 with different amounts of goal compression. The
higher $\beta$, the more compression there is.

Figure 5: Effect of constraining the space of options on overall performance.

We see in the figure that this dimension behaves like a usual hyperparameter:
There’s a sweetspot for highest performance at $32$, but going lower or higher
leads to worse performance. It is surprising, though, that a dimension of $32$
works best, as we were initially expecting more expressive goals to be more
effective.

We also evaluated the effect the Goal Encoder regularizer on the performance
of H2O2. We considered $\beta\in\\{10^{-9},10^{-3},10^{-2},10^{-1},1\\}$. When
$\beta=10^{-9}$ there is no compression of the goal space, but when $\beta=1$
the regularization is so strong that the posterior is also a standard normal.
We used $\beta=1$ for H2O2 in Fig. 2.

Figure 5(b) shows the performance of H2O2 for the different values of $\beta$,
and we see that H2O2 with the least diverse set of options ($\beta=1$)
performs best, along with the larger values of $\beta$. The data is consistent
with the hypothesis that too much flexibility in the goal space makes the
learning problem harder, so adding more options eventually damages the
performance of the agent.

###  4.4 How does H2O2 perform in similar domains from related work?

The work of Hafner et al. (2022) is the closest to ours: They introduced
Director, a hierarchical deep RL agent for complex partially observable,
first-person 3D environments. The was evaluated in Goals Small and Objects
Small in the DeepMind Lab environment (Beattie et al., 2016). These are first-
person maze tasks that require navigation, localization and recalling reward
locations within an episode. Director is an SMDP agent with call-and-return
options, but without access to primitive actions, and all options terminate
after a fixed number of steps. The options are “state”-directed, in the sense
that the LLC (“Worker”) is conditioned on latent vectors from the latent state
space (the analogue of our $b_{t}$ in Fig. 1), and is trained in hindsight for
achieving the corresponding latent states. Hafner et al. (2022) use a VQ-VAE
(Van Den Oord et al., 2017; Razavi et al., 2019) to discretize the latent
state space, which gives the HLC (”Manager”) a discrete action space.
Moreover, they use a World Model (Hafner et al., 2019b) to help shape
Director’s state representation. During training, the LLC in Director is
rewarded proportionally to the similarity of its latent state and the
conditioning input (the goal), at each timestep. The version of Director that
is competitive with their baseline (Dreamer; Hafner et al., 2019a) adds
extrinsic rewards to the reward provided to the LLC.

We evaluated H2O2 and our flat Muesli baseline in DeepMind Lab’s Goals Small
and Objects Small. Figure 6 shows the average return of different agents on
the two tasks. The “Flat Baseline” is a Muesli agent like the one used in the
Hard Eight tasks, but uses a replay-to-online ratio of $0.9$222This ratio
means that in each minibatch $90\%$ of the data is sampled from a replay, and
the other $10\%$ from online experience. This increases the data efficiency of
the agents and make them competitive in early training (the $180$-thousand-
frame regime). We present variants of H2O2 with the replay-to-online ratio
used for Hard Eight tasks ($0.5$) as well as $0.9$. The figure also shows the
final performance of Director and Dreamer (as the dotted line, both methods
have the same final performance). This variant of Director adds extrinsic
rewards to the LLC objective.

Figure 6: The average return across two levels of DMLab Beattie et al. (2016).
We also indicate the final performance of the Dreamer and Director baselines
Hafner et al. (2022) with the dotted line, after 50M frames.

Figure 6 shows that with an appropriate replay-to-online ratio both H2O2 and
Muesli baseline can match the data efficiency of Director and Dreamer, though
it’s unclear what the latter’s final performance would be if trained longer.

##  5 Conclusion

Our work introduces H2O2, the first demonstration of a complete hierarchical
agent that can use non-trivial options to attain strong performance in
visually complex partially observable tasks. H2O2 is competitive with a state-
of-the-art flat baseline, it discovers and learns to use its options, and it
does so from its own generated experience.

#### Relevance.

HRL has received much interest due to its potential to deliver powerful agents
with appealing capabilities—for example, transfer, skill reuse, planning and
exploration with abstraction. Recent successes with HRL in different domains
(Merel et al., 2019; Wulfmeier et al., 2021; Hafner et al., 2022) provide
evidence that practical, effective HRL agents are possible, even if existing
agents do not yet fully realize the potential of HRL. Therefore, it is
important to expand the coverage of effective hierarchical agents across
domains, and to identify and tackle practical challenges that can bring us
closer to a full-fledged hierarchical agent.

#### Significance.

Our work is an important contribution to the HRL research for two reasons:
H2O2 is a proof of concept, complete and effective HRL agent, and our work
highlights critical challenges for HRL in complex domains that are vastly
overlooked in the literature. It was only by going through the process of
designing and training a HRL agent for complex domains that we exposed some of
these issues.

#### Successes.

We built on existing work to tackle some of the practical challenges of HRL in
complex domains, such as learning goal-conditioned behaviors offline from any
experience generated by an agent (not jut expert behavior). To achieve this,
we introduced a regularized offline V-Trace algorithm and demonstrated how to
integrate the policy that executes these goal-conditioned behaviors (the LLC)
with a general-purpose RL algorithm that learns to select these behaviors as
options in order to solve tasks (the HLC).

#### Lessons learned.

We believe that our empirical findings apply to domains where a very large
number of options is conceivable, but for any one task a much smaller set of
behaviors is relevant and useful. Visually complex domains tend to naturally
have this property, and this is arguably the kind of rich domain we want
intelligent agents to be effective in. However, we think that many of the
challenges we observed would go away if we were to limit the learning to a
small set of options (Merel et al., 2019; Wulfmeier et al., 2021), or choose
them sensibly beforehand.

Within the scope of these “rich domains”, however, the lessons we can draw
from our experimental results can apply to various HRL agents beyond H2O2. The
lessons apply most closely to SMDP agents. The SMDP framework has been backed
with theoretical justification(Precup, 2000; Barto and Mahadevan, 2003), and
our work complements existing knowledge with empirical findings.

We noticed a strong contrast between how HRL is typically motivated in the
literature (e.g., Barto and Mahadevan, 2003; Pateria et al., 2021; Hutsebaut-
Buysse et al., 2022), and the practical challenges we encountered. It is often
claimed that hierarchical agents can demonstrate very appealing capabilities
and algorithmic strengths, such as sample efficiency, structured exploration,
temporal abstraction, improved credit assignment, state abstraction, jumpy
planning, transfer and generalization. These “HRL promises” can easily be
misconstrued as properties of hierarchical agents, which may lead to
misconceptions about how hierarchical agents will learn and perform.

Our empirical findings exposed some of these HRL misconceptions. For example,
the SMDP approach promises to simplify the problem for the general-purpose RL
algorithm. So one might expect that adding capabilities that are perceived as
strengths of HRL (for example, more expressive options) to the SMDP will cause
the general-purpose RL algorithm to solve the SMDP with less effort than if it
had simpler options, or only primitive actions. However, in some experiments
we showed the opposite.

In practice, the design of the LLC effectively changes the SMDP, and the
hierarchical agent can only be competitive with a flat agent if SMDP is easier
to solve than the original MDP (besides admitting a better solution).
Therefore both solution quality and learning dynamics are essential factors to
consider when designing the hierarchical agent.

#### Open challenges.

We also identified questions that remain open: How can we structure the goal
space to accelerate the HLC learning? Is it possible to learn effective HLCs
with a general-purpose RL algorithms? How can the HLC agent learn with a very
large number of complex options, but remain competitive with a flat baseline?
Are image goals good enough? What other goal modalities can we use? Which
goals should we train the LLC to achieve?

Some of these questions can be investigated in simple domains, as long as the
domains are designed to pose challenges that we observe in practice. For
example, a simple grid-world where there is an option to reach any cell from
any other cell can be a fruitful domain to explore. However, it may be
challenging to outperform strong flat deep RL baselines in such simple domains
if the options are not prescribed but learned end to end.

We presented simple approaches for some of the challenges above—e.g. the goal
sampling distribution for the LLC. We expect that the performance of H2O2 will
improve with goal sampling distributions that incorporate principled
techniques for option discovery (Machado and Bowling, 2016). H2O2 can be a
starting point for research that aims to investigate specific HRL sub-problems
without losing sight of the performance of the whole agent in complex tasks.

## References

  * Ajay et al. (2020) A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum.  Opal: Offline primitive discovery for accelerating offline reinforcement learning.  _arXiv preprint arXiv:2010.13611_ , 2020.
  * Andrychowicz et al. (2017) M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba.  Hindsight experience replay.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * Bacon et al. (2017) P.-L. Bacon, J. Harb, and D. Precup.  The option-critic architecture.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31, 2017.
  * Barreto et al. (2019) A. Barreto, D. Borsa, S. Hou, G. Comanici, E. Aygün, P. Hamel, D. Toyama, J. Hunt, S. Mourad, D. Silver, and D. Precup.  The option keyboard combining skills in reinforcement learning.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , pages 13052–13062, 2019.
  * Barto and Mahadevan (2003) A. G. Barto and S. Mahadevan.  Recent advances in hierarchical reinforcement learning.  _Discrete event dynamic systems_ , 13(1):41–77, 2003.
  * Beattie et al. (2016) C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik, J. Schrittwieser, K. Andeson, S. York, M. Cant, A. Cain, A. Bolton, S. Gaffney, H. King, D. Hassabis, S. Legg, and S. Petersen.  Deepmind lab.  _arXiv preprint arXiv:1612.03801_ , 2016.
  * Bellemare et al. (2013) M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , 47:253–279, 2013.
  * Cassandra et al. (1994) A. R. Cassandra, L. P. Kaelbling, and M. L. Littman.  Acting optimally in partially observable stochastic domains.  In _Aaai_ , volume 94, pages 1023–1028, 1994.
  * Cassirer et al. (2021) A. Cassirer, G. Barth-Maron, E. Brevdo, S. Ramos, T. Boyd, T. Sottiaux, and M. Kroiss.  Reverb: A framework for experience replay, 2021.
  * Dayan and Hinton (1992) P. Dayan and G. E. Hinton.  Feudal reinforcement learning.  _Advances in neural information processing systems_ , 5, 1992.
  * Ernst et al. (2005) D. Ernst, P. Geurts, and L. Wehenkel.  Tree-based batch mode reinforcement learning.  _Journal of Machine Learning Research_ , 6:503–556, 2005\.
  * Espeholt et al. (2018) L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu.  Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.  In _International Conference on Machine Learning_ , pages 1407–1416. PMLR, 2018.
  * Eysenbach et al. (2018) B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine.  Diversity is all you need: Learning skills without a reward function.  In _International Conference on Learning Representations_ , 2018.
  * Fu et al. (2020) J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Fujimoto et al. (2019) S. Fujimoto, D. Meger, and D. Precup.  Off-policy deep reinforcement learning without exploration.  In K. Chaudhuri and R. Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2052–2062. PMLR, 09–15 Jun 2019.
  * Gregor et al. (2017) K. Gregor, D. J. Rezende, and D. Wierstra.  Variational intrinsic control.  In _International Conference on Learning Representations_ , 2017.
  * Gulcehre et al. (2019) C. Gulcehre, T. Le Paine, B. Shahriari, M. Denil, M. Hoffman, H. Soyer, R. Tanburn, S. Kapturowski, N. Rabinowitz, D. Williams, G. Barth-Maron, Z. Wang, N. de Freitas, and Worlds Team.  Making efficient use of demonstrations to solve hard exploration problems.  In _International conference on learning representations_ , 2019.
  * Gulcehre et al. (2020) C. Gulcehre, Z. Wang, A. Novikov, T. Paine, S. Gómez, K. Zolna, R. Agarwal, J. S. Merel, D. J. Mankowitz, C. Paduraru, G. Dulac-Arnold, J. Li, M. Norouzi, M. Hoffman, N. Heess, and N. de Freitas.  Rl unplugged: A suite of benchmarks for offline reinforcement learning.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 7248–7259. Curran Associates, Inc., 2020.
  * Gulcehre et al. (2021) C. Gulcehre, S. G. Colmenarejo, Z. Wang, J. Sygnowski, T. Paine, K. Zolna, Y. Chen, M. Hoffman, R. Pascanu, and N. de Freitas.  Regularized behavior value estimation.  _arXiv preprint arXiv:2103.09575_ , 2021.
  * Gupta et al. (2020) A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman.  Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning.  In _Conference on Robot Learning_ , pages 1025–1037. PMLR, 2020.
  * Hafner et al. (2019a) D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi.  Dream to control: Learning behaviors by latent imagination.  _arXiv preprint arXiv:1912.01603_ , 2019a.
  * Hafner et al. (2019b) D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson.  Learning latent dynamics for planning from pixels.  In _International conference on machine learning_ , pages 2555–2565. PMLR, 2019b.
  * Hafner et al. (2022) D. Hafner, K.-H. Lee, I. Fischer, and P. Abbeel.  Deep hierarchical planning from pixels.  _arXiv preprint arXiv:2206.04114_ , 2022.
  * Harutyunyan et al. (2019) A. Harutyunyan, W. Dabney, D. Borsa, N. Heess, R. Munos, and D. Precup.  The termination critic.  _arXiv preprint arXiv:1902.09996_ , 2019.
  * Hessel et al. (2019) M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki, S. Schmitt, and H. van Hasselt.  Multi-task deep reinforcement learning with popart.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 3796–3803, 2019.
  * Hessel et al. (2021) M. Hessel, I. Danihelka, F. Viola, A. Guez, S. Schmitt, L. Sifre, T. Weber, D. Silver, and H. van Hasselt.  Muesli: Combining improvements in policy optimization.  _arXiv preprint arXiv:2104.06159_ , 2021.
  * Horgan et al. (2018) D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. van Hasselt, and D. Silver.  Distributed prioritized experience replay.  In _International Conference on Learning Representations_ , 2018.
  * Hutsebaut-Buysse et al. (2022) M. Hutsebaut-Buysse, K. Mets, and S. Latré.  Hierarchical reinforcement learning: A survey and open research challenges.  _Machine Learning and Knowledge Extraction_ , 4(1):172–221, 2022.
  * Islam et al. (2019) R. Islam, Z. Ahmed, and D. Precup.  Marginalized state distribution entropy regularization in policy optimization.  _arXiv preprint arXiv:1912.05128_ , 2019.
  * Khazatsky et al. (2021) A. Khazatsky, A. Nair, D. Jing, and S. Levine.  What can i do here? learning new skills by imagining visual affordances.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 14291–14297. IEEE, 2021.
  * Kingma and Welling (2013) D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Lange et al. (2012) S. Lange, T. Gabel, and M. Riedmiller.  Batch reinforcement learning.  In _Reinforcement learning_ , pages 45–73. Springer, 2012.
  * Lee et al. (2019) L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov.  Efficient exploration via state marginal matching.  _arXiv preprint arXiv:1906.05274_ , 2019.
  * Levine et al. (2020) S. Levine, A. Kumar, G. Tucker, and J. Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Lynch and Sermanet (2020) C. Lynch and P. Sermanet.  Grounding language in play.  _arXiv preprint arXiv:2005.07648_ , 2020.
  * Lynch et al. (2020) C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet.  Learning latent plans from play.  In _Conference on Robot Learning_ , pages 1113–1132. PMLR, 2020.
  * Machado and Bowling (2016) M. C. Machado and M. Bowling.  Learning purposeful behaviour in the absence of rewards.  _arXiv preprint arXiv:1605.07700_ , 2016.
  * Mathieu et al. (2021) M. Mathieu, S. Ozair, S. Srinivasan, C. Gulcehre, S. Zhang, R. Jiang, T. Le Paine, K. Zolna, R. Powell, J. Schrittwieser, D. Choi, P. Georgiev, D. Toyama, A. Huang, R. Ring, I. Babuschkin, T. Ewalds, M. Bordbar, S. Henderson, S. G. Colmenarejo, W. M. van den Oord, Aäron Czarnecki, N. de Freitas, and O. Vinyals.  Starcraft ii unplugged: Large scale offline reinforcement learning.  In _Deep RL Workshop NeurIPS 2021_ , 2021.
  * Mendonca et al. (2021) R. Mendonca, O. Rybkin, K. Daniilidis, D. Hafner, and D. Pathak.  Discovering and achieving goals via world models.  _Advances in Neural Information Processing Systems_ , 34:24379–24391, 2021.
  * Merel et al. (2019) J. Merel, L. Hasenclever, A. Galashov, A. Ahuja, V. Pham, G. Wayne, Y. W. Teh, and N. Heess.  Neural probabilistic motor primitives for humanoid control.  In _International Conference on Learning Representations_ , 2019.
  * Mnih et al. (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis.  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533, 2015.
  * Nachum et al. (2018) O. Nachum, S. S. Gu, H. Lee, and S. Levine.  Data-efficient hierarchical reinforcement learning.  _Advances in Neural Information Processing Systems_ , 31:3303–3313, 2018.
  * Nachum et al. (2019) O. Nachum, H. Tang, X. Lu, S. Gu, H. Lee, and S. Levine.  Why does hierarchy (sometimes) work so well in reinforcement learning?  _arXiv preprint arXiv:1909.10618_ , 2019.
  * Pateria et al. (2021) S. Pateria, B. Subagdja, A.-h. Tan, and C. Quek.  Hierarchical reinforcement learning: A comprehensive survey.  _ACM Computing Surveys (CSUR)_ , 54(5):1–35, 2021\.
  * Pomerleau (1989) D. A. Pomerleau.  Alvinn: An autonomous land vehicle in a neural network.  In D. Touretzky, editor, _Advances in Neural Information Processing Systems_ , volume 1. Morgan-Kaufmann, 1989.
  * Pong et al. (2020) V. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine.  Skew-fit: State-covering self-supervised reinforcement learning.  In _International Conference on Machine Learning_ , pages 7783–7792. PMLR, 2020.
  * Precup (2000) D. Precup.  _Temporal abstraction in reinforcement learning_.  University of Massachusetts Amherst, 2000.
  * Razavi et al. (2019) A. Razavi, A. Van den Oord, and O. Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  _Advances in neural information processing systems_ , 32, 2019.
  * Rezende et al. (2014) D. J. Rezende, S. Mohamed, and D. Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In _International conference on machine learning_ , pages 1278–1286. PMLR, 2014.
  * Shah et al. (2022) D. Shah, P. Xu, Y. Lu, T. Xiao, A. Toshev, S. Levine, and B. Ichter.  Value function spaces: Skill-centric state abstractions for long-horizon reasoning.  In _International Conference on Learning Representations_ , 2022.
  * Silver et al. (2018) D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  _Science_ , 362(6419):1140–1144, 2018.
  * Solway et al. (2014) A. Solway, C. Diuk, N. Córdova, D. Yee, A. G. Barto, Y. Niv, and M. M. Botvinick.  Optimal behavioral hierarchy.  _PLoS computational biology_ , 10(8):e1003779, 2014.
  * Sutton and Barto (2018) R. S. Sutton and A. G. Barto.  _Reinforcement Learning: An Introduction_.  A Bradford Book, Cambridge, MA, USA, 2018.  ISBN 0262039249.
  * Sutton et al. (1999) R. S. Sutton, D. Precup, and S. Singh.  Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.  _Artificial intelligence_ , 112(1-2):181–211, 1999.
  * Van Den Oord et al. (2017) A. Van Den Oord, O. Vinyals, and K. Kavukcuoglu.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Vezhnevets et al. (2017) A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu.  Feudal networks for hierarchical reinforcement learning.  In _International Conference on Machine Learning_ , pages 3540–3549. PMLR, 2017.
  * Ward et al. (2020) T. Ward, A. Bolt, N. Hemmings, S. Carter, M. Sanchez, R. Barreira, S. Noury, K. Anderson, J. Lemmon, J. Coe, P. Trochim, T. Handley, and A. Bolton.  Using Unity to help solve intelligence, 2020.
  * Wulfmeier et al. (2021) M. Wulfmeier, D. Rao, R. Hafner, T. Lampe, A. Abdolmaleki, T. Hertweck, M. Neunert, D. Tirumala, N. Siegel, N. Heess, and M. Riedmiller.  Data-efficient hindsight off-policy option learning.  In _International Conference on Machine Learning_ , pages 11340–11350. PMLR, 2021.

##  Appendix A SMDP Agent Design Details

An RL agent interacts with a semi-MDP (SMDP; Sutton et al., 1999) by selecting
options (temporally extended behaviors with initiation and termination
conditions), whereas an RL agent interacting with an MDP (Sutton and Barto,
2018) selects (primitive) actions. Figure 7 outlines these interaction loops,
which are intentionally similar.

Figure 7: Agent-environment interaction diagrams for MDPs (left) and SMDPs
(right).

We can turn an MDP into an SMDP by adding options to it. When we learn the
options with RL, it is useful to structure the agent interaction in terms of a
high-level controller (HLC) which interacts with the SMDP, and a low-level
controller (LLC), which interacts with the MDP and provides the options for
the SMDP. Figure 8 outlines the interaction loops for the LLC and the HLC.

Figure 8: Agent-environment interaction diagrams for an SMDP, framed in terms
of high-level and low-level controllers.

The diagram corresponds to how we structured the HLC and LLC in H2O2.

The LLC interacts directly with the MDP (the agent-MDP interaction in the left
diagram of Fig. 8, in red). It is responsible for changing its behavior in
response to the options received, and for providing the SMDP states and
rewards for the HLC, based on the MDP states and rewards it observes.

The LLC executes options in the call-and-return model (Dayan and Hinton,
1992): The HLC selects an option, then the LLC executes the option until it
terminates, then returns control to the HLC. From the perspective of the HLC,
the interaction with the environment is an agent-SMDP interaction (the right
diagram in Fig. 8, in dark blue).

In our work, the LLC allows the HLC to select primitive actions (Sutton et
al., 1999), which are implemented as one-step options where the LLC executes
the action requested by the HLC. Our LLC forwards environment observations to
the HLC along with a “termination reason”. The termination reason is a flag
indicating why the LLC execution terminated, and it can be one of the
following:

  * •

First timestep: The first timestep in an MDP episode requires the HLC to
select an action in the SMDP for the LLC to perform.

  * •

Last timestep: The last timestep in an MDP episode is also the last timestep
in the corresponding SMDP.

  * •

The option terminated. In this case the termination reason encodes why the
option terminated. This could be because the goal was attained, the option
timed out, or because it was a primitive action (which is framed as a one-step
option).

  * •

The option failed to initiate. This is because the goal corresponding to the
option is deemed unattainable. The LLC executes a no-op in the environment and
terminates. The forced no-op is an implementation decision to prevent the HLC
from selecting unattainable goals indefinitely.

##  Appendix B Implementation details

###  B.1 Goal Sampling Distribution

We rejected “similar” goals as follows: In a trajectory of potential goals
$g_{1},\ldots,g_{n}$, we first identify, for each candidate goal $g_{t}$, the
closest that any other goal is:

$\mathrm{sim}_{t}\doteq\max_{t^{\prime}}\frac{\langle
g_{t},g_{t^{\prime}}\rangle}{\|g_{t}\|_{2}\|g_{t^{\prime}}\|_{2}}.$

Then we exclude all pairs $(t_{s},t_{e})$ from sampling, such that
$\mathrm{sim}_{t_{e}}$ is larger than the $40$-percentile of
$(\mathrm{sim}_{1},\ldots,\mathrm{sim}_{n})$.

###  B.2 Low-Level Controller

#### Image Observations and Goals.

Image observations and image goals were 72 x 96 RGB first-person views of the
environment, encoded with a ResNet followed by an MLP (Hessel et al., 2019).
For detailed information on the various model hyper-parameters, refer to Table
1.

#### Vector Goals and Goal Distributions for the affordance model.

Throughout this work, we used a goal space defined as
$\mathcal{G}=[-1,1]^{32}$. We explained how the this goal space was modelled
as a multivariate Normal with independent components in Section 3.1. For the
affordance model on the other hand, we used a different parameterization of
the goal distribution that is more flexible and expressive than a Normal
distribution. Specifically, we parameterized the goal distributions with
categorical random variables over the discretized $[-1,1]$ range for each
dimension, similar to how we parameterized the continuous actions.

#### Policy.

The policy used in the LLC was the same as that used for the primitive actions
in the HLC, which is that used in Muesli (Hessel et al., 2021).

Table 1:  Training hyper-parameters for the LLC. Network |
---|---
Image encoder (Observation) | ResNet with SAME padding
ResNet channels | $(16,32,32)$
ResNet residual blocks | $(2,2,2)$
ResNet kernel sizes | $(3,3,3)$
ResNet kernel strides | $(1,1,1)$
ResNet pool sizes | $(3,3,3)$
ResNet pool strides | $(2,2,2)$
MLP sizes | $(512,512)$
Proprioceptive encoder MLP sizes | $(256,64)$
State LSTM size | $512$
Policy conditioning | Concatenation
Policy LSTM size | $512$
Policy Head MLP shapes | $(512,256,128,196)$
Number of components per continuous actions | $8$
Policy Distribution for continuous actions | Mixture of Clipped Logistics
Policy Distribution for discrete actions | Categorical
Value Head MLP shapes | $(512,256,128,1)$
Number of actions | $10$
Goal Encoder |
Goal ResNet Encoder | Same as Observation ResNet
Goal MLP sizes | $(512,512,256,256,32)$
Goal activation | LayerNorm + tanh activation
Auxiliary Losses |
Behavioral Policy Head MLP | $(512,256,128,196)$
Goal Model MLP | $(512,256,128)$
Goal Model Distribution | Binned bounded range per dim.
Number of bins per dimension | $21$
External State Value MLP | $(256,256,1)$
Attainment Predictor LSTM size | $256$
Attainment Predictor MLP sizes | $(256,256,64)$
Losses |
V-Trace discount factor $\gamma$ | $0.8$
$\widehat{V}^{\pi_{\mathrm{LLC}}}$ objective weight | $0.5$
KL-Regularizer (to Behavior Policy) weight ($\alpha$ in Eq. 1) | $10^{-2}$
Neg-entropy regularizer ($-H(\pi_{\mathrm{LLC}})$) weight | $10^{-3}$
$\rho$ clipping threshold | $1$
External state value discount factor | $0.99$
Optimizer | Adam
Learning Rate | $10^{-4}$
Global Norm clipping | $10000$
Batch size | $16$
Unroll Length | $64$
Action repeat | $4$

###  B.3 High-Level Controller

For the HLC in this work, we used a Muesli agent with adaptations to the
policy, so that it could select high-level actions (see Section 3.2). The
primitive observations were processed as in the original Muesli agent, and the
LLC observations were treated as scalar and vector observations and processed
like other scalar and vector observations in Muesli—that is, input to an MLP
and the result concatenated to the other processed observations. For specific
hyper-parameters different from those used by Hessel et al. (2021), refer to
Table 2.

#### Vector Goal Policy.

To select vector goals in the HLC, we augmented the action space of the Muesli
agent (Hessel et al., 2021) with 32 extra continuous actions, the same
dimension as our continuous goal space $G\in\mathcal{G}$. We also added a
Bernoulli distribution for choosing between primitive actions and options as
well as additional actions for the LLC hyperparamters set by the HLC, as
described in Section 3.2.

Table 2:  Training hyper-parametersfor the HLC and Flat Muesli baseline. The only difference is the number of actions, where for the baseline there are only the primitive actions. For parameters we do not provide values to, refer to Hessel et al. (2021). Replay proportion in batch | $0.5$
---|---
Discount factor | $0.997$
Optimizer | Adam
Learning rate | $3\cdot 10^{-4}$
Model unroll length | $2$
Policy weight | $3$
Auxiliary weight | $1$
Policy Distribution for discrete actions | Categorical
Policy Distribution for continuous actions | Mixture of Clipped Logistics
Number of components per continuous action | $8$
Number of actions |  $43$, factored as
|  $\quad 1$ (execute option or primitive action, binary)
|  $\quad 32$ (goal, continuous)
|  $\quad 10$ (primitive action, $8$ continuous, $2$ binary)
Action repeat | $4$

##  Appendix C Additional Results

###  C.1 Exploring the HLC-LLC Interface and per-level performance

It was very surprising to find out that the best H2O2 performance came from
the agent with maximum goal space compression, so we wanted to further
investigate whether completely removing the goal space and allowing the LLC to
execute an unconditional reward-maximising policy would perform at a similar
level. The hypothesis is that since the goal space is strongly regularized
towards a standard Normal, the _capacity_ of the goal space should be quite
low, and hence not much information should propagate through to the LLC. To
test this hypothesis, we implemented two variants of H2O2 with unconditional
LLCs.

The first one is "Autopilot (BC)", where only the Behavioral Cloning policy,
$\mu(a_{t}|b_{t})$ (notice how the dependence of the goal $g$ is dropped), is
used for acting and trained by maximising the log-likelihood of actions. These
action come from trajectories drawn from the replay and are generated while
the full agent is acting. We note that there is no special goal selection or
external reward used to train the "Autopilot (BC)". This LLC variant is simply
learning to re-generate experience it has already seen, and the HLC is only
requred to select when to deploy the LLC, rather than selecting which goals it
should attend.

The second variant is "Autopilot (Vtrace)", and is trained with the same
offline RL method described in Section 3.1 (Offline RL paragraph), but without
the conditioning on the goal (see Eq. 1). This means that the policy of the
LLC is only conditioned on the current recurrent state, $b_{t}$, as all
dependencies on a goal, $g$, are dropped. The data is drawn similar to
"Autopilot (BC)", but now the Advantage is calculated based on the external
reward. For this reason, "Autopilot (Vtrace)" is actually trained using
offline data to solve the task directly, and the HLC simply chooses when to
deploy the LLC.

We compare their performance on a per-task basis with the best performing H2O2
variant used in the main report and the Flat baseline in Fig. 9. As we can
see, both of these variants with unconditional, reward-maximising LLCs are
performing worse than H2O2 on 3 of the tasks, and about as well on the other
3. In addition, "Autopilot (Vtrace)" performs better than the "Autopilot (BC)"
on 4 out of 6 tasks, with the latter matching performance on the remaining
two. This is not very surprising as the V-trace loss allows us to consume sub-
optimal, off-policy data, whereas Behavioral Cloning is expected to work best
when using predominantly expert data.

What is somewhat surprising though, is that there is a significant difference
between the H2O2 variant with a strongly regularized goal space and an
unconditional LLC. This means that even though the goal space is strongly
regularized, it is still able to capture enough information and propagate it
to the LLC policy. An additional final reason of the much lower performance is
becayse none of the two Autopilot variants make use of any of the auxiliary
losses mentioned in Section 3.1, which could mean that they play a significant
role in shaping the LLC network.

Figure 9: Average episode return per task for different agents. We only show
performance on the 6 tasks that agents made some progress, and exclude the
results on Throw Across and Remember Sensor, where no agent made any progress
on. Figure 10: Proportion of LLC execution termination reasons as training
progresses.

###  C.2 Compare the number of model parameters

We cannot attribute the increased performance solely to more parameters.
Increasing the number of parameters of the flat baseline increases performance
in some of the hard eight tasks, but not every way to increase parameters is
effective. Even a flat baseline with more parameters fails to outperform H2O2
in one of the tasks.

We compared the performance of H2O2 against flat Muesli baselines with
different number of parameters333All numbers of parameters reported are
rounded to millions. in Fig. 11. The H2O2 has 46M parameters (34M on the HLC,
12M on the LLC). The flat baseline presented in the paper so far has 32M
parameters. We additionally experimented with a baseline with additional MLP
layers (“Deeper”, 35M parameters), and a baseline with wider hidden layers
(“Wider”, 68M parameters) and a baseline with similar number of parameters
(“Mix”, 48M parameters), achieved by both adding more layers and making the
hidden layers wider. Figure 11 shows the performance of the different agents
across the six tasks where performance was nontrivial, throughout training.

Figure 11: Average performance of H2O2 and flat baselines with different
number of parameters.

The “Deeper” flat baseline is an example where we increasing parameters but
achieve worse performance. The “Wider” flat baseline has more parameters than
H2O2 and matches or outperforms the flat baseline across the board. H2O2
outperforms the flat baseline in Baseball and Wall Sensor Stack. The “Wider”
baseline outperforms H2O2 in Baseball, but is outperformed in Wall Sensor
Stack. Surprisingly, the “Mix” baseline performed worst than any other agent,
even though it has the same number of parameters as H2O2.

We think that these results go to show that the number of parameters is not
the whole story, and that carefully tuning the architecture hyper-parameters,
such us number of layers and their size, can have a far greater impact.

###  C.3 Comparing against Action Repeat

Figure 12: The average return across levels for different action repeats (AR).

One could claim that the reason for H2O2’s strong performance is that it uses
options lasting about $2$ timesteps, which enable the HLC to utilize an
effective action repeat of $8$, rather than the actual action repeat of $4$
used. To examine this claim, we have run experiments with flat baselines that
instead use action repeats of $8$ and $12$.

As we can see in Fig. 12 though, the claim above is not supported by evidence,
since the flat baselines with larger action repeats perform significantly
worse. A possible reason for this is that conventional action repeats simply
repeat the same action for the specified number of steps, which means that the
agent loses the ability to execute different actions over that duration. This
can be catastrophic for many tasks that require finer motor control, and by
doing so these agent loose any benefits they might gain in terms of shorter
value backups, when learning their value function. On the contrary, since
short options seem to perform at a SOTA level, it means that the primitive
actions executed within the options are not simple action repeats, but most
likely combinations of different primitive actions, depending on the current
state of the agent.

###  C.4 Agent videos

We generated videos
444https://youtube.com/playlist?list=PLlHafZmkZCGmIYSHO9aot75l07kbiuLN0 of a
trained H2O2 agent solving the different tasks from the Hard Eight suite of
tasks. The video shows multiple episodes from the agent’s first-person view.

The frames in red correspond to the steps where the HLC selects an SMDP action
(a primitive action or a goal), whereas the other frames refer to MDP steps
where actions are selected by the LLC in response to the HLC’s SMDP action.
When the LLC is executing an option per se (i.e. following a goal), the latent
goal selected by the HLC is shown in the video in the “Latent Goal” box. The
latent goal is a vector in $[-1,1]^{32}$ and is visualized as a bar plot.

The “Option State Value” box shows the value estimate
$\widehat{V}^{\pi_{\mathrm{LLC}}}$ for the given goal, over time. The
rightmost point in the plot corresponds to the value estimate for the most
recent timestep in the video, and the flat red line is the threshold for
unattainable goals (note that points below the line are a plotting artifact as
they would refer to timesteps before the option started). This plot is only
valid when an option is being followed (that is, when a Latent Goal is shown).

The “Distance-to-Goal Predictor” box shows the probability estimates of the
predictor for how many steps into the future the goal is attained. This plot
is only valid when an option is being followed (that is, when a Latent Goal is
shown).

We can see in the videos that both the Option State Value and the Distance-to-
Goal Predictor probabilities behave sensibly. The option state value rises as
the agent approaches the goal, and the mass over distance-to-goal shifts
towards the zero bin. Once the probability mass is maximum at zero, the option
terminates, as specified by the termination criterion.

The majority of HLC actions in the videos are primitive actions, and when
goals are issued they typically last for a few steps (three to seven). The
goal-following behavior we observed in the videos seem to cover navigating
towards objects, picking them up and carrying them, whereas primitive actions
seem to be reorienting the agent’s view.

Qualitatively, we see from the video different signs that the hierarchical
behavior being learned is sound and in line with what we would expect to be
useful for solving the tasks, but that specific parts of the agent’s behavior
are still executed in terms of primitive actions rather than options.

###  C.5 Low Level Controller Heatmaps

In Fig. 13, we present a qualitative analyse of the goal-following performance
of the LLC, in isolation from the HLC. The setup is the following: 100
instances of the LLC are initialized in different instances of the same
evaluation Navigation Cubes task. Each of them is shown a goal image of a
golden apple being right in front of the agent, which would have been the
observation had the agent reached the golden apple. The LLCs then take
actions, trying to reach the given goal. If the LLC option execution is
terminated, the same goal image is presented again, for a total of 20 times
per episode, for each LLC instance. We record the trajectory of every LLC
instance and overlay them all on the same top-down map of the environment,
where each LLC instance can be seen as a single red dot. Figure 13 shows
snapshots of the overlaid trajectories at different timesteps.

At timestep 0, all instances start from the same position, and initially they
move in a similar direction. After about 40 timesteps, their trajectories
start to differentiate significantly, with some of them making it across the
cube pit, while others end up spending a long time around the corner of the
room. The main takeaway from this analysis is that indeed the different LLC
instances behave similar in the beginning, but their trajectories seem to
diverge at longer timescales, as they interact with the environment.

Figure 13: Top-down positions of the LLC at different timesteps in the
Navigation Cubes task. Note that there is no HLC here, as we are manually
giving the same image goal of a golden apple to the LLC, for 20 repetitions.
Note how all LLC instances are following a similar path towards the golden
apple, with some variation. Each red dot corresponds to the position of a
single LLC instance out of 100 identical LLCs initialized from the same state
and given the same goal. Note that when each LLC terminates for all 20
repeated goals, or the episode terminates by reaching the apple, its
corresponding red dot disappears.