# Contextualize Me – The Case for Context in Reinforcement Learning

Carolin Benjamins  Theresa Eimer  Frederik Schubert  Aditya Mohan  André
Biedenkapp  Bodo Rosenhahn  Frank Hutter  Marius Lindauer

###### Abstract

While Reinforcement Learning (RL) has made great strides towards solving
increasingly complicated problems, many algorithms are still brittle to even
slight changes in environments. Contextual Reinforcement Learning (cRL)
provides a theoretical framework to model such changes in a principled manner,
thereby enabling flexible, precise and interpretable task specification and
generation. Thus, cRL is an important formalization for studying
generalization in RL. In this work, we reason about solving cRL in theory and
practice. We show that theoretically optimal behavior in contextual Markov
Decision Processes requires explicit context information. In addition, we
empirically explore context-based task generation, utilizing context
information in training and propose cGate, our state-modulating policy
architecture. To this end, we introduce the first benchmark library designed
for generalization based on cRL extensions of popular benchmarks, CARL. In
short: Context matters!

RL, Generalization, Contextual Reinforcement Learning, cMDP

##  1 Introduction

Reinforcement Learning (RL) has driven progress in areas like game playing
[54, 4], robot manipulation [31], traffic control [2], chemistry [68] and
logistics [32]. At the same time, RL has shown little to no success in real-
world deployment in areas such as healthcare or autonomous driving. This is
largely explained by the fact that modern RL agents are often not primarily
designed for generalization, making them brittle when faced with even slight
variations in their environment [62, 37, 34].

To address this limitation, recent research has started to focus on
generalization approaches for RL. Ideally, these general agents should be
capable of zero-shot transfer to previously unseen environments and to changes
in the problem setting while interacting with an environment [64, 19, 61, 1,
56, 28]. Steps in this direction have been taken through new problem settings
where agents can test their transfer performance, e.g. the Arcade Learning
Environment’s flavors [35] or benchmarks utilizing Procedural Content
Generation (PCG) to increase task variation, e.g. ProcGen [9] or Alchemy [58].

While these new problem settings have expanded the possibilities for
benchmarking generalization in RL, often the degree of task variation is
unknown or cannot be controlled precisely. We believe that generalization in
RL is held back by these factors stemming in part from a lack of problem
formalization [28]. With contextual Reinforcement Learning (cRL) [23] as the
formal basis we can precisely define train and test settings in terms of
contexts. This allows us to reason about the influence of context on
generalization performance and gain theoretical insights into the limitations
of conventional RL policies on contextual problems.

Figure 1:  Example of a contextual extension of Brax’ Fetch [17] as part of
CARL. CARL allows to configure and modify existing environments through the
use of context. This context can be made visible to agents through context
features to inform them directly about the current instantiation of the
context.

In order to empirically study cRL we introduce an implementation of cRL in our
benchmark library CARL (Context-Adaptive Reinforcement Learning). CARL extends
existing environments with context features, that, wherever possible, are
based on physical properties, such as gravity, friction, or the mass of an
object (see Figure 1). With CARL, we can meticulously define on which contexts
our RL agents train and test on, allowing us to study in-and out-of-
distribution generalization.

With CARL, we empirically show how context variation significantly increases
the difficulty to learn for RL agents, even in simple environments. We also
verify the theoretical insight that access to context information is
beneficial for generalization tasks. Furthermore, we explore the
generalization behavior for different degrees of overlap in train and test
setting. As a result of these findings, we propose and evaluate a novel state-
modulating context architecture cGate which provides an easy to use approach
to include context information in the training procedure.

In short, our contributions are (i) the benchmark library CARL enabling fine-
grained context control in benchmarking contextual RL, (ii) providing a
theoretical and empirical characterization of contextual RL, and, stemming
from our key insight that adding context information aids learning, (iii) our
state modulating context architecture cGate.

##  2 Contextual Markov Decision Processes

In order to derive meaningful insights about generalization we need to
pinpoint the definition of a task collection an agent should train on and
generalize to [28]. This could be landing a spacecraft on planets with
different gravitational forces or different levels of a game in ProcGen [9].
Task collections can be defined via random seeds as in PCG or parameter
settings of a task.

Contextual Markov Decision Processes (cMDP) allow us to formalize such kinds
of task collections and extend the standard definition of an MDP in RL [23].
An MDP $M=(\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\rho)$ consists of
a state space $\mathcal{S}$, an action space $\mathcal{A}$, transition
dynamics $\mathcal{T}$, a reward function $\mathcal{R}$ and a distribution
over the initial states $\rho$. Through the addition of context, we can
define, characterize and parameterize the environment’s rules of behavior and
therefore induce tasks as variations on the problem. In the resulting cMDP,
the action space $\mathcal{A}$ and state space $\mathcal{S}$ stay the same;
only the transition dynamics $\mathcal{T}_{c}$, the reward $\mathcal{R}_{c}$
and the initial state distribution $\rho$ can change depending on the context
$c\in\mathcal{C}$. Through the context-depending initial state distribution
$\rho$, the agent may furthermore be exposed to different parts of the state
space for different contexts. The context space $\mathcal{C}$ can either be a
discrete set of contexts or defined via a context distribution
$p_{\mathcal{C}}$. A cMDP $\mathcal{M}$ therefore defines a space of context-
MDPs $\mathcal{M}=\\{M_{c}\\}_{c\sim p_{\mathcal{C}}}$.

Objectives Having defined the cMDP and what context is, we can now formulate
potential objectives being addressed by contextual reinforcement learning
(cRL). Contextual RL tries to learn a policy that performs as close to
optimally as possible on all the MDPs in a given cMDP. In contrast to standard
RL, the objective in contextual RL is not necessarily to just solve the cMDP,
but it can also be to transfer the performance to a different cMDP altogether.
For this purpose we define a target context distribution $p_{\mathcal{C}}$
that induces the target cMDP $\mathcal{M}_{target}$. Depending on the target
distribution we can now measure performance on the training distribution,
interpolation performance, distribution shift, out-of-distribution
generalization and more, e.g., solving a single expensive hard task by only
training on smaller ones.

Obtaining Context Features Not every task has an easily defined or measured
context that describes the task in detail. Therefore, it is important to
examine how they can be obtained in other ways. Often we can only extract very
simple context features for a task. For example, even though PCG based
environments do not allow control over the training and test distributions,
they can still be considered contextual environments since they are usually
seeded. This would give us the random seed as context information [28].
However, the seed provides no valuable information about the instance it
induces. Obtaining more useful context features should therefore be a focus of
cRL going forward. A possible direction to explore is feature extraction in
the context space analogously to ideas like Exploratory Landscape Analysis for
black box optimization [38, 67]. Learnt representations provide an opportunity
[26, 20, 63, 8] to do this in a data-driven manner. As these are learnt as the
representations of the tasks the agent needs to solve, subspaces of the
context space requiring different policies should naturally be represented
differently. Thus, even without readily available context, representation
learning could enable the reliable availability of quality information that
allows for generalization both in- and out-of-distribution.

##  3 Related Work

Even though a lot of work in RL makes no explicit assumptions about task
variations and thus generalization, there are extensions of the basic MDP
models beyond cMDPs that focus on generalization. One of these are _Hidden-
Parameter MDPs_ [11], which allow for changes in dynamics just as in cRL, but
keep the reward function fixed. _Block MDPs_ [12] are concerned with a
different form of generalization than cMDPs; instead of zero-shot policy
transfer, they are aimed at learning representations from large, unstructured
observation spaces. Settings being described as a collection of interacting
parts, e.g. Multi-Agent problems, can be formalized as such through _Factored
MDPs_ [6, 22]. The generalization target here is again not necessarily zero-
shot policy transfer, but generalization with respect to one or more of the
factored components, e.g. the behaviour of a competing agent. Both Block MDPs
and Factored MDPs are compatible with cMDPs, i.e., we can construct a Block
cMDP or a Factored cMDP in order to focus on multiple dimensions of
generalization [55]. An alternative approach is the _epistemic POMDP_ [21] as
a special case of a cMDP. Here, the context is assumed to be unobservable.
Their method then models the uncertainty about which instance of the cMDP the
agent is deployed on during test time.

Beyond formal structures for generalization, transfer performance of RL agents
has been at the center of several subcommunities within RL. Robustness, for
example, can be seen as a subcategory of generalization where variations to
the context are usually kept small and the goal is to avoid failures due to
exceptions on a single task [40, 44, 66]. Policy transfer is also concerned
with generalization in a sense, though here the goal has often been fine-
tuning a pre-trained policy, i.e., few-shot generalization instead of zero-
shot generalization [13, 15, 42]. The goal in Multi-Task Learning is to learn
a fixed set of tasks efficiently [62]. Meta-Learning in RL usually aims at
zero-shot policy generalization similar to cRL. This field is very broad with
different approaches like learning to learn algorithms or their components
[52, 13, 59], generating task curricula [36, 41] or meta-learning
hyperparameters [49, 65]. Most of these methods were not conceived with cRL in
mind, but many Meta-RL methods can or do make use of context information to
guide their optimization [29, 14].

Generalization across the training distribution has become a common goal in
standard RL environments, often in the form of generalization across more or
less randomly generated tasks [27, 9, 50]. The larger the degree of randomness
in the generation procedure, however, the less information and control are
available for Meta-Learning methods [28]. In contrast to other works on
generalization in RL, we therefore focus on sampling context without PCG but
from explicitly defined distributions.

Figure 2: A sample cMDP with three contexts. The original one (left), one
changing the transition function (middle) and another the reward function
(right).

##  4 The Implications of Context

In this section, we will provide both an intuition and a theoretical overview
of the implications of treating generalization problems that can be modelled
as cMDPs like standard non-contextual MDPs. This should serve as a
demonstration of why using the cMDP framework for generalization problems is
beneficial and should be explored further.

###  4.1 Theoretical Properties

A standard RL policy is defined as a mapping from a state observation
$s\in\mathcal{S}$ to an action $a\in\mathcal{A}$. The state space
$\mathcal{S}$ is shared among all MDPs within a cMDP, meaning that states can
occur in multiple MDPs even though the transition and reward functions might
differ. An example is given in Figure 2. For each context the states are the
same, but the transitions and the associated rewards vary. Without context, a
standard RL policy would not act optimally on every context of this cMDP as
the optimal policy given $s_{0}$ differs between them; e.g. we should always
play $a_{0}$ in $s_{0}$ in $c_{0}$, but should switch to playing $a_{1}$ in
$s_{0}$ in $c_{1}$ and $c_{2}$.

In order to formalize this intuitive explanation of why context is helpful in
generalization, let us first define what optimal performance means in cMDPs.

###### Definition 1

A policy $\pi^{*}$ is optimal for a given cMDP $\mathcal{M}$ with discount
factor $\gamma$ and target context distribution $p_{\mathcal{C}}$ iff
$\pi^{*}$ is optimal on every MDP in $\mathcal{M}$:

$\forall
c\sim\mathcal{C}:\pi^{*}\in\operatorname*{arg\,max}_{\pi\in\Pi}\mathbb{E}_{\pi}\sum_{t}\gamma^{t}R_{c}(s_{t},\pi(s_{t}))$
(1)

This definition requires that there exists a policy $\pi^{*}$ that does not
depend on the context $c$ and yet is optimal on every context. We will now
show that optimal policies like this do not exist in general, but that to
obtain optimality, the policy needs to have access to the context. To do this,
we assume the common definition of a policy in RL as a mapping from state to
action space: $\pi:\mathcal{S}\rightarrow\mathcal{A}$.

###### Proposition 1

For a given cMDP $\mathcal{M}=\\{M_{c_{1}},M_{c_{2}}\\}$ defined over two
possible contexts $c_{1}$ and $c_{2}$, there is either a common optimal policy
$\pi^{*}$ for both contexts or there is at least one state $s^{\prime}$ at
which the optimal policy differs between the contexts.

Proof Sketch Let us look at a given state $s\in\mathcal{S}$ that is reachable
in $M_{c_{1}}$. Further, let us assume $\pi^{*}_{c_{1}}$ is the optimal policy
for $M_{c_{1}}$ that is defined only on states reachable in $M_{c_{1}}$. We
have to consider the three possible cases:  (i) $s$is reachable in $M_{c_{2}}$
and $\pi^{*}_{c_{1}}(s)$ is not optimal on $M_{c_{2}}$;  (ii) $s$is reachable
in $M_{c_{2}}$ and $\pi^{*}_{c_{1}}(s)$ is optimal on $M_{c_{2}}$;  (iii)
$s$is not reachable in $M_{c_{2}}$.

If (i) is true for at least one state $s\in\mathcal{S}$, the optimal policy
obviously is different between the contexts in this state and therefore we
have found a conflict state $s^{\prime}$. The other two cases do not produce
such a conflict. We can, however, construct a policy $\pi^{*}$ that is optimal
on $M_{c_{1}}$ and $M_{c_{2}}$ from $\pi^{*}_{c_{1}}$ if for all
$s\in\mathcal{S}$ either (ii) or (iii) is true. For any state $s$ where (ii)
holds, we simply set $\pi^{*}(s)=\pi^{*}_{c_{1}}(s)$. For states $s$ that are
not reachable in $M_{c_{1}}$ as stated in (iii), $\pi^{*}_{c_{1}}$ is not
defined. We can therefore extend $\pi^{*}$ by these states without changing
its optimality on $M_{c_{1}}$. Let $a^{*}$ be the optimal action in such $s$
on $M_{c_{2}}$. Then, we define $\pi^{*}(s)=a^{*}$. By construction, $\pi^{*}$
is then optimal on all states $s\in\mathcal{S}$ and it exists iff there is no
state reachable in both contexts where the optimal action for $M_{c_{1}}$
differs from the one for $M_{c_{2}}$.$\hfill\blacksquare$

###### Theorem 1

An optimal policy $\pi^{*}$ for any given cMDP $\mathcal{M}$ is only
guaranteed to exist if it is conditioned on the context:
$\pi:\mathcal{S}\times\mathcal{C}\rightarrow\mathcal{A}$.

Proof Sketch Let us assume we know an optimal policy $\pi^{*}_{c}$ for any
context $c\in\mathcal{C}$. As $c$ induces an MDP and with it a corresponding
optimal policy, we know that $\pi^{*}_{c}$ exists. Furthermore, let the sets
of optimal policies between at least two MDPs $M_{c_{1}}$ and $M_{c_{2}}$ be
disjoint. This means no policy exists that is optimal on both $c_{1}$ and
$c_{2}$.

Now let us examine the optimal policy $\pi^{*}$ and assume that it exists for
this cMDP $\mathcal{M}$. By definition, $\pi^{*}$ is optimal on $c_{1}$ and
$c_{2}$. If it is _only_ conditioned on the state, $\pi^{*}(s)$ results in the
same action independent of the context. Because the sets of optimal policies
for $c_{1}$ and $c_{2}$ are disjoint, there must be at least one state
$s^{\prime}$, where
$\pi^{*}_{c_{1}}(s^{\prime})\neq\pi^{*}_{c_{2}}(s^{\prime})$ according to
Proposition 1. As both are optimal on their respective contexts but not on the
other and do not result in the same action for $s$, $\pi^{*}$ cannot actually
be optimal for both $c_{1}$ and $c_{2}$. Thus, the optimal policy $\pi^{*}$
does not exist.

On the other hand, if we can condition the policy on the context, such that
$\pi:\mathcal{S}\times\mathcal{C}\rightarrow\mathcal{A}$, we can circumvent
this problem (for a discussion on how this interacts with partial
observability, see Appendix C). $\pi^{*}(s,c)=\pi^{*}_{c}$ is optimal for each
context.$\hfill\blacksquare$

The question this raises is how relevant the assumption of disjoint sets of
optimal policies for different contexts is in practice. We can expect that the
optimal policies for different contexts will not always be the same; when we
are interested in an agent’s generalization capabilities, we can even expect
substantial differences. What is less clear is how the state observations
relate to different contexts. If we can guarantee that the agent sees
different states for each context, i.e. the context is at least in part given
or encoded in the state, then a single policy can solve these different
contexts optimally by never encountering the conflict above. This is how
generalization is commonly handled in RL: by assuming that the state features
will encode enough information to distinguish between situations that would
require different actions. Since this is not a reliable way of avoiding
conflicts between context-optimal policies on the same states, we believe that
optimal policies not conditioned on context are hard to obtain for many
contextual RL problems.

Another question raised by Theorem 1 is how heavily the return is impacted by
_not_ conditioning the policy on the context. It could be a negligible effect
or it could be catastrophic, similarly the cMDP in Figure 2 where a standard
policy will only ever be able to optimally solve $\frac{2}{3}$ of the cMDP,
but with potentially much worse rewards. We can easily construct MDPs where
any policy not conditioned on context does arbitrarily poorly. Formally, the
return reduction can be defined as the gap between the optimal return
$G_{c,\pi^{*}}$ and the best achievable return given the current policy:

$\mathcal{OG}:=\mathbb{E}_{p_{\mathcal{C}}}[G_{c,\pi^{*}}]-\max_{\pi\in\Pi}\mathbb{E}_{p_{\mathcal{C}}}[G_{c,\pi}]$

We call $\mathcal{OG}$ the _Optimality Gap_. As shown above, it is zero if the
policy is conditioned on the context, but may grow large for a standard
policy, depending on the structure of the cMDP at hand. Intuitively, the gap
should grow the broader $p_{\mathcal{C}}$ becomes and the more impact slight
changes in $c$ have on the transitions and rewards, although it is demanding
to formally assess $\mathcal{OG}$ across all possible cMDPs.

###  4.2 Practical Handling and cGate

The existence of such an Optimality Gap in combination with the fact that it
is hard to learn an optimal standard policy for many cMDPs means that
disregarding context information on a problem including variations in reward
and transition functions induces an obstacle to learning and also negatively
affects an upper bound for generalization performance. To show that this
effect holds true in practice, we will evaluate it empirically in the
following sections. In addition, we propose methods to communicate context to
the agent:  (i) concatenating the context (either the complete context or only
changing context features) to the state as suggested by Hallak et al., [23]
and  (ii) our proposed context-specific policy architecture, cGate.

In order to use context to switch between different policy behaviors, we
propose a novel architecture for context-conditioned policies, cGate. It
combines the learned embedding of the representation learning approach with
separate pre-processing pathways for state and context features (see Figure
3). The Hadamard product of separately learned state and context features is
mapped to the policy parameters using a two-layer MLP.

Figure 3: cGate architecture: the state representation $\phi(s)$ is modulated
by a learned context embedding $g(c)$ and mapped to a policy by $f$.

This results in a gating mechanism that modulates the chosen action depending
on the embedded context. The mechanism is similar to the architecture of
Implicit Quantile Networks (IQNs) [10], where the quantile fraction embedding
selects the possible outcome given an embedding of the state. The context
branch is learned during RL training, reducing the computational overhead.

##  5 The CARL Benchmark Library

Figure 4: The CARL benchmarks

(a) Single Context Feature Variation

(b) Compounding Variation

(c) Visible vs. hidden context for $\sigma_{\text{rel}}=0.5$

Figure 5: Training performance of a DDPG agent on CARLPendulumEnv with (5(a))
varying individual context features, (5(b)) varying multiple context features,
and (5(c)) the effect of visible context. $\sigma$ is the standard deviation
for sampling the context.

To analyze how the context and its augmentation influences the agent’s
generalization capabilities, learning and behavior, we propose CARL: a library
for _Context Adaptive Reinforcement Learning_ benchmarks following our cRL
formalism. In the first release of CARL benchmarks, we include and
contextually extend classic control and box2d environments from OpenAI Gym
[7], Google Brax’ walkers [17], a RNA folding environment [49] as well as
Super Mario levels [3, 51], see Figure 4. Although each environment has
different goals and mechanics, the behavior of the dynamics and the rewards is
influenced by physical properties. A more detailed description of the
environments and the contextualization is given in Appendix A.1 as well as a
characterization of the benchmarks in Appendix A.2.

##  6 Experiments

Having discussed the theoretical framework of cRL, the implications of context
and how context can be considered in the training process, we now study
several research questions regarding the empirical effects of context. First,
how do different degrees of context variation influence learning? Furthermore,
how does the performance on training and test context distribution relate to
one another? Finally, we empirically explore several ways of conditioning
policies on context information.

To explore our research questions we use our benchmark library CARL. Details
about the hyperparameter settings and used hardware for all experiments are
listed in Appendix B. In each experiment, we train and evaluate on $5$
different random seeds and a set of $100$ sampled contexts.

#### Q1: How Do Different Degrees of Context Variation Influence Learning?

In order to demonstrate the drastic impact on the learning behavior when
varying the context we deliberately choose an environment known to be simple
and solved by many RL algorithms in its vanilla form, _Pendulum_ from OpenAI
gym [7]. Pendulum’s dynamic equations (see Appendix A.3) mirror the simplicity
of the environment. We also use a DDPG [33] agent which is known to work well
on Pendulum.

CARL extends the original Pendulum to a contextual environment where we can
achieve context variation by sampling or perturbing context features. In
Pendulum, we can vary the context features gravity (g), integration time step
(dt), pendulum mass (m) and length (l), as well as the maximal speed
(max_speed). One simple way to create different contexts to train on is to
sample the context feature from a normal distribution around its mean with a
standard deviation relative to the feature’s range. Decreasing or increasing
the magnitude of variation influences the difficulty of the contextual
environment. Because we leave the option to simply pass a set of contexts to
the environment, we can define any distribution of context features – allowing
for very fine-grained control over the experiment design (e.g., see Section
6). Here, we sample a set of $100$ contexts for each considered context
feature while keeping the other context features fixed to their default. The
current context in an episode is selected with a round-robin schedule. We
train with three relative standard deviations
$\sigma_{\text{rel}}\in\\{0.1,0.25,0.5\\}$ to show the impact of varying
degrees of similarities in the instance distribution. We evaluate the agent
every $50\,000$ steps on the full context set. Further, we do not provide the
agent with the context directly in order to keep the state space consistent
with the original environment.

Single Variations As shown in Figure 5(a), while small changes in the context
barely have an impact on the training performance of the agent, large
variations of a _single context feature_ can make the learning problem
challenging. Also, performance is much more sensitive to some context features
than to others. This gives evidence that even simple, cheap-to-run
environments challenge the agent, depending on the level of context variation
and required generalization.

Compounding Variations So far only one context feature was varied and not
every context feature had the same impact on difficulty. In this experiment we
vary more and more context features simultaneously. The order is based on the
perceived difficulty of the previous experiment, being no context feature,
mass m, length l, timestep dt, gravity g and maximum speed max_speed. As shown
in Figure 5(b), the performance correlates with the number of varied context
features.

Providing Context Information Lastly, we investigate whether these variations
in performance can be mitigated by providing context information with the
state observations. As shown in Figure 5(c), concatenating the changing
context features to the state clearly boosts performance compared to hidden
context features. However, we still observe that variations in the context
make learning more challenging.

We refer to the Appendix D for the effect of context variation on other
classic control, box2d and brax environments, all yielding similar insights.
We also illustrate the influence of context on a vision-based environment,
CARLMarioEnv.

#### Q2: How Well Do Agents Generalize to In- and Out-Of-Distribution
Contexts?

Previously, we saw that the degree of context variation has an impact on
learning and generalization capabilities. We now investigate generalization to
in- and out-of-distribution test contexts which we can precisely define with
CARL. In order to demonstrate these dimensions of generalization, we construct
two different test settings: First, we follow the evaluation protocols for cRL
proposed by Kirk et al., [28] and second, we create a more in-depth
illustrative “Landing in Space”-scenario based on the LunarLander environment
[7].

Evaluation Protocols While for PCG environments we can only have evaluation
protocols defined over entire distribution of training and test contexts, our
fine-grained setup in CARL allows for more elaborate evaluation protocols as
proposed by Kirk et al., [28]. We can define train and test distributions or
sets for each dimension of the context space individually, allowing us to
study different relationships between train and test settings. If the test
context is inside the train distribution we speak of interpolation, if it is
outside, of extrapolation. By choosing two context features and defining
uniform train distributions on both, we get a convex training set in the
context feature space, see the left column of Figure 6 (mode A: ). The context
feature distributions can also be linked together to allow only a small
variation (middle column, mode B: ) or one single value in one direction
(right column, mode C: ), creating a non-convex train set. Thus, combinatorial
interpolation tests the performance in the convex hull of this non-convex set.
In addition, we can test by extrapolating along one or all factors, increasing
difficulty.

To demonstrate this, we again choose a simple environment, CartPole [7]
embedded by CARL, and the DQN [39] agent known to perform well in it. We train
the agent for $500\,000$ timesteps and vary the gravity and the pole length in
the environment. As shown in Section 6, this relatively easy setting can
already be very challenging for the agent. We repeat this with five random
seeds. For the train and test context sets we sample from the corresponding
distributions of the evaluation protocol, see Figure 6. For each region, we
sample $100$ test contexts which do not intersect with the train contexts. The
test performances are averaged per region across seeds. We make several
observations:  (i) Providing the agent context information boosts
generalization performance overall as seen by comparing the test performances
of hidden and visible context.  (ii) Extrapolation to unseen contexts across
both factors always is the hardest.  (iii) The combination of strongly varying
along one factor and mildly varying across the other is the easiest to find a
well-performing policy for (interpolation test performance in Scenario B), in
contrast to varying both with the same magnitude (too wide) and fixing one
while varying the other (too narrow).

Figure 6: Top: Train and test distributions for different evaluation protocols
[28] (seed 1). The different regions of inter- or extrapolation are marked by
differently colored backgrounds, each dot being a sample context. Middle: Test
performance for hidden context. Bottom: Test performance for concatenated
context. Figure 7: “Landing in Space” Test Performance

Landing in Space In order to see whether these findings hold true on more
specialized train and test combinations, we apply this evaluation protocol to
a more elaborate setup. In “Landing in Space”, an agent is challenged to land
a spacecraft on seen as well as unseen planets in the CARL-embedded
LunarLander [7] environment. The different planets are modeled by only
adjusting their gravities by a well-defined normal distribution, centered on
Mars ($\mu=$3.7\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$$). The LunarLander
is trained to land on smaller planets, with a standard deviation
($\sigma=$1.45\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$$) chosen such that
Mars and Moon are considered as in-distribution111A planet is deemed in-
distribution if its gravity is within the $95\text{\,}\mathrm{\char
37\relax}$-interval of the training distribution. whilst Pluto, Earth, Neptune
and Jupiter are considered as unseen and out-of-distribution, see Figure 16(b)
in the appendix. We train a DQN [39] agent on this rather narrow normal
context distribution. For testing, we then place this agent in a new context
which might not have been observed during training. For training, we sample
$100$ planets with varying gravities from this distribution. We use $5$ random
seeds for training and testing and collect $100$ episodes on each planet for
both cases where the context is hidden and where the context feature gravity
is visible to the agent. Note that although for each test episode on a planet
the gravity is the same, the LunarLander environment generates different
initial starting conditions and landscapes to land on. For this reason, the
lander might still fail to safely land and crash in some cases. To capture
crashes and to distinguish them from successful but suboptimal landings, we
increase the game-over penalty from $-100$ to $-10\,000$ during testing.

The test performances for landing on Mars and Moon, the interpolation case,
are most similar to the Mars-centered training distribution, see Figure 7. We
observe more crashes on out-of-distribution (extrapolation) planets like
Earth, Neptune and Jupiter. The agent fails least often on Pluto since (a) it
has a gravity still close to the training distribution and (b) it has a low
gravity which makes landing easier. The lower the gravity and the longer the
timeframe to land the more time an agent has to adjust the spacecraft before
reaching the surface. We can further note that fewer runs on Pluto lead to as
high rewards as on in-distribution planets Moon and Mars. This is likely due
to agents wasting fuel by anticipating a harder landing, thus burning more
fuel to counteract. Interestingly, even for more difficult out-of-distribution
planets, such as Earth and Neptune, we can observe positive landing results
for both agents trained with and without access to the context. However, test
performance deteriorates due to more frequent crashing on higher-gravity
planets. In Appendix 16 we provide another experiment with the gravity train
distribution modeled as regular intervals. While we have seen some capability
of trained agents to transfer even to out-of-distribution environments, we do
not expect vanilla agents to generalize to highly different environments. In
addition, the varying test performances can not only be attributed to
different generalization capabilities but also be a result of the effect of
the context feature on the environment as seen with the gravity.

To sum up, the agents generalize better in-distribution (interpolation) than
out-of-distribution (extrapolation), independent of the type of training
distribution. Extrapolating across all context features is the hardest case.
Again, we see a clear improvement when we provide context information to the
agent which leads us to the following section.

#### Q3: How Can We Provide Context Information To Improve Generalization
Performance?

So far, we have shown that providing context to the policy provides a
theoretical and practical advantage compared to standard RL policies.
Therefore, we now try to answer the question how to communicate the context
information to the agent. Here, we consider four methods of providing context
information during training. The variations are: concatenate the raw context
to the state, concatenate only non-constant context information to the state
or process the context via our specialized state-modulating policy
architecture cGate.

The experimental setup is to train a TD3 agent on CARLPendulumEnv with the
previously used ranges of standard deviations of $0.1$, $0.2$ and $0.5$ (for
further details, see Appendix B). We vary all five context features on their
own, as well as simultaneously, to test the overall capabilities of the
agents. More detailed results, including on another environment, can be found
in Appendix D.4.

As shown in Figure 8, there are significant differences in performance
depending on how the policy is conditioned on the context. The two variations
of concatenating raw context to the state perform fairly similar throughout
all context distributions. Concatenating all context features performs
slightly better for narrower context distributions than concatenating only
changing ones, though this gap closes and even reverses with more context
diversity.

cGate consistently performs better than the other methods with a large margin
of almost $200$ reward points for $\sigma_{rel}=0.1$ and $0.5$. On
$\sigma_{rel}=0.25$, single context feature concatenation is able to match the
performance of cGate, but full concatenation still performs significantly
worse. We believe that the good performance and robustness is due to cGate
dynamically learning a representation of the context space that works well for
the policy, while also being able to differentiate between context features
and observations.

Figure 8: Context-conditioning on CARLPendulum. Results are aggregated for no
variation, varying single context features and varying all context features.

##  7 Open Challenges in Contextual RL

Although these experiments only give a first impression on how cRL and CARL
can specifically be used to gain novel insights into RL agents, we see many
more possibilities for future research involving context. We discuss six open
challenges in generalization and how cRL could be used to tackle these.  (I)
As interpretable contexts provide ground truths for all considered context
features, it is suitable to study novel agents that separate representation
learning from policy learning.  (II) cRL will be useful in studying RL agents
for uncertain dynamics, by easily perturbing context features.  (III) So far,
we have only looked at static context distributions, but we can extend cRL to
include more complex dynamic changes to context features, e.g., in a life-long
learning setting.  (IV) The ground truth on contexts can also be used to study
explainability and interpretability methods of deep RL.  (V) With the
complexity of modern RL methods, they have become very sensitive to their
hyperparameters; we see this effect amplified for contextual agents (see
Appendix E). cRL’s flexibility and focus on generalization enables research
into AutoRL [43] methods that optimize agents for generality.  (VI) Finally,
it is an open question for safe RL whether context information could
contribute to decide whether policies are applicable to unseen instances of an
environment.  A benchmark with interpretable and deterministic contexts like
CARL can provide the basis to study these challenges and arrive at general and
robust agents. Please find an extended discussion of these challenges in cRL
in Appendix F.

##  8 Conclusion

Towards our goal of creating more general and robust agents, we need to factor
in possible changes in the environment. We model these changes with the
theoretical framework of contextual Reinforcement Learning (cRL) and reason
about what demands cRL introduces to the agents and the learning process,
specifically regarding the suboptimal nature of conventional RL policies in
cRL. With CARL, we provide a benchmark library designed for generalization to
study cRL and which contextualizes popular benchmarks. It allows us to
empirically demonstrate that contextual changes disturb the learning even in
simple settings, and that the final performance and the difficulty correlates
with the number of factors and the magnitude varying. We also verify that
context-conditioned policies outperform standard RL policies on average, as
suggested by our theoretical discussion. Our specialized, state-modulating
contextual policy architecture cGate provides a direct way of training such
context-conditioned policies in cRL. We expect this to be a first step towards
better solution mechanisms for contextual RL problems and therefore one step
closer to general and robust agents.

## References

  * Abdolshah et al., [2021] Abdolshah, M., Le, H., George, T. K., Gupta, S., Rana, S., and Venkatesh, S. (2021).  A new representation of successor features for transfer across dissimilar environments.  In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1–9. PMLR.
  * Arel et al., [2010] Arel, I., Liu, C., Urbanik, T., and Kohls, A. (2010).  Reinforcement learning-based multi-agent system for network traffic signal control.  IET Intelligent Transport Systems, 4(2):128–135.
  * Awiszus et al., [2020] Awiszus, M., Schubert, F., and Rosenhahn, B. (2020).  TOAD-GAN: Coherent style level generation from a single example.  In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 16.
  * Badia et al., [2020] Badia, A., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, Z., and Blundell, C. (2020).  Agent57: Outperforming the atari human benchmark.  In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 507–517. PMLR.
  * Biedenkapp et al., [2018] Biedenkapp, A., Marben, J., Lindauer, M., and Hutter, F. (2018).  CAVE: Configuration assessment, visualization and evaluation.  In Proceedings of the International Conference on Learning and Intelligent Optimization (LION), Lecture Notes in Computer Science. Springer.
  * Boutilier et al., [1999] Boutilier, C., Dean, T. L., and Hanks, S. (1999).  Decision-theoretic planning: Structural assumptions and computational leverage.  J. Artif. Intell. Res., 11:1–94.
  * Brockman et al., [2016] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016).  Openai gym.  CoRR, abs/1606.01540.
  * Castro et al., [2021] Castro, P., Kastner, T., Panangaden, P., and Rowland, M. (2021).  Mico: Learning improved representations via sampling-based state similarity for markov decision processes.  CoRR, abs/2106.08229.
  * Cobbe et al., [2020] Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. (2020).  Leveraging procedural generation to benchmark reinforcement learning.  In Proceedings of the 37th International Conference on Machine Learning, ICML, volume 119 of Proceedings of Machine Learning Research, pages 2048–2056. PMLR.
  * Dabney et al., [2018] Dabney, W., Ostrovski, G., Silver, D., and Munos, R. (2018).  Implicit quantile networks for distributional reinforcement learning.  In Proceedings of the 35th International Conference on Machine Learning, ICML, volume 80 of Proceedings of Machine Learning Research, pages 1104–1113. PMLR.
  * Doshi-Velez and Konidaris, [2016] Doshi-Velez, F. and Konidaris, G. D. (2016).  Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations.  In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI, pages 1432–1440. IJCAI/AAAI Press.
  * Du et al., [2019] Du, S., Krishnamurthy, A., Jiang, N., Agarwal, A., Dudík, M., and Langford, J. (2019).  Provably efficient RL with rich observations via latent state decoding.  In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1665–1674. PMLR.
  * Duan et al., [2016] Duan, Y., Schulman, J., Chen, X., Bartlett, P., Sutskever, I., and Abbeel, P. (2016).  Rl$^2$: Fast reinforcement learning via slow reinforcement learning.  CoRR, abs/1611.02779.
  * Eimer et al., [2021] Eimer, T., Biedenkapp, A., Hutter, F., and Lindauer, M. (2021).  Self-paced context evaluation for contextual reinforcement learning.  In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning (ICML’21), volume 139 of Proceedings of Machine Learning Research, pages 2948–2958. PMLR.
  * Finn et al., [2017] Finn, C., Abbeel, P., and Levine, S. (2017).  Model-agnostic meta-learning for fast adaptation of deep networks.  In Precup, D. and Teh, Y., editors, Proceedings of the 34th International Conference on Machine Learning (ICML’17), volume 70, pages 1126–1135. Proceedings of Machine Learning Research.
  * Franke et al., [2021] Franke, J., Köhler, G., Biedenkapp, A., and Hutter, F. (2021).  Sample-efficient automated deep reinforcement learning.  In 9th International Conference on Learning Representations, ICLR. OpenReview.net.
  * Freeman et al., [2021] Freeman, C., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O. (2021).  Brax - A differentiable physics engine for large scale rigid body simulation.  CoRR, abs/2106.13281.
  * [18] Fu, H., Tang, H., Hao, J., Chen, C., Feng, X., Li, D., and Liu, W. (2021a).  Towards effective context for meta-reinforcement learning: an approach based on contrastive learning.  In Proceedings of the Conference on Artificial Intelligence (AAAI’21), pages 7457–7465. AAAI Press.
  * [19] Fu, X., Yang, G., Agrawal, P., and Jaakkola, T. (2021b).  Learning task informed abstractions.  In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 3480–3491. PMLR.
  * Gelada et al., [2019] Gelada, C., Kumar, S., Buckman, J., Nachum, O., and Bellemare, M. (2019).  Deepmdp: Learning continuous latent space models for representation learning.  In Proceedings of the 36th International Conference on Machine Learning, ICML, volume 97 of Proceedings of Machine Learning Research, pages 2170–2179. PMLR.
  * Ghosh et al., [2021] Ghosh, D., Rahme, J., Kumar, A., Zhang, A., Adams, R. P., and Levine, S. (2021).  Why generalization in RL is difficult: Epistemic pomdps and implicit partial observability.  CoRR, abs/2107.06277.
  * Guestrin et al., [2001] Guestrin, C., Koller, D., and Parr, R. (2001).  Multiagent planning with factored mdps.  In Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS, pages 1523–1530. MIT Press.
  * Hallak et al., [2015] Hallak, A., Castro, D. D., and Mannor, S. (2015).  Contextual markov decision processes.  arXiv:1502.02259 [stat.ML].
  * J. Parker-Holder et al., [2020] J. Parker-Holder, V., Nguyen, S. J., and Roberts (2020).  Provably efficient online hyperparameter optimization with population-based bandits.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Proceedings of the 33rd International Conference on Advances in Neural Information Processing Systems (NeurIPS’20), volume 33, pages 17200–17211.
  * [25] Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., Fernando, C., and Kavukcuoglu, K. (2017a).  Population based training of neural networks.  arXiv:1711.09846 [cs.LG].
  * [26] Jaderberg, M., Mnih, V., Czarnecki, W., Schaul, T., Leibo, J., Silver, D., and Kavukcuoglu, K. (2017b).  Reinforcement learning with unsupervised auxiliary tasks.  In 5th International Conference on Learning Representations, ICLR. OpenReview.net.
  * Juliani et al., [2019] Juliani, A., Khalifa, A., Berges, V., Harper, J., Teng, E., Henry, H., Crespi, A., Togelius, J., and Lange, D. (2019).  Obstacle tower: A generalization challenge in vision, control, and planning.  In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI, pages 2684–2691. ijcai.org.
  * Kirk et al., [2021] Kirk, R., Zhang, A., Grefenstette, E., and Rocktäschel, T. (2021).  A survey of generalisation in deep reinforcement learning.  CoRR, abs/2111.09794.
  * Klink et al., [2020] Klink, P., D’Eramo, C., Peters, J., and Pajarinen, J. (2020).  Self-paced deep reinforcement learning.  In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS.
  * Kostas et al., [2021] Kostas, J., Chandak, Y., Jordan, S., Theocharous, G., and Thomas, P. (2021).  High confidence generalization for reinforcement learning.  In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5764–5773. PMLR.
  * Lee et al., [2020] Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., and Hutter, M. (2020).  Learning quadrupedal locomotion over challenging terrain.  Science in Robotics, 5.
  * Li et al., [2019] Li, X., Zhang, J., Bian, J., Tong, Y., and Liu, T. (2019).  A cooperative multi-agent reinforcement learning framework for resource balancing in complex logistics network.  In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS, pages 980–988. International Foundation for Autonomous Agents and Multiagent Systems.
  * Lillicrap et al., [2016] Lillicrap, T., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016).  Continuous control with deep reinforcement learning.  In Bengio, Y. and LeCun, Y., editors, 4th International Conference on Learning Representations, ICLR.
  * Lu et al., [2020] Lu, M., Shahn, Z., Sow, D., Doshi-Velez, F., and Lehman, L. H. (2020).  Is deep reinforcement learning ready for practical applications in healthcare? A sensitivity analysis of duel-ddqn for hemodynamic management in sepsis patients.  In AMIA 2020, American Medical Informatics Association Annual Symposium, Virtual Event, USA, November 14-18, 2020. AMIA.
  * Machado et al., [2018] Machado, M., Bellemare, M., Talvitie, E., Veness, J., Hausknecht, M., and Bowling, M. (2018).  Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents.  J. Artif. Intell. Res., 61:523–562.
  * Matiisen et al., [2020] Matiisen, T., Oliver, A., Cohen, T., and Schulman, J. (2020).  Teacher-student curriculum learning.  IEEE Trans. Neural Networks Learn. Syst., 31(9):3732–3740.
  * Meng and Khushi, [2019] Meng, T. and Khushi, M. (2019).  Reinforcement learning in financial markets.  Data, 4(3):110.
  * Mersmann et al., [2011] Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., and Rudolph, G. (2011).  Exploratory landscape analysis.  In Krasnogor, N. and Lanzi, P. L., editors, 13th Annual Genetic and Evolutionary Computation Conference, GECCO, pages 829–836. ACM.
  * Mnih et al., [2015] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A., Veness, J., Bellemare, M., Graves, A., Riedmiller, M., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. (2015).  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533.
  * Morimoto and Doya, [2000] Morimoto, J. and Doya, K. (2000).  Robust reinforcement learning.  In Leen, T. K., Dietterich, T. G., and Tresp, V., editors, Advances in Neural Information Processing Systems 13, Papers from Neural Information Processing Systems (NIPS) 2000, Denver, CO, USA, pages 1061–1067. MIT Press.
  * Nguyen et al., [2021] Nguyen, S., Duminy, N., Manoury, A., Duhaut, D., and Buche, C. (2021).  Robots learn increasingly complex tasks with intrinsic motivation and automatic curriculum learning.  Künstliche Intell., 35(1):81–90.
  * Nichol et al., [2018] Nichol, A., Achiam, J., and Schulman, J. (2018).  On first-order meta-learning algorithms.  CoRR, abs/1803.02999.
  * Parker-Holder et al., [2022] Parker-Holder, J., Rajan, R., Song, X., Biedenkapp, A., Miao, Y., Eimer, T., Zhang, B., Nguyen, V., Calandra, R., Faust, A., Hutter, F., and Lindauer, M. (2022).  Automated reinforcement learning (autorl): A survey and open problems.  CoRR, abs/2201.03916.
  * Pinto et al., [2017] Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. (2017).  Robust adversarial reinforcement learning.  In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 2817–2826. PMLR.
  * Raffin, [2020] Raffin, A. (2020).  Rl baselines3 zoo.  https://github.com/DLR-RM/rl-baselines3-zoo.
  * Raffin et al., [2019] Raffin, A., Hill, A., Ernestus, M., Gleave, A., Kanervisto, A., and Dormann, N. (2019).  Stable baselines3.  https://github.com/DLR-RM/stable-baselines3.
  * Rakelly et al., [2019] Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, D. (2019).  Efficient off-policy meta-reinforcement learning via probabilistic context variables.  In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning (ICML’19), volume 97, pages 5331–5340. PMLR.
  * Rice, [1976] Rice, J. (1976).  The algorithm selection problem.  Advances in Computers, 15:65–118.
  * Runge et al., [2019] Runge, F., Stoll, D., Falkner, S., and Hutter, F. (2019).  Learning to Design RNA.  In Proceedings of the International Conference on Learning Representations (ICLR’19).  Published online: iclr.cc.
  * Samvelyan et al., [2021] Samvelyan, M., Kirk, R., Kurin, V., Parker-Holder, J., Jiang, M., Hambro, E., Petroni, F., Küttler, H., Grefenstette, E., and Rocktäschel, T. (2021).  Minihack the planet: A sandbox for open-ended reinforcement learning research.  CoRR, abs/2109.13202.
  * Schubert et al., [2021] Schubert, F., Awiszus, M., and Rosenhahn, B. (2021).  Toad-gan: a flexible framework for few-shot level generation in token-based games.  IEEE Transactions on Games, pages 1–1.
  * Schulman et al., [2016] Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2016).  High-dimensional continuous control using generalized advantage estimation.  In Bengio, Y. and LeCun, Y., editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
  * Schulman et al., [2017] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017).  Proximal policy optimization algorithms.  CoRR, abs/1707.06347.
  * Silver et al., [2016] Silver, D., Huang, A., Maddison, C., Guez, A., Sifre, L., Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. (2016).  Mastering the game of go with deep neural networks and tree search.  Nature, 529(7587):484–489.
  * [55] Sodhani, S., Denoyer, L., Kamienny, P., and Delalleau, O. (2021a).  Mtenv - environment interface for mulit-task reinforcement learning.  Github.
  * [56] Sodhani, S., Zhang, A., and Pineau, J. (2021b).  Multi-task reinforcement learning with context-based representations.  In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9767–9779. PMLR.
  * van Rijn and Hutter, [2018] van Rijn, J. and Hutter, F. (2018).  Hyperparameter importance across datasets.  In Guo, Y. and F.Farooq, editors, Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 2367–2376. ACM Press.
  * Wang et al., [2021] Wang, J., King, M., Porcel, N., Kurth-Nelson, Z., Zhu, T., Deck, C., Choy, P., Cassin, M., Reynolds, M., Song, H., Buttimore, G., Reichert, D., Rabinowitz, N., Matthey, L., Hassabis, D., Lerchner, A., and Botvinick, M. (2021).  Alchemy: A structured task distribution for meta-reinforcement learning.  CoRR, abs/2102.02926.
  * Wang et al., [2017] Wang, J., Kurth-Nelson, Z., Soyer, H., Leibo, J., Tirumala, D., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. (2017).  Learning to reinforcement learn.  In Gunzelmann, G., Howes, A., Tenbrink, T., and Davelaar, E., editors, Proceedings of the 39th Annual Meeting of the Cognitive Science Society. cognitivesciencesociety.org.
  * Xu et al., [2010] Xu, L., Hoos, H., and Leyton-Brown, K. (2010).  Hydra: Automatically configuring algorithms for portfolio-based selection.  In Fox, M. and Poole, D., editors, Proceedings of the Twenty-fourth National Conference on Artificial Intelligence (AAAI’10), pages 210–216. AAAI Press.
  * Yarats et al., [2021] Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. (2021).  Reinforcement learning with prototypical representations.  In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11920–11931. PMLR.
  * Yu et al., [2019] Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. (2019).  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In Conference on Robot Learning (CoRL).
  * [63] Zhang, A., McAllister, R. T., Calandra, R., Gal, Y., and Levine, S. (2021a).  Learning invariant representations for reinforcement learning without reconstruction.  In 9th International Conference on Learning Representations, ICLR. OpenReview.net.
  * [64] Zhang, A., Sodhani, S., Khetarpal, K., and Pineau, J. (2021b).  Learning robust state abstractions for hidden-parameter block mdps.  In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net.
  * [65] Zhang, B., Rajan, R., Pineda, L., Lambert, N., Biedenkapp, A., Chua, K., Hutter, F., and Calandra, R. (2021c).  On the importance of hyperparameter optimization for model-based reinforcement learning.  In The 24th International Conference on Artificial Intelligence and Statistics, AISTATS, volume 130 of Proceedings of Machine Learning Research, pages 4015–4023. PMLR.
  * [66] Zhang, H., Chen, H., Boning, D., and Hsieh, C. (2021d).  Robust reinforcement learning on state observations with learned optimal adversary.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  * Zhou et al., [2019] Zhou, W., Pinto, L., and Gupta, A. (2019).  Environment probing interaction policies.  In Proceedings of the International Conference on Learning Representations (ICLR’19).  Published online: iclr.cc.
  * Zhou et al., [2017] Zhou, Z., Li, X., and Zare, R. (2017).  Optimizing chemical reactions with deep reinforcement learning.  ACS central science, 3(12):1337–1344.

##  Appendix A The CARL Benchmarks

This section provides an overview over the benchmarks included in CARL
(Section A.1), a characterization of the properties of the benchmarks (Section
A.2) as well as Pendulum’s dynamic equations and contextualization (Section
A.3).

###  A.1 Benchmark Categories

(a)

(b)

(c)

(d)

Figure 9: CARL Environments; listed from top to bottom. (9(a)) OpenAI Gym’s
[7] Acrobot and Pendulum, CartPole, MountainCar. (9(b)) OpenAI Gym’s [7]
BipedalWalker, LunarLander, CarRacing. (9(c)) Brax [17] Ant and HalfCheetah,
Fetch and Humanoid, Grasp and UR5E. (9(d)) Super Mario [3, 51] and RNADesign
[49].

To encourage generalization in RL, we chose a wide variety of common problem
characteristics as well as well-known environments as the basis of CARL, see
Figure 9. The physical simulation environments (Brax, box2d and classic
control) defining a dynamic body in a static world have similar context
features like gravity, geometry of the moving body, position and velocity,
mass, friction and joint stiffness. For brevity, we only detail the context
features of CARLFetch and list all other environments’ context features in
Section G of the appendix.

CARLFetch embeds Brax’ Fetch [17] as a cMDP, see Figure 10. The goal of Fetch
is to move the agent to the target area. The context features joint stiffness,
gravity, friction, (joint) angular damping, actuator strength, torso mass as
well as target radius and distance define the context. The defaults of the
context features are copied from the original environment. Furthermore,
appropriate bounds must be set for the specific application. We set the bounds
such that the environment’s purpose is not violated, e.g., restricting the
gravity towards the ground greater than $0$ (otherwise the agent would fly up
and it would be impossible to act).

Besides physical simulation environments, CARL provides two more specific,
challenging environments. The first is the CARLMarioEnv environment built on
top of the TOAD-GAN level generator [3, 51]. It provides a procedurally
generated game playing environment that allows customization of the generation
process. This environment is therefore especially interesting for exploring
representation learning for the purpose of learning to better generalize.
Secondly, we move closer to real-world application by including the
CARLRNADesignEnvironment [49]. The challenge here is to design RNA sequences
given structural constraints. As two different datasets of structures and
their instances are used in this benchmark, it is ideally suited for testing
policy transfer between RNA structures.

Figure 10:  Example of a contextual extension of Brax’ Fetch [17] as part of
CARL. CARL allows to configure and modify existing environments through the
use of context. CARL makes the context defining the behavior of the
environment visible and configurable. This way we can train for generalization
over different contexts of the same environment. Here, we show all context
features for Brax’ Fetch [17] and sketch possible instantiations by setting
the context features to different values.

###  A.2 Properties of Benchmarks

While the categorization of the CARL benchmarks above provides an overview of
the kinds of environments included, we also discuss them in terms of relevant
environment attributes that describe the nature of their problem setting, see
Figure 11.

State Space Most of our benchmarks have vector based state spaces, allowing to
concatenate context information. Their sizes range from only two state
variables in the CARLMountainCar environments to 299 for the CARLHumanoid
environment. The notable exceptions here are CARLVehicleRacing and
CARLToadGAN, which exclusively use pixel-based observations.

Action Space We provide both discrete and continuous environments, with six
requiring discrete actions and the other ten continuous ones. The number of
actions can range from a single action to 19 different actions.

Quality of Reward We cover different types of reward signals with our
benchmarks, ranging from relatively sparse step penalty style rewards where
the agent only receives a reward of $-1$ each step to complex composite reward
functions in e.g. the Brax-based environments. The latter type is quite
informative, providing updates on factors like movement economy and progress
towards the goal whereas the former does not let the agents distinguish
between transitions without looking at the whole episode. Further examples for
sparse rewards are the CARLCartPoleEnv and CARLVehicleRacingEnv.

Context Spaces While the full details of all possible context configurations
can be seen in Appendix G, for brevity here we only discuss the differences
between context spaces and the configuration possibilities they provide.
Depending on the environment the context features have different influence on
the dynamics and the reward. Of all $131$ registered context features,
$98\text{\,}\mathrm{\char 37\relax}$ influence the dynamics. This means that
if a context feature is changed then the transition from states to their
successors is affected and likely changed as well. Only
$5\text{\,}\mathrm{\char 37\relax}$ of the context features shape the reward.
Most context features ($87\text{\,}\mathrm{\char 37\relax}$) are continuous;
the rest is categorical or discrete. Further, the provided bounds and sampling
distributions of the context spaces that are provided as part of CARL enable
better comparability and reproducibility for future research efforts in the
realm of general RL agents.

Summary Comparing our benchmarks along these attributes, we see a wide spread
in most of them (Figure 11). CARL focuses on popular environments and will
grow over time, increasing the diversity of benchmarks. Already now, CARL
provides a benchmarking collection that tasks agents with generalizing in
addition to solving the problem most common in modern RL while providing a
platform for reproducible research.

Figure 11: Characteristics of each environment of the environment families
showing the action space size, state space size (log-scale), number of context
features ($n_{cf}$), the number of context features directly shaping the
reward ($n_{cf,reward}$) and the ones changing the dynamics
($n_{cf,dynamics}$). All axes are scaled to the global extrema and the state
space size is additionally on a logarithmic scale.

###  A.3 Pendulum’s Dynamic Equations

Figure 12: CARLPendulumEnv

Because we use CARLPendulumEnv embedding gym’s Pendulum [7] for our task
variation experiment (see Section 6), we provide the dynamic equations to show
the simplicity of the system. The state consists of the angular position
$\theta$ and velocity $\dot{\theta}$ of the pendulum. The discrete equation
defining the behavior of the environment is defined as follows:

$\displaystyle\dot{\theta}_{k+1}$
$\displaystyle=\dot{\theta}_{k}+\left(-\frac{3{{\color[rgb]{0.4,0.4,1.}\definecolor[named]{pgfstrokecolor}{rgb}{0.4,0.4,1.}g}}}{2{{\color[rgb]{0.796875,0.46875,0.734375}\definecolor[named]{pgfstrokecolor}{rgb}{0.796875,0.46875,0.734375}l}}}\sin(\theta_{k}+\pi)+\frac{3}{{{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}m}}\cdot{{\color[rgb]{0.796875,0.46875,0.734375}\definecolor[named]{pgfstrokecolor}{rgb}{0.796875,0.46875,0.734375}l}}^{2}}{{\color[rgb]{0.0078125,0.6171875,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.6171875,0.44921875}u}}_{k}\right)\cdot{{\color[rgb]{0.8671875,0.55859375,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.8671875,0.55859375,0.01953125}\Delta}{\color[rgb]{0.8671875,0.55859375,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.8671875,0.55859375,0.01953125}t}}$
$\displaystyle\theta_{k+1}$
$\displaystyle=\theta_{k}+\dot{\theta}_{k+1}\cdot{{\color[rgb]{0.8671875,0.55859375,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.8671875,0.55859375,0.01953125}\Delta}{\color[rgb]{0.8671875,0.55859375,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.8671875,0.55859375,0.01953125}t}}\,.$

Here, $k$ is the index of the iteration/step. The dynamic system is
parametrized by the context, which consists of
${\color[rgb]{0.4,0.4,1.}\definecolor[named]{pgfstrokecolor}{rgb}{0.4,0.4,1.}g}$
the gravity,
${\color[rgb]{0.796875,0.46875,0.734375}\definecolor[named]{pgfstrokecolor}{rgb}{0.796875,0.46875,0.734375}l}$
and
${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}m}$
the length and mass of the pendulum,
${\color[rgb]{0.0078125,0.6171875,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.6171875,0.44921875}u}$
the control input and
${\color[rgb]{0.8671875,0.55859375,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.8671875,0.55859375,0.01953125}\Delta}{\color[rgb]{0.8671875,0.55859375,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.8671875,0.55859375,0.01953125}t}$
the timestep. Figure 12 shows how Pendulum is embedded in CARL.

##  Appendix B Hyperparameters and Hardware

#### Hyperparameters and Training Details

Table 1: TD3 Hyperparameters noise ($\mu$) | $0$
---|---
noise ($\sigma$) | $0.2$
noise ($\theta$) | $0.15$
noise decay | $0.99$
learning rate | $0.001$
$\gamma$ | $0.98$
pi target tau | $0.001$
replay buffer size | $200000$
warmup frames | $10000$
policy warmup frames | $15000$

For TD3 on Pendulum we used our own implementation and the hyperparameters in
Table 1. For cGate the width of the context branch is $512$.

We used agents from stable baselines 3 [46] (version 1.1.0) for the “Landing
in Space” scenario as well as for the experiments in the appendix (DDPG, PPO
and DQN). The hyperparameters for the stable baselines 3 agents are adopted
from the stable baselines zoo [45].

#### Hardware

All experiments on all benchmarks were conducted on a slurm CPU and GPU
cluster (see Table 2). On the CPU partition there are 1592 CPUs available
across nodes.

Table 2: GPU cluster used for training

Type | Model | Quantity | RAM CPU (G)
---|---|---|---
GPU | NVIDIA Quattro M5000 | $1$ | $256$
GPU | NVIDIA RTX 2080 Ti | $56$ | $384$
GPU | NVIDIA RTX 2080 Ti | $12$ | $256$
GPU | NVIDIA RTX 1080 Ti | $6$ | $512$
GPU | NVIDIA GTX Titan X | $4$ | $128$
GPU | NVIDIA GT 640 | $1$ | $32$

##  Appendix C Partial Observability in cMDPs

Discussing the visibility of context for an agent could in some ways be linked
to the partial observability we see in POMDPs. We believe it is useful to
differentiate between the visibility of context and state features as both
serve a different function in a cMDP. The state features describe the current
state while the context describes the current MDP. Therefore making one only
partially observable should influence the learning dynamics in different ways.
Therefore we define a cMDP as a special case of a POMDP, analogous to Kirk et
al., [28], where we have an emission function $\phi:\mathcal{S}\times
p_{\mathcal{C}}\rightarrow O_{s}\times O_{c}$ mapping the state space to some
state observation space $O_{s}$ and context observation space $O_{c}$. $\phi$
differentiates between state $s$ and context $c$ to allow different degrees of
observability in state and context, e.g. hiding the context completely but
exposing the whole state, in order to enable more flexible learning.

##  Appendix D Additional Experimental Results

In this section we provide additional information and results for our
experiments section (section 6).

###  D.1 Task Variation Through Context

Following the experimental setup in Section 6 we conducted further experiments
on representative CARL-environments.

CARLMountainCarEnv In Figure 13 we see that the DQN [39] agent is robust to
most changing context features. Varying goal_velocity, goal_position and
max_position affect the performance most. Altering those leads to a higher
required policy diversity. For example, when increasing the max_position the
car can move beyond the hill down again because the mountains are modeled with
a sinusoid.

CARLHalfcheetah In Figure 14 we see that the PPO [53] agent is robust to
varying context if we hide it. Unintuitively, the performance drops for
visible context. This might be due to unnormalized observations and context
features in combination. The difference is most pronounced for
angular_damping, friction and gravity.

###  D.2 CARLMarioEnv

To further illustrate the influence of varying context, we show experimental
results for a PPO agent trained on the CARLMarioEnv. In CARLMarioEnv different
instances (Mario levels) are created by using TOAD-GAN [51]. By varying the
noise input vector for TOAD-GAN we can generate different levels and the
greater the noise, the greater the differences to the original level. Because
CARLMarioEnv is pixel-based the context is implicitly coded in the state and
we hide the explicit context vector. As we can see in Figure 15(a) a diverse
training distribution ($\sigma_{\text{rel}}=0.2$) increases the performance
and facilitates generalization. On the other hand if the noise becomes too
large ($\sigma_{\text{rel}}=0.5$) the performance decreases again. A reason
might be that the levels for this noise level are very diverse and thus the
current setup, with only implicit context, might not be suitable.

One context feature that heavily influences the dynamics in CARLMarioEnv
environment is the inertia of Mario. In Figure 15(b), we see that a higher
variation of the inertia improves the performance of the PPO agent and leads
to faster training. This effect can be explained by the influence of Mario’s
inertia on the exploration of the agent (i.e. a lower inertia makes it easier
for the agent to move).

(a) Hidden Context

(b) Visible Context

(c) Hidden vs. Visible, $\sigma_{rel}=0.5$

Figure 13: Task Variation on CARLMountainCarEnv with DQN

(a) Hidden Context

(b) Visible Context

(c) Hidden vs. Visible, $\sigma_{rel}=0.5$

Figure 14: Task Variation on CARLHalfcheetah with PPO

(a) Varying noise on the input vector.

(b) Varying inertia.

Figure 15: Training performance on CARLMarioEnv.

###  D.3 Landing In Space

Normal Distribution In our “Landing in Space” scenario (section 7), we train
and test the LunarLander in- and out-of-distribution on carefully defined
context distributions. The LunarLander should land on seen and unseen planets.
The different planets are modeled by varying the gravity context feature. See
Figure 16 for a visualization of the sampled train contexts and the train
distribution.

(a) Sampled Train Contexts

(b) Train Distribution

Figure 16: For the “Landing in Space” scenario we sample (16(a)) training
contexts (gravity) from (16(b)) the gravity distribution. Test planets (test
contexts) are marked with a magenta vertical line.

Two-Interval Distribution Because we modeled the gravity distribution as a
normal distribution, we want to now model it as uniform intervals, see Figure
17(a). The gravity $g$ is uniformly sampled from
$g\sim\mathcal{U}($-20\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$,$-15\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$)$
and from
$g\sim\mathcal{U}($-5\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$,$-0.001\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$)$.
Intervals provide a clear cutoff. Again, we increase the game over penalty
from $-100$ to $-10000$ during testing to distinguish between successful but
suboptimal landings and crashes. In general, providing the varying context
feature gravity to the agent shows a clear benefit: The agent reaches a higher
reward in all cases, see Figure 17(b). Furthermore, we can observe that with
higher magnitude of gravity the difficulty also increases. That is the case
because the agent has less time to act before reaching the ground if the
gravity increases. In addition, if we compare the performance on the gravities
$g\in\\{$-14\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$,$-11\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$\\}$
(in-distribution) to the performance on
$g\in\\{$-8\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$,$-6\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$\\}$
(out-of-distribution) we notice more crashes on the out-of-distribution case.
The large spread of the rewards for the lowest gravity
($g=$-0.001\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$$) can also be
attributed to the long timeframe available for landing as the landing duration
is penalized by the environment.

(a) Sampled Train Contexts

(b) Test Performance

Figure 17: (17(a)) For another “Landing in Space” scenario we sample training
contexts (gravity) from two intervals. Test planets (test contexts) are marked
with a magenta vertical line. (17(b)) Test performance. The train intervals
are highlighted in grey.

###  D.4 Context-Conditioned Policies

In Figure 18 we show the broken-down results from the context-conditioning
experiment for the Pendulum environment contextualized with CARL. Overall,
cGate performs consistently well, except for varying all context features with
high magnitude. In this case it can be argued that cGate needs more training
as the learning curve indicates that training has not converged yet.
Concatenating all, raw context features or just the changing ones can work at
times, but is also unstable and performs worse than cGate. In summary, cGate
offers a reliable and well performing method to provide context information to
the agent.

In addition, we also show the results for the context-conditioning experiment
on Ant in Figure 19. We either vary nothing, vary torso_mass, friction and
gravity on their own and together. Here, providing context information via
cGate is the only viable option. Maybe the observations and the context should
be normalized when concatenating the raw context feature values.

(a) Train Performance for No Variation

(b) Train Performance for Varying Single Context Features

(c) Train Performance for Varying All Context Features

(d) Train Performance for No Variation, Varying One and All

Figure 18: Context-conditioning on CARLPendulumEnv. The legend indicates the
method of providing context information. We can either concatenate all, raw
context features (concat all), concatenate the changing context features
(concat) or use our cGate.

(a) Train Performance for No Variation

(b) Train Performance for Varying Single Context Features

(c) Train Performance for Varying All Context Features

(d) Train Performance for No Variation, Varying One and All

Figure 19: Context-conditioning on CARLAnt. The legend indicates the method of
providing context information. We can either concatenate all, raw context
features (concat all) or use our cGate. We plot single runs because the
evaluations happened at different steps.

##  Appendix E Hyperparameter Optimization in cRL

We have observed significant differences in learning performance for hidden
and full visible context, but the same is also true for hyperparameter tuning
in both of these settings. We use the same DQN and DDPG algorithms as in our
other experiments with a narrow context distribution of $0.1$ for the CARL
Pendulum, Acrobot and LunarLander environments to show this point. To tune the
hyperparameters, we use PB2 [24] for learning rate, target update interval and
discount factor.

As shown in Figure 20, the evaluation performances of the found hyperparameter
schedules differ significantly in terms of learning speed, stability and
results per environment. Contrary to what we have observed for policies in
Section 6, providing the context seems to increase the difficulty of the
problem, i.e. finding a good hyperparameter configuration happens more often
and more reliably when the policy is not given the context. We can only
speculate on the reasons why this happens, but shows that context introduces
complexities to the whole training process beyond simply the policy
architecture.

(a) CARLAcrobotEnv

(b) CARLLunarLanderEnv

(c) CARLPendulumEnv

Figure 20: Hyperparameter Optimization with PB2 [24]. Hidden means that the
context is hidden, visible means that the full context is appended to the
state.

##  Appendix F Open Challenges in cRL

We used the concept of cRL and its instantiation in CARL to demonstrate the
usefulness of context information in theory and in practice. More
specifically, we showed that making such information about the environment
explicitly available to the agent enables faster training and transfer of
agents (see Section 6). While this already provides valuable insights to the
community that increasingly cares about learning agents capable of
generalization (see Sections 1 & 3) cRL and by extension CARL enables to study
further open challenges for general RL.

###  F.1 Challenge I: Representation Learning

Our experiments demonstrated that an agent with access to context information
is capable of learning better than an agent that has to learn behaviors given
an implicit context via state observations. This provides evidence that
disentangling the representation learning aspect from the policy learning
problem reduces complexity. As CARL provides a ground truth for
representations of environment properties we envision future work on
principled studies of novel RL algorithms that, by design, disentangle
representation learning and policy learning (see, e.g., [47, 18, 64] as first
works along this line of research). The ground truth given by the context
would allow to measure the quality of learned representations and allows us to
relate this to true physical properties of an environment.

Another direction of research under the umbrella of representation learning
follows the work of environment probing policies [67]. There, exploratory
policies are learned that allow to identify which environment type an agent
encounters. This is complementary to the prior approaches as representations
are not jointly learned with the behaviour policies as in the previously
discussed approaches but rather in a separate offline phase. Based on CARL,
huge amounts of meta-data could be collected that will enable the community to
make use of classical meta-algorithmic approaches such as algorithm selection
[48] for selecting previously learned policies or learning approaches.

###  F.2 Challenge II: Uncertainty of RL Agents

With the access to context information we are able to study the influence of
noise on RL agents in a novel way. While prior environments enabled studies on
the behavior of agents when they could not be certain about their true state
in a particular environment, the framework of cRL further allows to study
agents behaviors in scenarios with uncertainty on their current contextual
environment, e.g., because of noise on the context features. In practical
deployment of RL, this is a reasonable concern since context feature have to
be measured somehow by potentially noisy sensors. As this setting affects the
overall transition dynamics, cRL provides a unique test-bed in which the
influence of uncertainty can be studied and how RL agents can deal with such.

###  F.3 Challenge III: Continual Learning

In the current definition of cRL, continual learning, i.e. shifting context
during learning can be formalized as long as the context only changes between
episodes. This is especially interesting for long term practical deployment of
RL agents as their environment may similarly shift over time. With the
flexibility and easy modifiability of contexts, cRL can therefore be used for
studying continual reinforcement learning agents. To take an example from
CARL, we could evaluate the behaviour of an agent in the Brax environments
where one or more joints become stiffer over time. A learning agent would need
to be able to handle this and adapt its gait accordingly. In particular, one
could at some point “repair” the agent and reset the joints to their original
stiffness. This would then allow to evaluate whether the agent has “unlearned”
the original gait. In the same way, cRL also allows to study how agents would
react to spontaneous, drastic changes, e.g., broken legs or changes of the
environment such as changes of weather conditions.

###  F.4 Challenge IV: Interpretable and Explainable Deep RL

Trust in the policy is a crucial factor, for which interpretability or
explainability often is mandatory. With the provided ground truth through the
explicit use of context features, cRL could be the base for studying
interpretability and explainability of (deep) RL. By enabling AutoRL studies
and different representation learning approaches, cRL will contribute to
better interpret the training procedures.

cRL further allows to study explainability on the level of learned policies.
We propose to study the sensitivity of particular policies to different types
of context. Thus, the value and variability of a context might serve as a
proxy to explain the resulting learned behavior. Such insights might then be
used to predict how policies might look like or act (e.g., in terms of
frequency of action usage) in novel environments, solely based on the provided
context features.

###  F.5 Challenge V: AutoRL

AutoRL [43] addresses the optimization of the RL learning process. To this
end, hyperparameters, architectures or both of agents are adapted either on
the fly [25, 16] or once at the beginning of a run [49]. However, as AutoRL
typically requires large compute resources for this procedure, optimization is
most often done only on a per-environment basis. It is reasonable to assume
that such hyperparameters might not transfer well to unseen environments, as
the learning procedures were not optimized to be robust or to facilitate
generalization, but only to improve the reward on a particular instance.

As we have shown above in Appendix E, cRL provides an even greater challenge
for AutoRL methods. On the other hand, as CARL provides easy-to-use contextual
extensions of a diverse set of RL problems, it could be used to drive research
in this open challenge of AutoRL. First of all, it enables a large scale-study
to understand how static and dynamic configuration approaches complement each
other and when one approach is to be preferred over another. Such a study will
most likely also lead to novel default hyperparameter configurations that are
more robust and tailored to fast learning and good generalization. In
addition, it will open up the possibility to study whether it is reasonable to
use a single hyperparameter configurations or whether a mix of configurations
for different instances is required [60]. Furthermore, with the flexibility of
defining a broad variety of instance distributions for a large set of provided
context features, experiments with CARL would allow researchers to study which
hyperparameters play a crucial role in learning general agents similar to
studies done for supervised machine learning [57] or AI algorithms [5].

###  F.6 Challenge VI: High Confidence Generalization

The availability of explicit context enables tackling another challenge in the
field of safe RL. High Confidence Generalization Algorithms (HCGAs) [30]
provide safety guarantees for the generalization of agents in testing
environments. Given a worst-case performance bound, the agent can be tasked to
decide whether a policy is applicable in an out-of-distribution context or
not. This setting is especially important for the deployment of RL algorithms
in the real world where policy failures can be costly and the context of an
environment is often prone to change. cRL has the potential to facilitate the
development of HCGAs that base their confidence estimates on the context of an
environment.

##  Appendix G Context Features for Each Environment

We list all registered context features with their defaults, bounds and types
for each environment family in Table 3 (classic control), Table 4 (box2d),
Table 5 (brax) and Table 6 (RNA and Mario).

Table 3: Context Features: Defaults, Bounds and Types for OpenAI gym’s Classic
Control environments [7]

(a) CARLCartPoleEnv Context Feature | Default | Bounds | Type
---|---|---|---
force_magnifier | 10.00 | (1, 100) | int
gravity | 9.80 | (0.1, inf) | float
masscart | 1.00 | (0.1, 10) | float
masspole | 0.10 | (0.01, 1) | float
pole_length | 0.50 | (0.05, 5) | float
update_interval | 0.02 | (0.002, 0.2) | float

(b) CARLPendulumEnv Context Feature | Default | Bounds | Type
---|---|---|---
dt | 0.05 | (0, inf) | float
g | 10.00 | (0, inf) | float
l | 1.00 | (1e-06, inf) | float
m | 1.00 | (1e-06, inf) | float
max_speed | 8.00 | (-inf, inf) | float

(c) CARLMountainCarEnv Context Feature | Default | Bounds | Type
---|---|---|---
force | 0.00 | (-inf, inf) | float
goal_position | 0.50 | (-inf, inf) | float
goal_velocity | 0.00 | (-inf, inf) | float
gravity | 0.00 | (0, inf) | float
max_position | 0.60 | (-inf, inf) | float
max_speed | 0.07 | (0, inf) | float
min_position | -1.20 | (-inf, inf) | float
start_position | -0.50 | (-1.5, 0.5) | float
start_position_std | 0.10 | (0.0, inf) | float
start_velocity | 0.00 | (-inf, inf) | float
start_velocity_std | 0.00 | (0.0, inf) | float

(d) CARLAcrobotEnv Context Feature | Default | Bounds | Type
---|---|---|---
link_com_1 | 0.50 | (0, 1) | float
link_com_2 | 0.50 | (0, 1) | float
link_length_1 | 1.00 | (0.1, 10) | float
link_length_2 | 1.00 | (0.1, 10) | float
link_mass_1 | 1.00 | (0.1, 10) | float
link_mass_2 | 1.00 | (0.1, 10) | float
link_moi | 1.00 | (0.1, 10) | float
max_velocity_1 | 12.57 | (1.26, 125.66) | float
max_velocity_2 | 28.27 | (2.83, 282.74) | float

Table 4: Context Features: Defaults, Bounds and Types for OpenAI gym’s Box2d
environments [7]

(a) CARLBipedalWalkerEnv Context Feature | Default | Bounds | Type
---|---|---|---
FPS | 50.00 | (1, 500) | float
FRICTION | 2.50 | (0, 10) | float
GRAVITY_X | 0.00 | (-20, 20) | float
GRAVITY_Y | -10.00 | (-20, -0.01) | float
INITIAL_RANDOM | 5.00 | (0, 50) | float
LEG_DOWN | -0.27 | (-2, -0.25) | float
LEG_H | 1.13 | (0.25, 2) | float
LEG_W | 0.27 | (0.25, 0.5) | float
LIDAR_RANGE | 5.33 | (0.5, 20) | float
MOTORS_TORQUE | 80.00 | (0, 200) | float
SCALE | 30.00 | (1, 100) | float
SPEED_HIP | 4.00 | (1e-06, 15) | float
SPEED_KNEE | 6.00 | (1e-06, 15) | float
TERRAIN_GRASS | 10.00 | (5, 15) | int
TERRAIN_HEIGHT | 5.00 | (3, 10) | float
TERRAIN_LENGTH | 200.00 | (100, 500) | int
TERRAIN_STARTPAD | 20.00 | (10, 30) | int
TERRAIN_STEP | 0.47 | (0.25, 1) | float
VIEWPORT_H | 400.00 | (200, 800) | int
VIEWPORT_W | 600.00 | (400, 1000) | int

(b) CARLLunarLanderEnv Context Feature | Default | Bounds | Type
---|---|---|---
FPS | 50.00 | (1, 500) | float
GRAVITY_X | 0.00 | (-20, 20) | float
GRAVITY_Y | -10.00 | (-20, -0.01) | float
INITIAL_RANDOM | 1000.00 | (0, 2000) | float
LEG_AWAY | 20.00 | (0, 50) | float
LEG_DOWN | 18.00 | (0, 50) | float
LEG_H | 8.00 | (1, 20) | float
LEG_SPRING_TORQUE | 40.00 | (0, 100) | float
LEG_W | 2.00 | (1, 10) | float
MAIN_ENGINE_POWER | 13.00 | (0, 50) | float
SCALE | 30.00 | (1, 100) | float
SIDE_ENGINE_AWAY | 12.00 | (1, 20) | float
SIDE_ENGINE_HEIGHT | 14.00 | (1, 20) | float
SIDE_ENGINE_POWER | 0.60 | (0, 50) | float
VIEWPORT_H | 400.00 | (200, 800) | int
VIEWPORT_W | 600.00 | (400, 1000) | int

(c) CARLVehicleRacingEnv Context Feature | Default | Bounds | Type
---|---|---|---
VEHICLE | 0 | - |  categorical, $n=29$

Table 5: Context Features: Defaults, Bounds and Types for Google Brax
environments [17]

(a) CARLAnt Context Feature | Default | Bounds | Type
---|---|---|---
actuator_strength | 300.00 | (1, inf) | float
angular_damping | -0.05 | (-inf, inf) | float
friction | 0.60 | (-inf, inf) | float
gravity | -9.80 | (-inf, -0.1) | float
joint_angular_damping | 35.00 | (0, inf) | float
joint_stiffness | 5000.00 | (1, inf) | float
torso_mass | 10.00 | (0.1, inf) | float

(b) CARLHalfcheetah Context Feature | Default | Bounds | Type
---|---|---|---
angular_damping | -0.05 | (-inf, inf) | float
friction | 0.60 | (-inf, inf) | float
gravity | -9.80 | (-inf, -0.1) | float
joint_angular_damping | 20.00 | (0, inf) | float
joint_stiffness | 15000.00 | (1, inf) | float
torso_mass | 9.46 | (0.1, inf) | float

(c) CARLFetch Context Feature | Default | Bounds | Type
---|---|---|---
actuator_strength | 300.00 | (1, inf) | float
angular_damping | -0.05 | (-inf, inf) | float
friction | 0.60 | (-inf, inf) | float
gravity | -9.80 | (-inf, -0.1) | float
joint_angular_damping | 35.00 | (0, inf) | float
joint_stiffness | 5000.00 | (1, inf) | float
target_distance | 15.00 | (0.1, inf) | float
target_radius | 2.00 | (0.1, inf) | float
torso_mass | 1.00 | (0.1, inf) | float

(d) CARLGrasp Context Feature | Default | Bounds | Type
---|---|---|---
actuator_strength | 300.00 | (1, inf) | float
angular_damping | -0.05 | (-inf, inf) | float
friction | 0.60 | (-inf, inf) | float
gravity | -9.80 | (-inf, -0.1) | float
joint_angular_damping | 50.00 | (0, inf) | float
joint_stiffness | 5000.00 | (1, inf) | float
target_distance | 10.00 | (0.1, inf) | float
target_height | 8.00 | (0.1, inf) | float
target_radius | 1.10 | (0.1, inf) | float

(e) CARLHumanoid Context Feature | Default | Bounds | Type
---|---|---|---
angular_damping | -0.05 | (-inf, inf) | float
friction | 0.60 | (-inf, inf) | float
gravity | -9.80 | (-inf, -0.1) | float
joint_angular_damping | 20.00 | (0, inf) | float
torso_mass | 8.91 | (0.1, inf) | float

(f) CARLUr5e Context Feature | Default | Bounds | Type
---|---|---|---
actuator_strength | 100.00 | (1, inf) | float
angular_damping | -0.05 | (-inf, inf) | float
friction | 0.60 | (-inf, inf) | float
gravity | -9.81 | (-inf, -0.1) | float
joint_angular_damping | 50.00 | (0, 360) | float
joint_stiffness | 40000.00 | (1, inf) | float
target_distance | 0.50 | (0.01, inf) | float
target_radius | 0.02 | (0.01, inf) | float
torso_mass | 1.00 | (0, inf) | float

Table 6: Context Features: Defaults, Bounds and Types for RNA Design [49] and
Mario Environment [3, 51]

(a) CARLRnaDesignEnv Context Feature | Default | Bounds | Type
---|---|---|---
mutation_threshold | 5 | (0.1, inf) | float
reward_exponent | 1 | (0.1, inf) | float
state_radius | 5 | (1, inf) | float
dataset | eterna | - |  categorical, $n=3$
target_structure_ids | f(dataset) | (0, inf) | list of int

(b) CARLMarioEnv Context Feature |  Default | Bounds | Type
---|---|---|---
level_index |  0 | - |  categorical, $n=15$
noise |  f(level_index, width, height) | (-1, 1) | float
mario_state |  0 | - |  categorical, $n=3$
mario_inertia |  0.89 | (0.5, 1.5) | float