# Evolving Pareto-Optimal Actor-Critic Algorithms
for Generalizability and Stability

Juan Jose Garau-Luis1, Yingjie Miao2, John D. Co-Reyes2, Aaron Parisi2
Jie Tan2, Esteban Real2, Aleksandra Faust2
1MIT, 2Google Brain

<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Generalizability and stability are two key objectives for operating
reinforcement learning (RL) agents in the real world. Designing RL algorithms
that optimize these objectives can be a costly and painstaking process. This
paper presents MetaPG, an evolutionary method for automated design of actor-
critic loss functions. MetaPG explicitly optimizes for generalizability and
performance, and implicitly optimizes the stability of both metrics. We
initialize our loss function population with Soft Actor-Critic (SAC) and
perform multi-objective optimization using fitness metrics encoding single-
task performance, zero-shot generalizability to unseen environment
configurations, and stability across independent runs with different random
seeds. On a set of continuous control tasks from the Real-World RL Benchmark
Suite, we find that our method, using a single environment during evolution,
evolves algorithms that improve upon SAC’s performance and generalizability by
4% and 20%, respectively, and reduce instability up to 67%. Then, we scale up
to more complex environments from the Brax physics simulator and replicate
generalizability tests encountered in practical settings, such as different
friction coefficients. MetaPG evolves algorithms that can obtain 10% better
generalizability without loss of performance within the same meta-training
environment and obtain similar results to SAC when doing cross-domain
evaluations in other Brax environments. The evolution results are
interpretable; by analyzing the structure of the best algorithms we identify
elements that help optimizing certain objectives, such as regularization terms
for the critic loss.

##  1 Introduction

Figure 1: In many practical contexts, designing the RL agent that achieves the
right performance, generalizability, and stability is an empirical and costly
iteration-based process.

Two key bottlenecks for the deployment of reinforcement learning (RL) in the
real world are failing to generalize beyond the training distribution and
unstable training. Both are common in practical settings and constitute two
important aspects of RL robustness. On one hand, many real-world environments
present themselves in multiple configurations (e.g., different sizes,
structure, context, properties) and practitioners expect zero-shot
generalization when facing new configurations in robot manipulation (Ibarz et
al., 2021), navigation (Chiang et al., 2019), energy systems (Perera &
Kamalaruban, 2021), and fluid dynamics (Garnier et al., 2021). On the other
hand, real-world elements such as stochastic dynamics should not result in
unstable learning behaviors that lead to undesired performance drops. Even for
state-of-the-art RL algorithms, zero-shot generalization and instability are
considerable challenges (Henderson et al., 2018; Dulac-Arnold et al., 2021).

Improving generalizability has been addressed by learning or encoding
inductive biases in RL algorithms (Raileanu & Fergus, 2021; Vlastelica et al.,
2021). Gains mostly come by manually modifying existing algorithms (Cobbe et
al., 2019; Igl et al., 2019; Cobbe et al., 2021). As RL design for real-world
environments tends to be empirical (Andrychowicz et al., 2020; Linke et al.,
2020), finding the suitable algorithmic changes that benefit generalizability
might take many iterations, especially if performance loss must be avoided
(see Figure 1). As environments become more complex and inductive biases
become environment-specific, the cost of human-driven design might be too
expensive when optimizing for generalizability (Zhao et al., 2019), let alone
when optimizing for both performance and generalizability (Hessel et al.,
2019), which are just two objectives we consider but there are many more. In
addition, stable algorithms that show consistent performance and
generalizability across independent runs leads to higher algorithm reusability
within the same environment and across environments. We argue that optimizing
generalizability and stability in addition to performance in RL builds the
case for automating algorithm design and speeding up the process of RL
algorithm discovery. Automated Machine Learning or AutoML (Hutter et al.,
2019) has proven to be a successful tool for supervised learning problems
(Vinyals et al., 2016; Zoph et al., 2018; Real et al., 2019; Finn et al.,
2017), and it has been recently applied in the context of RL for automating
loss function search (Oh et al., 2020; Xu et al., 2020; Co-Reyes et al., 2021;
Bechtle et al., 2021; He et al., 2022; Lu et al., 2022).

This paper proposes MetaPG (see Figure 2), a method that evolves a population
of actor-critic RL algorithms (Sutton & Barto, 2018), identified by their loss
functions, with the goal of increasing single-task performance, zero-shot
generalizability, and stability across independent runs. Loss functions are
represented symbolically as directed acyclic computation graphs and two
independent fitness scores are used to encode performance and
generalizability, respectively. A measure for stability is accounted for in
both objectives and the multi-objective ranking algorithm NSGA-II (Deb et al.,
2002) is used to identify the best algorithms. Compared to manual design, this
strategy allows us to explore the algorithm space more efficiently by
automating search operations. MetaPG finds algorithm improvement directions
that jointly optimize both objectives until it obtains a Pareto Front of loss
functions that maximizes fitness with respect to each objective, approximating
the underlying tradeoff between them. On one end of the Pareto Front we obtain
algorithms that perform well in the training task (beneficial when faster
learning is required and overfitting is not a concern), on the other end we
find algorithms that generalize better to unseen configurations, and in
between there are algorithms that interpolate between both behaviors.

To evaluate MetaPG, we run experiments using, first the zero-shot
generalizability benchmark from the Real-World RL environment suite (Dulac-
Arnold et al., 2021); and second, the Ant and Humanoid environments from Brax
physics simulator Freeman et al. (2021), where we simulate perturbations like
mass changes, different friction coefficients, and joint torques. We warm-
start the evolution with a graph-based representation of Soft Actor-Critic
(SAC) (Haarnoja et al., 2018), and demonstrate that our method is able to
evolve a Pareto Front of multiple actor-critic algorithms that outperform SAC
and increase its performance and generalizability up to 4% and 20%,
respectively, and decrease instability up to 67%. In the Brax environments,
the evolved algorithms outperform SAC by 15% and 10% in performance and
generalizability, respectively. Instability is reduced 23% in that case.
Furthermore, we observe that algorithms evolved in Ant show minimal
generalizability loss when transferred to Humanoid and vice versa. Finally, by
inspecting the graphs of the evolved algorithms, we interpret which
substructure drive the gains. For instance, we find that MetaPG evolves loss
functions that remove the entropy term in SAC to increase performance.

Figure 2: MetaPG overview, example with two fitness scores encoding two RL
objectives. (a) The method starts by taking a warm-start RL algorithm with its
loss function represented in the form of a directed acyclic graph. MetaPG
consists of a meta-evolution process that, after initializing algorithms to
the warm-start, discovers a population of new algorithms. (b) Each evolved
graph is evaluated by training an agent following the algorithm encoded by it,
and then computing two fitness scores based on the training outcome. (c) After
evolution, all RL algorithms can be represented in the fitness space and a
Pareto-optimal set of algorithms can be identified. (d) Identifying which
graph substructures change across the algorithms in the Pareto set reveals
which operations are useful for specific RL objectives. MetaPG can be scaled
to more than two RL objectives.

In summary, this paper makes three main contributions:

  1. 1.

A method that combines multi-objective evolution with a search language
representing actor-critic RL algorithms as graphs, which can discover new loss
functions over a set of different objectives.

  2. 2.

A formulation of stability-adjusted scores that explicitly optimize for
performance and generalizability and implicitly encourage stability.

  3. 3.

A dataset111The dataset can be found at:
https://github.com/authors2022/dataset of Pareto-optimal actor-critic loss
functions which outperform baselines like SAC on multiple objectives. This
dataset may be further analyzed to understand how algorithmic changes affect
the tradeoff between different objectives.

##  2 Related Work

Generalizability in RL One of the aspects of RL robustness is generalizability
(Kirk et al., 2022; Xu et al., 2022). Increasing generalizability has been
addressed by means of environment randomization (Tobin et al., 2017; Peng et
al., 2018; Akkaya et al., 2019). Other authors have shown that removing or
adding certain algorithmic components impacts generalizability (e.g. using
batch normalization (Cobbe et al., 2019), adding elements to rewards (Chen,
2020), or using regularizers (Igl et al., 2019; Cobbe et al., 2019)). Others
directly achieve generalizability gains by modifying existing actor-critic RL
algorithms (Raileanu & Fergus, 2021; Cobbe et al., 2021). Vlastelica et al.
(2021) propose a hybrid architecture combining a neural network with a
shortest path solver to find better inductive biases that improve generalizing
to unseen environment configurations. We automate the search of algorithmic
changes for actor-critic algorithms using generalizability as one of the
search metrics.

Stability in RL Achieving stable behaviors during training is essential in
many domains, particularly in control applications (Ibarz et al., 2021; Azar
et al., 2021). It has been shown that randomness can play a substantial role
in the outcome of a training run (Henderson et al., 2018). Stable learning has
been sought by means of algorithmic innovation (Haarnoja et al., 2018; Fox et
al., 2015; Bao et al., 2021; Jiang et al., 2021). New stable algorithms have
been mainly developed after looking into stability in isolation; in this work
we focus on stability as one of the three objectives to simultaneously
optimize.

Optimizing RL components Automated RL or AutoRL seeks to meta-learn RL
components (Parker-Holder et al., 2022), such as RL algorithms (Co-Reyes et
al., 2021; Kirsch et al., 2020; Oh et al., 2020; Bechtle et al., 2021), their
hyperparameters (Zhang et al., 2021; Hertel et al., 2020; Xu et al., 2018),
policy/neural network (Gaier & Ha, 2019; Miao et al., 2021), or the
environment (Ferreira et al., 2021; Gur et al., 2021; Dennis et al., 2020;
Florensa et al., 2018; Volz et al., 2018; Faust et al., 2019). This work
evolves RL loss functions and leaves other elements of the RL problem out of
the scope. We focus on the RL loss function given its interaction with all
elements in a RL problem: states, actions, rewards, and the policy.

Evolutionary AutoML Neuro-evolution introduced evolutionary methods in the
context of AutoML (Miller et al., 1989; Stanley & Miikkulainen, 2002),
including neural network architecture search (Stanley et al., 2009; Jozefowicz
et al., 2015; Real et al., 2019). In the RL context evolution searched for
policy gradients (Houthooft et al., 2018) and value iteration losses (Co-Reyes
et al., 2021). Our work is also related to the field of genetic programming,
in which the goal is to discover computer code (Koza, 1994; Real et al., 2020;
Co-Reyes et al., 2021). In this work we use a multi-objective evolutionary
method to discover new RL algorithms, specifically actor-critic algorithms
(Sutton & Barto, 2018), represented as graphs.

Learning RL loss functions Loss functions play a central role in RL algorithms
and are traditionally designed by human experts. Recently, several lines of
work propose to view RL loss functions as tunable objects that can be
optimized automatically (Parker-Holder et al., 2022). One popular approach is
to use neural loss functions whose parameters are optimized via meta-gradient
(Kirsch et al., 2020; Bechtle et al., 2021; Oh et al., 2020; Xu et al., 2020;
Lu et al., 2022). An alternative is to use symbolic representations of loss
functions and formulate the problem as optimizing over a combinatorial space.
One example is (Alet et al., 2019), which represents extrinsic rewards as a
graph and optimizes it by cleverly pruning a search space. Learning value-
based RL loss functions by means of evolution was first proposed by Co-Reyes
et al. (2021), and was applied to solving discrete action problems. He et al.
(2022) propose a method to evolve auxiliary loss functions which complemented
predefined standard RL loss functions. MetaPG focuses on continuous control
problems and searches for complete symbolic loss functions of actor-critic
algorithms.

##  3 Methods

We represent actor-critic loss functions (policy loss and critic loss) as
directed acyclic graphs and use an evolutionary algorithm to evolve a
population of graphs, which are ranked based on their fitness scores. The
population is warm-started with known algorithms such as SAC and undergoes
mutations over time. Each graph’s fitnesses are measured by training from
scratch an RL agent with the corresponding loss function and encode
performance and generalizability as explicit objectives, and stability as a
third objective defined implicitly. We use the multi-objective evolutionary
algorithm NSGA-II (Deb et al., 2002) to jointly optimize all fitness scores
until growing a Pareto-optimal set of graphs or Pareto Front. Algorithm 1
summarizes the process; Offspring and RankAndSelect are NSGA-II subroutines.

Section 3.1 provides RL algorithm graph representation details. The main logic
of MetaPG is contained in the evaluation routine, which computes fitness
scores (Section 3.2) and employs several techniques to speed up the evolution
and evaluation processes (Section 3.3). See Appendix B for further
implementation details.

Algorithm 1 MetaPG Overview

Input: Training environments $\mathcal{E}$
Initialize: Initialize population $P_{0}$ of loss function graphs (random
initialization or bootstrap with an algorithm such as SAC).

1:for $L$ in $P_{0}$ do $L.score\leftarrow\text{Eval}(L,\mathcal{E})$

2:end for

3:$Q_{0}\leftarrow$ Offspring($P_{0}$) $\triangleright$ NSGA-II

4:for $L$ in $Q_{0}$ do $L.score\leftarrow\text{Eval}(L,\mathcal{E})$

5:end for

6:for $t=1$ to $G$ do

7: $R\leftarrow P_{t-1}\bigcup Q_{t-1}$

8: $P_{t}\leftarrow$ RankAndSelect(R) $\triangleright$ NSGA-II

9: $Q_{t}\leftarrow$ Offspring($P_{t}$) $\triangleright$ NSGA-II

10: for $L$ in $Q_{t}$ do $L.score\leftarrow\text{Eval}(L,\mathcal{E})$

11: end for

12:end for

13:Output: Pareto Front of all loss function graphs.

###  3.1 RL algorithm representation

MetaPG encodes loss functions as graphs consisting of typed nodes sufficient
to represent a wide class of actor-critic algorithms. Compared to the prior
value-based RL evolutionary search method introduced by Co-Reyes et al.
(2021), MetaPG’s search space greatly expands on it and adds input and output
types to manage the search complexity. As a representative example, Figure 8
in Appendix D presents the encoding for SAC that we use in this paper. In our
experiments we limit the number of nodes per graph to 60 and 80, which can
represent approximately $10^{300}$ and $10^{400}$ graphs, respectively (see
Appendix A.2). Nodes in the graph encode loss function inputs, operations, and
loss function outputs. The inputs include elements from transition tuples,
constants such as the discount factor $\gamma$, a policy network $\pi$, and
multiple critic networks $Q_{i}$. Operation nodes support intermediate
algorithm instructions such as basic arithmetic neural network operations.
Then, outputs of the graphs correspond to the policy and critic losses. The
gradient descent minimization process takes these outputs and computes their
gradient with respect to the respective network parameters. In Appendix A.1 we
provide a full description of the search language and nodes considered.
MetaPG’s search language supports both on-policy and off-policy algorithms;
however, in this paper we focus on off-policy algorithms given their better
sample efficiency.

###  3.2 Fitness scores

This work focuses on optimizing single-task performance, zero-shot
generalizability, and stability across independent runs with different random
seeds. The process to compute the fitness scores is depicted in Figure 3. We
use $N$ random seeds and a set of environments $\mathcal{E}$, which comprises
multiple instances of the same environment class, including a training
instance $E_{train}\in\mathcal{E}$. For example, $\mathcal{E}$ is the set of
all RWRL Cartpole environments with different pole lengths (0.1 meters to 3
meters in 10-centimeter intervals), and $E_{train}$ corresponds to an instance
with a specific pole length (1 meter). The first step (see Figure 3a) is to
compute raw performance and generalizability scores for each individual seed.
Using $E_{train}$ and seed $k$ to train a policy $\pi_{k}$, the performance
score $f_{perf_{k}}$ is the average evaluation return on the training
environment configuration:

$f_{perf_{k}}=\frac{1}{N_{eval}}\sum_{n=1}^{N_{eval}}G_{n}(\pi_{k},E_{train}),$
(1)

where $G_{n}$ corresponds to the normalized return for episode $n$ given a
policy and an environment instance, and $N_{eval}$ is the number of evaluation
episodes. Algorithms that learn faster in the training environment and overfit
to it obtain higher performance scores. The generalizability score
$f_{gen_{k}}$ is in turn computed as the average evaluation return of the
policy trained on $E_{train}$ over the whole range of environment
configurations. We emphasize that the policy is trained on a single
environment configuration (for example 1.0 meter pole length) and then is
evaluated in a zero-shot fashion to new unseen environment
configurations:222More precisely, it should be
$E\in\mathcal{E}\setminus\\{E_{train}\\}$. In practice, we find this makes no
significant difference in the metric because the number of test configurations
is normally around 30.

$f_{gen_{k}}=\frac{1}{|\mathcal{E}|N_{eval}}\sum_{E\in\mathcal{E}}\sum_{n=1}^{N_{eval}}G_{n}(\pi_{k},E)$
(2)

We look for stable training results for both performance and generalizability.
To that end, once we have independent raw scores for each seed ($N$ different
performance and generalizability scores) we define the stability-adjusted
scores (see Figure 3b) as

$\tilde{f}=\mu(\\{f_{n}\\}_{n=1}^{N})-\kappa\cdot\sigma(\\{f_{n}\\}_{n=1}^{N})$
(3)

where $f$ is a score (performance or generalizability), $f_{n}$ denotes the
score for seed $n$; $\mu$ and $\sigma$ are the mean and standard deviation
across the $N$ seeds, respectively; and $\kappa$ is a penalization
coefficient. The final fitness of a graph is the tuple ($\tilde{f}_{perf}$,
$\tilde{f}_{gen}$).

Figure 3: Process to compute fitness scores. (a) Independently for each seed
from a set of $N$ seeds, we first train a policy $\pi_{k}$ using environment
$E_{train}$ and seed $k$. During training the policy is allowed to take
stochastic actions. Then, we evaluate $\pi_{k}$ deterministically on that same
environment $E_{train}$ to get a raw performance score $f_{perf_{k}}$, and on
a set of environments $\mathcal{E}$ (same environment with different
configurations; e.g., different pole lengths as shown) to get a raw
generalizability score $f_{gen_{k}}$. (b) We compute stability-adjusted
fitness scores $\tilde{f}_{perf}$ and $\tilde{f}_{gen}$ by aggregating raw
scores from each seed using Equation 3.

###  3.3 Evolution details

Mutation The population is initialized with a warm-start RL algorithm; all
individuals are copies of this algorithm’s graph at the beginning. Once the
population is initialized, individuals undergo mutations that change the
structure of their respective graphs. Specifically, mutations consist of
either replacing one or more nodes in the graph or changing the connectivity
for one edge. The specific number of nodes that are affected by mutation is
randomly sampled for each different individual; see Appendix B for more
details.

Operation consistency To prevent introducing corrupted child graphs into the
population, MetaPG checks operation consistency, i.e., for each operation, it
makes sure the shapes of the input tensors are valid and compatible, and
computes the shape of the output tensor. These shapes and checks are
propagated along the computation graph.

Hashing To avoid repeated evaluations, MetaPG hashes (Real et al., 2020) all
graphs in the population. Once the method produces a child graph and proves
its consistency, it computes a hash value and, in case of a cache hit, reads
the fitness scores from the cache. Since only the gradient of a loss function
matters during training, we hash a graph by computing the corresponding loss
function’s gradient on synthetic inputs.

Hurdle evaluations We carry out evaluations for different individuals in the
population in parallel, while evaluating across seeds for one algorithm is
done sequentially. To prevent spending too many resources on algorithms that
are likely to yield bad policies, MetaPG uses a simple hurdle environment (Co-
Reyes et al., 2021) and a number of hurdle seeds. We first evaluate the
algorithm on the hurdle environment for each hurdle seed, and only proceed
with more complex and computationally expensive environments if the resulting
policy performs above a certain threshold on the hurdle environment.

##  4 Results

This section aims to answer the following questions:

  1. 1.

Is MetaPG capable of evolving algorithms that improve upon performance,
generalizability, and stability in different practical settings?

  2. 2.

How well do discovered algorithms do in environments different from those used
to evolve them?

  3. 3.

Are the evolutionary results interpretable?

We run the experiments following the process represented in Figure 4; we
divide them into meta-training, meta-validation, and meta-testing phases. Each
phase relies on a different set of $N$ random seeds: $S_{train}$, $S_{valid}$,
and $S_{test}$, respectively. Each MetaPG run begins with the meta-training
phase, which consists of the evolution process described in the previous
section. The result of this phase is a population of evolved algorithms, each
with a pair of meta-training fitness scores. Since the evolution process is
non-deterministic, we run each experiment multiple times without configuration
changes and aggregate all resulting populations into one single larger
population.

Figure 4: Running an experiment with MetaPG is divided into three phases. 1\.
Meta-training: we evolve a population of algorithms using a set of random
seeds $S_{train}$ to compute scores. 2\. Meta-validation: to prevent
overfitting, we reevaluate the scores of all algorithms in the population
using a different set of random seeds $S_{valid}$. 3\. Meta-testing: specific
algorithms such as those in the Pareto Front are tested in different
environments using a third set of random seeds $S_{test}$.

Then, to avoid selecting algorithms that overfit to the set of seeds
$S_{train}$, we reevaluate all algorithms in the population with a different
set of seeds $S_{valid}$; this corresponds to the meta-validation phase, which
provides updated fitness scores for all algorithms. Finally, to assess the
fitness of specific algorithms (e.g., the Pareto Front) when deployed in
different environments, we use a third set of seeds $S_{test}$ that provides
realistic fitness scores in the new environments; this corresponds to the
meta-testing phase. In our analyses of the results, we focus on the set of
meta-validated algorithms and then meta-test some of them.

###  4.1 Training setup

Training environments We use as training environments: Cartpole and Walker
from the RWRL Environment Suite (Dulac-Arnold et al., 2021), Gym Pendulum, and
Ant and Humanoid from the Brax physics simulator (Freeman et al., 2021). We
define different instances of these environments by varying the pole length in
Cartpole, the thigh length in Walker, the pendulum length and mass in
Pendulum, and, to mimic a practical setting, the mass, friction coefficient,
and torque in Ant and Humanoid. See Appendix C for the specifics.

Meta-training details The population and maximum graph size consist of 100
individuals and 80 nodes in the Brax environments, respectively, and 1,000
individuals and 60 nodes in the rest of the environments. All are initialized
using SAC as a warm-start (see Appendix D). For RL algorithm evaluation, we
use 10 different seeds $S_{train}$ and fix the number of evaluation episodes
$N_{eval}$ to 20. In the case of Brax, since training takes longer, we use 4
different seeds but increase $N_{eval}$ to 32. We meta-train using 100 TPU 1x1
v2 chips for 4 days in the case of Brax environments ($\sim$200K and $\sim$50K
evaluated graphs in Ant and Humanoid, respectively), and using 1,000 CPUs for
10 days in the rest of environments ($\sim$100K evaluated graphs per
experiment). In all cases we normalize the fitness scores to the range [0, 1].
We set $\kappa=1$ in equation 3. Additional details are in Appendix B.

Meta-validation details During meta-validation, we use a set of 10 seeds
$S_{valid}$, disjoint with respect to $S_{train}$. In the case of Brax
environments, we use 4 meta-validation seeds. Same applies during meta-
testing. In each case, we use a number of seeds that achieve a good balance
between preventing overfitting and having affordable evaluation time. The
value of $N_{eval}$ during meta-validation and meta-testing matches the one
used in meta-training.

Hyperparameter tuning We use the same fixed hyperparameters during all meta-
training. Algorithms are also meta-validated using the same hyperparameters.
In the case of Brax environments, we do hyperparameter-tune the algorithms
during meta-validation; additional details can be found in Appendix G.5. We
also hyperparameter-tune all baselines we compare our evolved algorithms
against.

RL Training details The architecture of the policies corresponds to two-layer
MLPs with 256 units each. Additional training details are presented in
Appendix E.

###  4.2 Optimizing performance, generalizability, and stability in RWRL
Environment Suite

We apply MetaPG to RWRL Cartpole and compare the evolved algorithms in the
meta-validated Pareto Front with the warm-start SAC and ACME SAC (Hoffman et
al., 2020) (Figure 5(a)). When running ACME SAC we first do hyperparameter
tuning and pick the two configurations that lead to the best stability-
adjusted performance and the best stability-adjusted generalizability (ACME
SAC HPT Perf and ACME SAC HPT Gen, respectively). In Table 1, we show numeric
values for each of the three metrics. In the case of stability, we show a
measure of instability represented as the change in standard deviation
compared to the warm-start (i.e., warm-start has the value 1.0). We
independently measure instability with respect to performance and
generalizability.

(a) Stability-adjusted fitness scores (computed using Equation 3) for
algorithms in the Pareto-optimal set.

(b) Average return and standard deviation across seeds when evaluating trained
policies in multiple RWRL Cartpole instances with different pole lengths
(Training configuration is 1.0, see Appendix C).

Figure 5: Evolution results (meta-validation across 10 different seeds)
alongside the warm-start algorithm (SAC), and the hyperparameter-tuned ACME
SAC when using the RWRL Cartpole environment for training. We show the Pareto
Front of algorithms that results after merging the 10 populations
corresponding to the 10 repeats of the experiment. The best performer and best
generalizer correspond to the algorithms with the highest stability-adjusted
performance and generalizability scores, respectively, according to Equations
1, 2, and 3.

The results from Table 1 show that MetaPG discovers RL algorithms that improve
upon the warm-start’s and ACME SAC’s performance, generalizability, and
stability in the same environment we used during evolution. Compared to the
warm-start, the best performer achieves a 4% improvement in the stability-
adjusted performance score (from 0.836 to 0.868), the best generalizer
achieves a 20% increase in the stability-adjusted generalizability score (from
0.460 to 0.551), and the selected algorithm in the Pareto-optimal set (Pareto
point 6) achieves a 2% and a 12% increase in both stability-adjusted
performance and stability-adjusted generalizability, respectively. Then, in
terms of the stability objective, the best performer reduces performance
instability by 67% and the best generalizer achieves a reduction of 30% for
generalizability instability.

The gains in generalizability and stability are substantial when comparing the
results to hyperparameter-tuned ACME SAC. The best generalizer achieves a 15%
increase in stability-adjusted generalizability compared to ACME SAC tuned for
such metric. The instability in the hyperparameter-tuned SAC is twice as high
(1.48 vs. 0.70, as shown in Table 1). In terms of performance, the best
performer achieves a slightly better result compared to SAC hyperparameter-
tuned for performance.

We report the complete results in Appendix F.1 and repeat the same experiments
in RWRL Walker (Appendix F.2) and Gym Pendulum (Appendix F.3) and observe that
MetaPG also discovers a Pareto Front of algorithms that outperform SAC in both
environments. Additional information on the stability of the algorithms is
presented in Appendix F.4.

Figure 5(b) compares how the best performer and the best generalizer behave in
different instances of the environment in which we change the pole length (all
instances form the environment set $\mathcal{E}$ used during evolution). We
follow the same procedure described by Dulac-Arnold et al. (2021). The best
performer achieves better return in the training configuration than the warm-
start’s. The best generalizer in turn achieves a lower return but it trades it
for higher returns in configurations outside of the training regime, being
better at zero-shot generalization. The same behavior holds when using RWRL
Walker and Gym Pendulum as training environments (see Appendix F.2 and F.3,
respectively).

Table 1: We compare three algorithms in the Pareto Front with SAC (warm-start and hyperparameter-tuned ACME SAC) using metrics obtained in the RWRL Cartpole environment: average performance and generalizability, stability-adjusted performance and generalizability scores, and measure of instability (standard deviation $\sigma$ divided by the warm-start’s $\sigma_{WS}$). We compute these metrics across 10 seeds and the best result in a _column_ is bolded. †In contrast to performance and generalizability, the lower the instability the better. | Performance | Generalizability |  Instability† ($\sigma$/$\sigma_{WS}$)
---|---|---|---
RL Algorithm | $f_{perf}$ | $\tilde{f}_{perf}$ | $f_{gen}$ | $\tilde{f}_{gen}$ | Perf. | Gen.
Pareto point 1: Best performer | 0.871 | 0.868 | 0.475 | 0.459 | 0.33 | 0.59
Pareto point 6 | 0.854 | 0.852 | 0.531 | 0.514 | 0.22 | 0.63
Pareto point 10: Best generalizer | 0.770 | 0.756 | 0.570 | 0.551 | 1.55 | 0.70
Warm-start SAC | 0.845 | 0.836 | 0.487 | 0.460 | 1.0 | 1.0
ACME SAC HPT Perf | 0.865 | 0.864 | 0.372 | 0.312 | 0.11 | 2.22
ACME SAC HPT Gen | 0.845 | 0.833 | 0.518 | 0.478 | 1.33 | 1.48

###  4.3 Transferring evolved algorithms between Brax environments

Figure 6 shows the behaviour of evolved algorithms when meta-tested in
perturbed Brax Ant environments (changes in friction coefficient, mass, and
torque, see Appendix C). We first evolve algorithms independently in both Ant
and Humanoid, then select those algorithms that have the highest stability-
adjusted generalizability score $\tilde{f}_{gen}$ during meta-validation. We
then re-evaluate using the meta-testing seeds. We compare algorithms evolved
in Ant and Humanoid with hyperparameter-tuned SAC.

Figure 6: Meta-testing in Brax Ant. We compare, after hyperparameter tuning,
loss functions evolved in Brax Ant, Brax Humanoid (to assess cross-domain
transfer), and the SAC baseline used as warm-start. We plot the average return
and standard deviation across random seeds when evaluating in multiple Brax
Ant instances with different friction coefficients, mass coefficients, and
torque multipliers (in all cases 1.0 is used as training configuration). The
loss function evolved in Ant improves upon SAC’s performance and
generalizability.

These results highlight that an algorithm evolved by MetaPG in Brax Ant
performs and generalizes better than a SAC baseline. Specifically, we observe
a 15% improvement in stability-adjusted performance and 10% improvement in
stability-adjusted generalizability. We also obtain a 23% reduction in
instability. In addition, we observe that an algorithm initially evolved using
Brax Humanoid and meta-validated in Ant transfers reasonably well to Ant
during meta-testing, achieving slight loss of performance compared to
hyperparameter-tuned SAC (adjusting for stability, 17% less performance and
13% less generalizability compared to SAC). We evolved fewer graphs in the
case of Humanoid (50K compared to 200K evolved graphs for Brax Ant), as
training a policy in Humanoid is more costly. We expect these results to
improve if more algorithms are evolved in the population. We present complete
results for the Brax Ant environment in Appendix F.5 and repeat the same
analysis for the Brax Humanoid environment in Appendix F.6, in which we
observe similar results.

###  4.4 Analyzing the evolved RL algorithms

Next, we analyze evolved algorithms from our experiments on RWRL Cartpole. We
pick the best meta-validated performer and generalizer, both evolved from the
warm-start SAC (see Appendix D for its graph representation in the search
space). The policy loss $L_{\pi}$ and critic losses $L_{Q_{i}}$ (one for each
critic network $Q_{i}$, see Appendix A.1 for details) observed from the graph
structure for the best performer are the following:

$\displaystyle L_{\pi}^{perf}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},s_{t+1})\sim\mathcal{D}}\left[\log({\color[rgb]{0,0,1}\min(}\pi({\color[rgb]{0,0,1}\tilde{a}_{t+1}}|{\color[rgb]{0,0,1}{s_{t+1}}}){\color[rgb]{0,0,1},\gamma)})-\underset{i}{\min}\,Q_{i}(s_{t},\tilde{a}_{t})\right]$
(4) $\displaystyle L_{Q_{i}}^{perf}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[\left(r_{t}+\gamma\left({\color[rgb]{1,0,0}\underset{i}{\min}\,}Q_{targ_{i}}(s_{t+1},\tilde{a}_{t+1}){\color[rgb]{1,0,0}-\log\pi(\tilde{a}_{t+1}|s_{t+1})}\right)-Q_{i}(s_{t},a_{t})\right)^{2}\right]$
(5)

where $\tilde{a}_{t}\sim\pi(\cdot|s_{t})$,
$\tilde{a}_{t+1}\sim\pi(\cdot|s_{t+1})$, and $\mathcal{D}$ is an experience
dataset extracted from the replay buffer. We highlight in blue the changes and
additions with respect to SAC, and in red the elements of the SAC loss
function that evolution removes. The loss equations for the best generalizer
are:

$\displaystyle L_{\pi}^{gen}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},s_{t+1})\sim\mathcal{D}}\left[\log\pi(\tilde{a}_{t}|s_{t})-\underset{i}{\min}\,Q_{i}({\color[rgb]{0,0,1}s_{t+1}},\tilde{a}_{t})\right]$
(6) $\displaystyle L_{Q_{i}}^{gen}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[{\color[rgb]{0,0,1}\mathrm{atan}}\left(\left(r_{t}+\gamma\left(\underset{i}{\min}\,Q_{targ_{i}}(s_{t+1},{\color[rgb]{0,0,1}\tilde{a}_{t}})-\log\pi({\color[rgb]{0,0,1}\tilde{a}_{t}}|{\color[rgb]{0,0,1}s_{t}})\right)-Q_{i}(s_{t},a_{t})\right)^{2}\right)\right]$
(7)

(a) Average entropy of the policy during training for RWRL Cartpole.

(b) Average gradient norm of the actor loss during training for RWRL Cartpole.

Figure 7: Analysis of the entropy and gradient norm of the actor when
evaluating the best generalizer from RWRL Cartpole in comparison to the warm-
start. We hypothesize this increase in entropy and decrease in gradient norm
with respect to SAC contribute to achieve better generalizability.

While both algorithms resemble the warm-start SAC (see Appendix D), we observe
that the best performer does not include the entropy term in the critic loss
while the best generalizer does (i.e., they correspond to setting $\alpha$ to
0 and 1 in the original SAC algorithm (Haarnoja et al., 2018), respectively).
This aligns with the hypothesis that, since ignoring the entropy pushes the
agent to exploit more and explore less, the policy of the best performer
overfits better to the training configuration compared to SAC. In contrast,
the best generalizer is able to explore more. Figure 7(a) validates the latter
observation showing a higher entropy for the best generalizer’s actor compared
to the warm-start’s.

The use of arctangent in the critic loss of the best generalizer is also
noticeable as, supported by Figure 7(b), we observe this operation serves as a
way of clipping the loss, which makes gradients smaller and thus prevents the
policy’s parameters from changing too abruptly. In our experiments, we fix the
number of training episodes as a compromise between achievable returns and
evaluation runtimes. Clipping the loss has then an early-stopping effect
compared to the baseline and results in a policy less overfitted, which
benefits generalization. In Appendix G.3, we show both extended results that
ignore the fix budget requirement and the equations for the best evolved
algorithms in the remaining environments. Appendix G.1 and G.2 present the
equations for the other RWRL environments and the Brax environments,
respectively.

###  4.5 Discussion & Future Work

We have shown that MetaPG can discover novel RL loss functions that achieve
better training stability and zero-shot generalization compared to warm-start
algorithms such as SAC. Our search space consists only of primitive operators
similar to Co-Reyes et al. (2021). While this promotes expressiveness of
algorithms in the space, it requires many nodes and edges to represent a loss
function which makes our search space vast and good loss functions are
extremely sparse in this space. It is challenging for evolutionary algorithms
to traverse this space given limited search budget. One area for improvement
is to design a more efficient search space so that we have a greater chance of
discovering better algorithms under the same computation budget.

Another direction of future work is to improve the transferrability of evolved
algorithms to new domains. We have observed that, while evolved algorithms
transfer reasonably well (especially best performers, see Appendix G.6),
sometimes they do not perform better than SAC in the new environments without
hyperparameter tuning first. At the same time, it poses an interesting
research question of determining whether MetaPG is better suited to find
“super algorithms” for specific environments or a new generation of all-
purpose algorithms.

Finally, it would be interesting to ensemble the evolved loss functions on the
Pareto front. Such loss function may give additional flexibility for
practitioners when designing an RL system by encoding complex design choices
into an interpolation across objectives. We hope the released dataset of
evolved algorithms help the community gain deeper insights on differences
between loss functions.

##  5 Conclusion

We presented MetaPG, a method that evolves actor-critic RL loss functions to
optimize multiple RL objectives simultaneously and applied it to discovering
algorithms that perform well, achieve zero-shot generalization across
different environment configurations, and are stable; a triad of objectives
with real-world implications. The experiments in RWRL Cartpole, RWRL Walker,
and Gym Pendulum demonstrated that MetaPG discovered algorithms that, when
using one environment during evolution and then meta-validating in that same
environment, outperform SAC, achieving a 4% and 20% improvement in stability-
adjusted performance and stability-adjusted generalizability, respectively,
and a reduction of up to 67% in instability. Experiments on Brax Ant and Brax
Humanoid proved evolution is successful in more complex environments,
achieving a 15% and 10% increase in stability-adjusted performance and
stability-adjusted generalizability, respectively. We also observed that, when
transferring evolved algorithms to environments different from those used
during evolution, the loss of performance and generalizability in the new
environment is minimal and is comparable to SAC. Finally, we have analyzed the
evolved loss functions and linked specific elements in their structure to
fitness results, such as the removal of the entropy term to benefit
performance.

#### Author Contributions

Hidden for the double-blind review process.

#### Acknowledgments

Hidden for the double-blind review process.

## References

  * Akkaya et al. (2019) Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al.  Solving rubik’s cube with a robot hand.  _arXiv preprint arXiv:1910.07113_ , 2019.
  * Alet et al. (2019) Ferran Alet, Martin F Schneider, Tomas Lozano-Perez, and Leslie Pack Kaelbling.  Meta-learning curiosity algorithms.  In _International Conference on Learning Representations_ , 2019.
  * Andrychowicz et al. (2020) Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al.  What matters in on-policy reinforcement learning? a large-scale empirical study.  _arXiv preprint arXiv:2006.05990_ , 2020.
  * Azar et al. (2021) Ahmad Taher Azar, Anis Koubaa, Nada Ali Mohamed, Habiba A Ibrahim, Zahra Fathy Ibrahim, Muhammad Kazim, Adel Ammar, Bilel Benjdira, Alaa M Khamis, Ibrahim A Hameed, et al.  Drone deep reinforcement learning: A review.  _Electronics_ , 10(9):999, 2021.
  * Bao et al. (2021) Yaoyao Bao, Yuanming Zhu, and Feng Qian.  A deep reinforcement learning approach to improve the learning performance in process control.  _Industrial & Engineering Chemistry Research_, 60(15):5504–5515, 2021.
  * Bechtle et al. (2021) Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier.  Meta Learning via Learned Loss.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , pp. 4161–4168. IEEE, jan 2021.
  * Chen (2020) Jerry Zikun Chen.  Reinforcement learning generalization with surprise minimization.  _arXiv preprint arXiv:2004.12399_ , 2020.
  * Chiang et al. (2019) Hao-Tien Lewis Chiang, Aleksandra Faust, Marek Fiser, and Anthony Francis.  Learning navigation behaviors end-to-end with autorl.  _IEEE Robotics and Automation Letters_ , 4(2):2007–2014, April 2019.
  * Co-Reyes et al. (2021) John D Co-Reyes, Yingjie Miao, Daiyi Peng, Esteban Real, Sergey Levine, Quoc V Le, Honglak Lee, and Aleksandra Faust.  Evolving reinforcement learning algorithms.  In _International Conference on Learning Representations_ , 2021.
  * Cobbe et al. (2019) Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman.  Quantifying generalization in reinforcement learning.  In _International Conference on Machine Learning_ , pp. 1282–1289. PMLR, 2019.
  * Cobbe et al. (2021) Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman.  Phasic policy gradient.  In _International Conference on Machine Learning_ , pp. 2020–2027. PMLR, 2021.
  * Deb et al. (2002) Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan.  A fast and elitist multiobjective genetic algorithm: Nsga-ii.  _IEEE transactions on evolutionary computation_ , 6(2):182–197, 2002.
  * Dennis et al. (2020) Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M. Bayen, Stuart Russell, Andrew Critch, and Sergey Levine.  Emergent complexity and zero-shot transfer via unsupervised environment design.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.
  * Dulac-Arnold et al. (2021) Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester.  Challenges of real-world reinforcement learning: definitions, benchmarks and analysis.  _Machine Learning_ , pp. 1–50, 2021.
  * Faust et al. (2019) Aleksandra Faust, Anthony Francis, and Dar Mehta.  Evolving rewards to automate reinforcement learning.  In _6th ICML Workshop on Automated Machine Learning_ , 2019.
  * Ferreira et al. (2021) Fabio Ferreira, Thomas Nierhoff, and Frank Hutter.  Learning synthetic environments for reinforcement learning with evolution strategies.  _arXiv preprint arXiv:2101.09721_ , 2021.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International conference on machine learning_ , pp. 1126–1135. PMLR, 2017.
  * Florensa et al. (2018) Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel.  Automatic goal generation for reinforcement learning agents.  In Jennifer Dy and Andreas Krause (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 1515–1528. PMLR, 10–15 Jul 2018.
  * Fox et al. (2015) Roy Fox, Ari Pakman, and Naftali Tishby.  Taming the noise in reinforcement learning via soft updates.  _arXiv preprint arXiv:1512.08562_ , 2015.
  * Freeman et al. (2021) C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem.  Brax - a differentiable physics engine for large scale rigid body simulation, 2021.  URL http://github.com/google/brax.
  * Gaier & Ha (2019) Adam Gaier and David Ha.  Weight agnostic neural networks.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Garnier et al. (2021) Paul Garnier, Jonathan Viquerat, Jean Rabault, Aurélien Larcher, Alexander Kuhnle, and Elie Hachem.  A review on deep reinforcement learning for fluid mechanics.  _Computers & Fluids_, 225:104973, 2021.
  * Gur et al. (2021) Izzeddin Gur, Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and Aleksandra Faust.  Environment generation for zero-shot compositional reinforcement learning.  In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , pp. 4157–4169, 2021.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International conference on machine learning_ , pp. 1861–1870. PMLR, 2018.
  * He et al. (2022) Tairan He, Yuge Zhang, Kan Ren, Minghuan Liu, Che Wang, Weinan Zhang, Yuqing Yang, and Dongsheng Li.  Reinforcement learning with automated auxiliary loss search.  _arXiv preprint arXiv:2210.06041_ , 2022.
  * Henderson et al. (2018) Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger.  Deep reinforcement learning that matters.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018.
  * Hertel et al. (2020) Lars Hertel, Pierre Baldi, and Daniel L Gillen.  Quantity vs. quality: On hyperparameter optimization for deep reinforcement learning.  _arXiv preprint arXiv:2007.14604_ , 2020.
  * Hessel et al. (2019) Matteo Hessel, Hado van Hasselt, Joseph Modayil, and David Silver.  On inductive biases in deep reinforcement learning.  _arXiv preprint arXiv:1907.02908_ , 2019.
  * Hoffman et al. (2020) Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas.  Acme: A research framework for distributed reinforcement learning.  _arXiv preprint arXiv:2006.00979_ , 2020.
  * Houthooft et al. (2018) Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAI Jonathan Ho, and Pieter Abbeel.  Evolved policy gradients.  In _Advances in Neural Information Processing Systems_ , pp. 5400–5409, 2018.
  * Hutter et al. (2019) Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren.  _Automated machine learning: methods, systems, challenges_.  Springer Nature, 2019.
  * Ibarz et al. (2021) Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine.  How to train your robot with deep reinforcement learning: lessons we have learned.  _The International Journal of Robotics Research_ , 40(4-5):698–721, 2021.
  * Igl et al. (2019) Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann.  Generalization in reinforcement learning with selective noise injection and information bottleneck.  _Advances in neural information processing systems_ , 32, 2019.
  * Jiang et al. (2021) Ray Jiang, Tom Zahavy, Zhongwen Xu, Adam White, Matteo Hessel, Charles Blundell, and Hado Van Hasselt.  Emphatic algorithms for deep reinforcement learning.  In _International Conference on Machine Learning_ , pp. 5023–5033. PMLR, 2021.
  * Jozefowicz et al. (2015) Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever.  An empirical exploration of recurrent network architectures.  In _International conference on machine learning_ , pp. 2342–2350. PMLR, 2015.
  * Kirk et al. (2022) Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel.  A survey of generalisation in deep reinforcement learning.  _arXiv preprint arXiv:2111.09794_ , 2022.
  * Kirsch et al. (2020) Louis Kirsch, Sjoerd van Steenkiste, and Juergen Schmidhuber.  Improving generalization in meta reinforcement learning using learned objectives.  In _International Conference on Learning Representations_ , 2020.
  * Koza (1994) John R Koza.  Genetic programming as a means for programming computers by natural selection.  _Statistics and computing_ , 4(2):87–112, 1994\.
  * Linke et al. (2020) Cam Linke, Nadia M Ady, Martha White, Thomas Degris, and Adam White.  Adapting behavior via intrinsic reward: A survey and empirical study.  _Journal of Artificial Intelligence Research_ , 69:1287–1332, 2020.
  * Lu et al. (2022) Chris Lu, Jakub Grudzien Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster.  Discovered policy optimisation.  _arXiv preprint arXiv:2210.05639_ , 2022.
  * Miao et al. (2021) Yingjie Miao, Xingyou Song, Daiyi Peng, Summer Yue, John D Co-Reyes, Eugene Brevdo, and Aleksandra Faust.  Rl-darts: differentiable architecture search for reinforcement learning.  2021\.
  * Miller et al. (1989) Geoffrey F Miller, Peter M Todd, and Shailesh U Hegde.  Designing neural networks using genetic algorithms.  In _ICGA_ , volume 89, pp. 379–384, 1989.
  * Oh et al. (2020) Junhyuk Oh, Matteo Hessel, Wojciech M Czarnecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver.  Discovering reinforcement learning algorithms.  _Advances in Neural Information Processing Systems_ , 33:1060–1070, 2020.
  * Parker-Holder et al. (2022) Jack Parker-Holder, Raghu Rajan, Xingyou Song, André Biedenkapp, Yingjie Miao, Theresa Eimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, Frank Hutter, and Marius Lindauer.  Automated reinforcement learning (autorl): A survey and open problems.  _Journal of Artificial Intelligence Research_ , 74:517–568, 2022.
  * Peng et al. (2018) Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel.  Sim-to-real transfer of robotic control with dynamics randomization.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pp. 3803–3810. IEEE, 2018.
  * Perera & Kamalaruban (2021) A.T.D. Perera and Parameswaran Kamalaruban.  Applications of reinforcement learning in energy systems.  _Renewable and Sustainable Energy Reviews_ , 137:110618, 2021.
  * Raileanu & Fergus (2021) Roberta Raileanu and Rob Fergus.  Decoupling value and policy for generalization in reinforcement learning.  In _International Conference on Machine Learning_ , pp. 8787–8798. PMLR, 2021.
  * Real et al. (2019) Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le.  Regularized evolution for image classifier architecture search.  In _Proceedings of the aaai conference on artificial intelligence_ , volume 33, pp. 4780–4789, 2019.
  * Real et al. (2020) Esteban Real, Chen Liang, David So, and Quoc Le.  Automl-zero: Evolving machine learning algorithms from scratch.  In _International Conference on Machine Learning_ , pp. 8007–8019. PMLR, 2020.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Stanley & Miikkulainen (2002) Kenneth O Stanley and Risto Miikkulainen.  Evolving neural networks through augmenting topologies.  _Evolutionary computation_ , 10(2):99–127, 2002\.
  * Stanley et al. (2009) Kenneth O Stanley, David B D’Ambrosio, and Jason Gauci.  A hypercube-based encoding for evolving large-scale neural networks.  _Artificial life_ , 15(2):185–212, 2009.
  * Sutton & Barto (2018) Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Tobin et al. (2017) Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel.  Domain randomization for transferring deep neural networks from simulation to the real world.  In _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_ , pp. 23–30. IEEE, 2017.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Vlastelica et al. (2021) Marin Vlastelica, Michal Rolínek, and Georg Martius.  Neuro-algorithmic policies enable fast combinatorial generalization.  _arXiv preprint arXiv:2102.07456_ , 2021.
  * Volz et al. (2018) Vanessa Volz, Jacob Schrum, Jialin Liu, Simon M Lucas, Adam Smith, and Sebastian Risi.  Evolving mario levels in the latent space of a deep convolutional generative adversarial network.  In _Proceedings of the genetic and evolutionary computation conference_ , pp. 221–228, 2018.
  * Xu et al. (2022) Mengdi Xu, Zuxin Liu, Peide Huang, Wenhao Ding, Zhepeng Cen, Bo Li, and Ding Zhao.  Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability.  _arXiv preprint arXiv:2209.08025_ , 2022.
  * Xu et al. (2018) Zhongwen Xu, Hado P van Hasselt, and David Silver.  Meta-gradient reinforcement learning.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.
  * Xu et al. (2020) Zhongwen Xu, Hado P van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver.  Meta-gradient reinforcement learning with an objective discovered online.  _Advances in Neural Information Processing Systems_ , 33:15254–15264, 2020.
  * Zhang et al. (2021) Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, and Roberto Calandra.  On the importance of hyperparameter optimization for model-based reinforcement learning.  In Arindam Banerjee and Kenji Fukumizu (eds.), _Proceedings of The 24th International Conference on Artificial Intelligence and Statistics_ , volume 130 of _Proceedings of Machine Learning Research_ , pp. 4015–4023. PMLR, 13–15 Apr 2021.
  * Zhao et al. (2019) Chenyang Zhao, Olivier Sigaud, Freek Stulp, and Timothy M Hospedales.  Investigating generalisation in continuous deep reinforcement learning.  _arXiv preprint arXiv:1902.07015_ , 2019.
  * Zoph et al. (2018) Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le.  Learning transferable architectures for scalable image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 8697–8710, 2018.

##  Appendix A Search space details

In this section we present the details of the search space; we divide the
nodes into input, output, and operation nodes. Output nodes correspond to
losses computed by the algorithm, whose gradient with respect to the algorithm
inputs is then computed in a training loop. In this paper we do steepest
gradient descent to update network parameters; given the search space
complexity, we leave incorporating other gradient descent strategies into the
search space out of its scope. However, other strategies such as natural
gradient or conjugate gradient could be incorporated, as they do gradient
transformations and are agnostic to loss functions.

During the training process using an evolved algorithm, agents then learn the
policy by means of experience tuples coming from a replay buffer. MetaPG
admits both continuous and discrete action spaces; specific nodes in the
graphs —e.g., the neural networks— are adapted to work with the corresponding
space.

During the evolution process, we fix a maximum number of nodes (60 and 80 in
our experiments), which consists of the aforementioned input and output nodes,
and several operation nodes. The majority of operation nodes treat input
elements as tensors with variable shapes in order to maximize graph
flexibility. Each node possesses a certain number of input and output edges,
which are determined by the specific operation this node carries out. For
example, a node that takes in two tensors and multiplies them element-wise has
two input edges and a single output edge.

###  A.1 List of nodes

A complete list of the nodes considered follows:

##### Input nodes

We only encode canonical RL elements as inputs:

  * •

Policy network $\pi$

  * •

Two critic networks, $Q_{1}$ and $Q_{2}$, and two target critic networks,
$Q_{targ_{1}}$ and $Q_{targ_{2}}$

  * •

Batch of states $s_{t}$ and next states $s_{t+1}$

  * •

Batch of actions $a_{t}$

  * •

Batch of rewards $r_{t}$

  * •

Discount factor $\gamma$

##### Output nodes

The output of these nodes is used as loss function to compute gradient descent
on:

  * •

Policy loss $L_{\pi}$

  * •

Critic loss $L_{Q_{i}}$

##### Operation nodes

These nodes operate generally on tensors and can broadcast operations when
input sizes do not match:

  * •

Addition: add two, three, or four tensors

  * •

Multiplication: compute element-wise product of two or three tensors

  * •

Subtract two tensors

  * •

Divide two tensors and add constant $\epsilon$ to the denominator

  * •

Neural network operations: Action distribtuion from state, stopping gradient
computation

  * •

Operations with action distributions: Sample, Log-probability

  * •

Mean, sum, and standard deviation over last axis of array or over entire array

  * •

Cumulative sum, cumulative sum with discount

  * •

Squared difference

  * •

Multiply by a constant: -1, 0.1, 0.01, 0.5, 2.0

  * •

Minimum and maximum over last axis of a tensor

  * •

Minimum and maximum element-wise between two tensors

  * •

Other general operations: clamp, absolute value, square, logarithm,
exponential

  * •

Trigonometry functions

###  A.2 Size of the search space

To get an upper bound of the size of the search space in terms of the number
of possible graphs (not all of them valid), we consider the ($k$+1)-th node in
the graph of size $K$ nodes. This node can correspond to one of the $N$
different operation nodes. Assuming that this node has two inputs, there are
$\binom{k}{2}$ possibilities of connecting to previous nodes in the graph.
Therefore, we have a total of $\frac{1}{2}Nk(k-1)$ possible combinations for
the ($k$+1)-th node. Then, an upper bound of the total number of possible
graphs is

$\left(\frac{NK(K-1)}{2}\right)^{K}$ (8)

With $N=33$ and $K=60$ we obtain approximately 10286 graphs. This number
increases to 10401 if $K=80$ instead.

##  Appendix B Additional implementation details

##### Multi-objective evolution

We do 10 independent repeats of MetaPG when evolving on RWRL Cartpole, RWRL
Walker, and Gym Pendulum; 5 repeats when evolving on Brax Ant; and 3 repeats
for Brax Humanoid.

##### Warm-starting

Algorithms are initialized using the warm-start SAC graph (see Appendix D),
which consists of 33 nodes. Additional operation nodes are added to each
individual until reaching the maximum amount of 60 nodes.

##### Mutation

During mutation, there is a 50% chance an individual undergoes node mutation
and a 50% chance it undergoes edge mutation. During node mutation, there is a
50% chance of replacing one node, a 25% chance of replacing 2 nodes, a 12.5%
chance of replacing 4 nodes, and a 6.25% chance of replacing 8 and 16 nodes,
respectively. During edge mutation, only one edge in the graph is replaced
randomly.

##### Hashing

In the hashing process we use a fixed set of synthetic inputs with a batch
size of 16.

##### Encoding multiple objectives

MetaPG keeps the population to a fixed size during evolution. To decide which
individuals should be removed in the process, the method makes use of
different fitness scores that encode each of the RL objectives considered and
are adjusted for stability. These scores are not combined but treated
separately in a multi-objective fashion. This means that, after evaluating a
graph $i$, it will have fitness scores $\\{f_{i,1},f_{i,2},...,f_{i,F}\\}$,
where $F$ is the number of objectives considered. Then, when comparing two
graphs $i$ and $j$, we say $i$ has higher fitness than $j$ iff $f_{i,k}\geq
f_{j,k},\forall k$, with at least one fitness score $k^{\prime}$ such that
$f_{i,k^{\prime}}>f_{j,k^{\prime}}$. In this case we also say graph $i$
Pareto-dominates graph $j$. If neither $i$ Pareto-dominates $j$ nor vice
versa, we say both graphs are Pareto-optimal (e.g., if
$f_{i,k^{\prime}}>f_{j,k^{\prime}}$ and
$f_{i,k^{\prime\prime}}<f_{j,k^{\prime\prime}}$, then $i$ and $j$ are Pareto-
optimal).

The process of removing individuals from the population follows the NSGA-II
algorithm (Deb et al., 2002) which, assuming a maximum population size of
$P_{max}$ individuals:

  1. 1.

From a set of $P$ individuals, with $|P|>P_{max}$, it computes the set
$P_{opt}$ of Pareto-optimal fittest graphs. None of the graphs in $P_{opt}$ is
Pareto-dominated by any other graph in the population and, if a graph $i$ in
$P$ is Pareto-dominated by at least one other graph, then $i$ does not belong
to the Pareto-optimal set.

  2. 2.

If $|P_{opt}|\geq P_{max}$, the graphs are ranked based on their crowding
distance in the fitness space. This favors individuals that are further apart
from other individuals in the fitness space. The fittest $P_{max}$ individuals
of the Pareto-optimal set $P_{opt}$ are kept in the population.

  3. 3.

Otherwise, if $|P_{opt}|<P_{max}$, the set $P_{opt}$ is kept in the population
and the process is repeated taking $P\leftarrow P$ \$P_{opt}$ and
$P_{max}\leftarrow P_{max}-|P_{opt}|$.

##  Appendix C Environment Configurations

In this work we use multiple environments for our experiments: Cartpole and
Walker from the RWRL Environment Suite (Dulac-Arnold et al., 2021), Gym
Pendulum, and Ant and Humanoid from the Brax physics simulator (Freeman et
al., 2021). In Table 2 we list the training configuration used for each and
the parameters that we use to assess the generalizability of the policies. The
parameters that are not listed are fixed to the default values for the
environment in question.

In the case of Gym Pendulum and the Brax environments, we have more than one
perturbation parameter; the generalizability score is computed by first
sweeping through the perturbation values for one, taking the average
$f_{gen_{1}}$, repeating the same process for the others to compute
$f_{gen_{2}}$, …, $f_{gen_{P}}$, and then computing the average of both to get
the final score, i.e., $f_{gen}=(f_{gen_{1}}+...+f_{gen_{P}})/P$, where $P$ is
the total number of different parameters that are perturbed.

For the parameters that undergo perturbations, we select the specific value
used in the training configuration based on the following criteria: in the
case of the RWRL Environment Suite, we pick the training value used by Dulac-
Arnold et al. (2021), which corresponds to the default value as described in
each benchmark; in the case of Gym Pendulum, we pick the default values given
by the environment as training configuration; in the case of Brax, we follow
the rationale of selecting training configurations that represent a baseline
scenario (i.e., no mass variations, normal friction, and normal torque).

Table 2: Environment parameters and perturbations. Environment parameter | Value
---|---
RWRL Cartpole
Rollout length | 1,000
Min. return | 0
Max. return | 1,000
Training episodes | 150
Perturbation parameter (PP) | Pole length
PP Default value | 1.0
PP Generalizability values | 0.1 to 3.0 in steps of 0.1
RWRL Walker
Rollout length | 1,000
Min. return | 0
Max. return | 1,000
Training episodes | 225
Perturbation parameter (PP) | Thigh length
PP Default value | 0.225
PP Generalizability values | .1, .125, .15, .175, .2, .225, .25, .3, .35, .4, .45, .5, .55, .6, .7
Gym Pendulum
Rollout length | 2,000
Min. return | -2,000
Max. return | 0
Training episodes | 100
Perturbation parameter 1 (PP1) | Pendulum mass
PP1 Default value | 1.0
PP1 Generalizability values | .1, .2, .4, .5, .75, 1.0, 1.5, 2.0, 3.0, 5.0, 7.5, 10.0
Perturbation parameter 2 (PP2) | Pendulum length
PP2 Default value | 1.0
PP2 Generalizability values | .1, .2, .4, .5, .75, 1.0, 1.5, 2.0, 3.0, 5.0, 7.5, 10.0
Brax environments
Rollout length | 1,000
Min. return | 0
Max. return | 10,000 (Ant) and 14,000 (Humanoid)
Training episodes | 1,000
Perturbation parameter 1 (PP1) | Mass coefficient
PP1 Default value | 1.0
PP1 Generalizability values | 0.8 to 1.2 in steps of 0.05
Perturbation parameter 2 (PP2) | Friction coefficient
PP2 Default value | 1.0
PP2 Generalizability values | 0.3 to 1.0 in steps of 0.05
Perturbation parameter 3 (PP3) | Torque multiplier
PP3 Default value | 1.0
PP3 Generalizability values | 0.5 to 1.0 in steps of 0.05
Perturbation parameter 4 (PP4) | Combined parameters PP1, PP2, and PP3
PP4 Default value | 1.0 for each
PP4 Generalizability values | Grid search over individual generalizability values

##  Appendix D Warm-start SAC

We present the version of Soft Actor-Critic (SAC) (Haarnoja et al., 2018) used
in this work as the warm-start algorithm to initialize the population. We
first present the equations for the policy loss $L_{\pi}^{WS}$ and critic loss
$L_{Q_{i}}^{WS}$:

$\displaystyle L_{\pi}^{WS}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t})\sim\mathcal{D}}\left[\log\pi(\tilde{a}_{t}|s_{t})-\underset{i}{\min}\,Q_{i}(s_{t},\tilde{a}_{t})\right]$
(9) $\displaystyle L_{Q_{i}}^{WS}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[\left(r_{t}+\gamma\left(\underset{i}{\min}\,Q_{targ_{i}}(s_{t+1},\tilde{a}_{t+1})-\log\pi(\tilde{a}_{t+1}|s_{t+1})\right)-Q_{i}(s_{t},a_{t})\right)^{2}\right]$
(10)

where $\tilde{a}_{t}\sim\pi(\cdot|s_{t})$,
$\tilde{a}_{t+1}\sim\pi(\cdot|s_{t+1})$, and $\mathcal{D}$ is a dataset from
the replay buffer. Then, in Figure 8 we represent these two equations that
define the SAC algorithm in the form of a graph with typed input and outputs.
MetaPG then modifies this graphs following the procedure described in Section
3.

Figure 8: Soft Actor-Critic (SAC) algorithm represented as a graph to
initialize the population as a warm-start algorithm.

##  Appendix E Additional RL training details

An individual encoding a RL algorithm in the form of a graph is evaluated by
training an agent using such algorithm. We use an implementation based on an
ACME agent (Hoffman et al., 2020) for the RWRL and Gym environments, and an
implementation based on the Brax physics simulator (Freeman et al., 2021) for
the Brax environments. The configuration of the training setup are shown in
Table 3 for RWRL and Gym environments, and in Table 4 for the Brax
environments.

Table 3: RL Training setup for the RWRL and Gym environments. Parameter | Value
---|---
Discount factor $\gamma$ | 0.99
Batch size | 64 (RWRL Cartpole and Gym Pendulum)
| 128 (RWRL Walker)
Learning rate | $3\cdot 10^{-4}$
Target smoothing coeff. $\tau$ | 0.005
Replay buffer size | 1,000,000
Min. num. samples in the buffer | 10,000
Gradient updates per learning step | 1
n step | 1
Reward scale | 5.0
Actor network | MLP (256, 256)
Actor activation function | ReLU
Tanh on output of actor network | Yes
Critic networks | MLP (256, 256)
Critic activation function | ReLU
Table 4: RL Training setup for the Brax environments. Parameter | Value
---|---
Discount factor $\gamma$ | 0.95
Batch size | 128
Learning rate | $6\cdot 10^{-4}$
Target smoothing coeff. $\tau$ | 0.005
Replay buffer size | 1,000,000
Min. num. samples in the buffer | 1,000
Gradient updates per learning step | 64
Reward scale | 10.0
Number of parallel environments | 128
Actor network | MLP (256, 256)
Actor activation function | ReLU
Tanh on output of actor network | Yes
Critic networks | MLP (256, 256)
Critic activation function | ReLU

##  Appendix F Additional evolution results

In this section we present the additional evolution results of the paper. We
first introduce the remaining figures for RWRL Cartpole, then outline
evolution results for RWRL Walker and Gym Pendulum, then show how different
algorithms in the population for all three environments compare in terms of
stability, and finally provide results for the Brax environments.

###  F.1 Evolution results for RWRL Cartpole

Figure 9 shows the resulting population when running evolution using the RWRL
Cartpole environment (Dulac-Arnold et al., 2021) and Table 5 shows the raw and
stability-adjusted fitness scores for each algorithm in the Pareto Front.
Figure 10 shows the performance of the evolved algorithms across different
environment configurations, it updates Figure 5(b) by introducing PPO Schulman
et al. (2017) in the comparison. We found that PPO was not well-suited for the
continuous control tasks explored in this work.

Figure 9: Meta-training and meta-validation stability-adjusted fitness scores (computed using Equation 3 across 10 seeds) for each RL algorithm in the population alongside the warm-start (SAC) and ACME SAC when using the RWRL Cartpole environment for training. We show the meta-validated Pareto Front of algorithms that results after merging the 10 populations corresponding to the 10 repeats of the experiment. Figure 10: Average and standard deviation across seeds of the meta-validation performance when training on a single configuration of RWRL Cartpole and evaluating on multiple unseen ones. We compare the best performer, the best generalizer, the warm start SAC, and we also add two hyperparameter-tuned PPO runs, one tuned for performance and the other for generalizability (see Appendix G.4 for hyperparameter tuning details). The pole length changes across environment configurations and a length of 1.0 is used as training configuration. Table 5: Similar to Table 1, we compare all algorithms in the Pareto Front with SAC (warm-start and hyperparameter-tuned ACME SAC) using metrics obtained in the RWRL Cartpole environment: average performance and generalizability, stability-adjusted performance and generalizability scores, and measure of instability (standard deviation $\sigma$ divided by the warm-start’s $\sigma_{WS}$. We compute these metrics across 10 seeds and the best result in a _column_ is bolded. †In contrast to performance and generalizability, the lower the instability the better.  | Performance | Generalizability |  Instability† ($\sigma$/$\sigma_{WS}$)
---|---|---|---
RL Algorithm | $f_{perf}$ | $\tilde{f}_{perf}$ | $f_{gen}$ | $\tilde{f}_{gen}$ | Perf. | Gen.
Pareto 1: Best performer | 0.871 | 0.868 | 0.475 | 0.459 | 0.33 | 0.59
Pareto 2 | 0.857 | 0.856 | 0.513 | 0.488 | 0.11 | 0.93
Pareto 3 | 0.857 | 0.855 | 0.514 | 0.489 | 0.22 | 0.93
Pareto 4 | 0.856 | 0.854 | 0.517 | 0.493 | 0.22 | 0.89
Pareto 5 | 0.855 | 0.853 | 0.520 | 0.497 | 0.22 | 0.85
Pareto 6 | 0.854 | 0.852 | 0.531 | 0.514 | 0.22 | 0.63
Pareto 7 | 0.798 | 0.788 | 0.540 | 0.524 | 1.11 | 0.59
Pareto 8 | 0.794 | 0.784 | 0.546 | 0.528 | 1.11 | 0.67
Pareto 9 | 0.783 | 0.776 | 0.569 | 0.543 | 0.78 | 0.96
Pareto 10: Best generalizer | 0.770 | 0.756 | 0.570 | 0.551 | 1.55 | 0.70
Warm-start SAC | 0.845 | 0.836 | 0.487 | 0.460 | 1.0 | 1.0
ACME SAC HPT Perf | 0.865 | 0.864 | 0.372 | 0.312 | 0.11 | 2.22
ACME SAC HPT Gen | 0.845 | 0.833 | 0.518 | 0.478 | 1.33 | 1.48

###  F.2 Evolution results for RWRL Walker

We present evolution results when running MetaPG with RWRL Walker as the
training environment. In Figures 11 and 12 we show the resulting population
and the performance across environment configurations for the best performer
and the best generalizer in the Pareto Front, respectively. Exact numbers for
each algorithm in the Pareto Front can be found in Table 6. This table also
shows the scores of the warm-start and ACME SAC. As covered in Appendix G.4,
we do not hyperparameter-tune the warm-start before the experiments. As a
result, the warm-start might perform poorly, as is the case in this
environment. We can observe MetaPG is able to increase the performance and
generalizability of the evolved algorithms during the evolution process.

Figure 11: Meta-training and meta-validation stability-adjusted fitness scores (computed using Equation 3 across 10 seeds) for each RL algorithm in the population alongside the warm-start (SAC) and ACME SAC when using the RWRL Walker environment for training. We show the meta-validated Pareto-optimal set of algorithms that results after merging the 10 populations corresponding to the 10 repeats of the experiment. Figure 12: Average and standard deviation across seeds of the meta-validation performance of the best performer, the best generalizer, and the warm-start (SAC) when training on a single configuration of RWRL Walker and evaluating on multiple unseen ones. The thigh length changes across environment configurations and a length of 0.225 is used as training configuration. Table 6: Similar to Table 1, we compare all algorithms in the Pareto Front with SAC (warm-start and hyperparameter-tuned ACME SAC) using metrics obtained in the RWRL Walker environment: average performance and generalizability, stability-adjusted performance and generalizability scores, and measure of instability (standard deviation $\sigma$ divided by the warm-start’s $\sigma_{WS}$. We compute these metrics across 10 seeds and the best result in a _column_ is bolded. †In contrast to performance and generalizability, the lower the instability the better.  | Performance | Generalizability |  Instability† ($\sigma$/$\sigma_{WS}$)
---|---|---|---
RL Algorithm | $f_{perf}$ | $\tilde{f}_{perf}$ | $f_{gen}$ | $\tilde{f}_{gen}$ | Perf. | Gen.
Pareto 1: Best performer | 0.963 | 0.961 | 0.544 | 0.526 | 1.0 | 18.0
Pareto 2 | 0.962 | 0.959 | 0.536 | 0.524 | 1.0 | 12.0
Pareto 3 | 0.960 | 0.958 | 0.542 | 0.527 | 1.5 | 15.0
Pareto 4 | 0.959 | 0.956 | 0.541 | 0.528 | 1.0 | 13.0
Pareto 5 | 0.960 | 0.955 | 0.541 | 0.532 | 1.5 | 9.0
Pareto 6 | 0.954 | 0.951 | 0.555 | 0.546 | 2.5 | 9.0
Pareto 7: Best generalizer | 0.955 | 0.950 | 0.569 | 0.554 | 2.5 | 15.0
Warm-start SAC | 0.028 | 0.026 | 0.033 | 0.032 | 1.0 | 1.0
ACME SAC HPT Perf | 0.968 | 0.965 | 0.444 | 0.430 | 1.5 | 14.0
ACME SAC HPT Gen | 0.926 | 0.918 | 0.510 | 0.498 | 4.0 | 12.0

###  F.3 Evolution results for Gym Pendulum

We present evolution results when running MetaPG with Gym Pendulum as the
training environment. In Figures 13 and 14 we show the resulting population
and the performance across environment configurations for the best performer
and the best generalizer in the Pareto Front, respectively. In the case of
Pendulum, the generalizability fitness score is computed across the
perturbation of two different parameters: the pendulum mass and the pendulum
length. These parameters are changed separately, as opposed to varying both
the mass and length of the pendulum in the same run. Exact numbers can be
found in Table 7, in which an average improvement over the warm-start of 2% in
performance and 15% in generalizability is achieved.

Figure 13: Meta-training and meta-validation stability-adjusted fitness scores (computed using Equation 3 across 10 seeds) for each RL algorithm in the population alongside the warm-start (SAC) and ACME SAC when using the Gym Pendulum environment for training. We show the meta-validated Pareto-optimal set of algorithms that results after merging the 10 populations corresponding to the 10 repeats of the experiment. Figure 14: Average and standard deviation across seeds of the meta-validation performance of the best performer, the best generalizer, and the warm-start (SAC) when training on a single configuration of Gym Pendulum and evaluating on multiple unseen ones. The pendulum mass and the pendulum length independently change across environment configurations (we change one at a time). The training configurations use a pendulum mass and a pendulum length of 1.0 and 1.0, respectively. Table 7: Similar to Table 1, we compare all algorithms in the Pareto Front with SAC (warm-start and hyperparameter-tuned ACME SAC) using metrics obtained in the Gym Pendulum environment: average performance and generalizability, stability-adjusted performance and generalizability scores, and measure of instability (standard deviation $\sigma$ divided by the warm-start’s $\sigma_{WS}$. We compute these metrics across 10 seeds and the best result in a _column_ is bolded. †In contrast to performance and generalizability, the lower the instability the better.  | Performance | Generalizability |  Instability† ($\sigma$/$\sigma_{WS}$)
---|---|---|---
RL Algorithm | $f_{perf}$ | $\tilde{f}_{perf}$ | $f_{gen}$ | $\tilde{f}_{gen}$ | Perf. | Gen.
Pareto 1: Best performer | 0.887 | 0.877 | 0.360 | 0.349 | 0.45 | 0.73
Pareto 2 | 0.885 | 0.876 | 0.381 | 0.364 | 0.41 | 1.13
Pareto 3 | 0.887 | 0.877 | 0.391 | 0.377 | 0.45 | 0.93
Pareto 4 | 0.887 | 0.876 | 0.392 | 0.379 | 0.45 | 0.87
Pareto 5 | 0.887 | 0.876 | 0.393 | 0.386 | 0.41 | 0.47
Pareto 6 | 0.886 | 0.876 | 0.433 | 0.415 | 0.45 | 1.20
Pareto 7 | 0.886 | 0.875 | 0.437 | 0.418 | 0.45 | 1.27
Pareto 8: Best generalizer | 0.868 | 0.834 | 0.445 | 0.424 | 1.55 | 1.40
Warm-start SAC | 0.879 | 0.857 | 0.383 | 0.368 | 1.0 | 1.0
ACME SAC HPT Perf | 0.880 | 0.866 | 0.392 | 0.380 | 0.64 | 0.80
ACME SAC HPT Gen | 0.879 | 0.865 | 0.400 | 0.391 | 0.64 | 0.60

###  F.4 Stability analyses for RWRL Cartpole, RWRL Walker, and Gym Pendulum

We present additional figures focused on stability, which, as introduced in
Section 3.2, is accounted for by penalizing the standard deviation across
seeds, following Equation 3. In this section, for each environment considered
in this work, we select a subset of the meta-validated graphs that covers all
the explored fitness space and, in Figure 15, show the average and standard
deviation of each fitness score. Algorithms in the Pareto Front and those
closer to it present lower variability, emphasizing that MetaPG is successful
in improving the stability of RL algorithms.

(a) RWRL Cartpole

(b) RWRL Walker

(c) Gym Pendulum

Figure 15: From a subset of the meta-validated graphs, for each of them, we
show the average fitness scores surrounded by an ellipse with semiaxes
representing the standard deviation across seeds for each fitness score.

###  F.5 Evolution results for Brax Ant

Table 8 shows the meta-testing fitness scores of the best performer algorithms
obtained after running evolution on Brax Ant and Brax Humanoid, respectively.
We compare those against the hyperparameter-tuned warm-start SAC and observe
an improvement of 15% and 10% in stability-adjusted performance and
generalizability, respectively, and up to a 23% reduction in instability. We
observe that, after cross-domain transfer, the algorithm evolved in Brax
Humanoid achieves comparable scores to SAC. We follow the hyperparameter
tuning procedure explained in Appendix G.5.

Table 8: Meta-tested average and stability-adjusted performance and generalizability scores, and measure of instability (standard deviation $\sigma$ divided by the warm-start’s $\sigma_{WS}$) for algorithms first evolved in Brax Ant and Brax Humanoid, and then evaluated on Brax Ant on a different set of seeds. We compare these scores against the hyperparameter-tuned warm start SAC. We compute these metrics across 4 seeds and the best result in a _column_ is bolded. | Performance | Generalizability |  Instability† ($\sigma$/$\sigma_{WS}$)
---|---|---|---
RL Algorithm | $f_{perf}$ | $\tilde{f}_{perf}$ | $f_{gen}$ | $\tilde{f}_{gen}$ | Perf. | Gen.
Ant performer | 0.770 | 0.729 | 0.627 | 0.592 | 0.77 | 0.97
Humanoid performer | 0.643 | 0.526 | 0.553 | 0.467 | 2.21 | 2.39
Warm-start SAC (HP tuned) | 0.685 | 0.632 | 0.573 | 0.537 | 1.0 | 1.0

###  F.6 Evolution results for Brax Humanoid

Figure 16 shows the behaviour of evolved algorithms when meta-tested in Brax
Humanoid, using an evaluation consisting of perturbations encountered in many
practical settings (changes in friction coefficient, mass, and torque), as
outlined in Appendix C. We first evolve algorithms independently in both
Humanoid and Ant, then, for each case, select the algorithm with the best
stability-adjusted performance $\tilde{f}_{perf}$ (in this case we focus on
only one of the evolved algorithms since there is strong correlation between
both metrics; the best performer and best generalizer are close, sometimes
even encode the same algorithm), then meta-validate it with different
hyperparameter sets (see Appendix G.5), and select the hyperparameter set that
leads to the best stability-adjusted generalizability. We then fix the
hyperparameters and re-evaluate using the meta-testing seeds. We compare
algorithms evolved in Ant and Humanoid with hyperparameter-tuned SAC.

Figure 16: Meta-testing in Brax Humanoid. We compare, after hyperparameter
tuning, loss functions evolved in Brax Humanoid, Brax Ant (to assess cross-
domain transfer), and the SAC baseline used as warm-start. We plot the average
return and standard deviation across random seeds when evaluating in multiple
Brax Humanoid instances with different friction coefficients, mass
coefficients, and torque multipliers (in all cases 1.0 is used as training
configuration).

The numerical results of this analysis are found in Table 9. We observe that,
while the evolved algorithm achieves better stability-adjusted
generalizability and a clear reduction in instability (up to 40%), the evolved
algorithm performs worse than SAC when meta-testing. In this case the
algorithms are hyperparameter-tuned for generalizability, hence the higher
score. Note we evolved less graphs in the specific case of Humanoid (50K
compared to 200K evolved graphs for Brax Ant), as training a policy in
Humanoid is more costly. We expect these results to improve if more algorithms
are evolved in the population. Then, as in the case where we used Brax Ant as
evaluation environment, we achieve a good cross-domain fitness compared to
SAC, but the scores are lower. In this case, however, the algorithm evolved in
Brax Ant shows more stability than SAC when both are evaluated in Humanoid.

Table 9: Meta-tested average and stability-adjusted performance and generalizability scores, and measure of instability (standard deviation $\sigma$ divided by the warm-start’s $\sigma_{WS}$) for algorithms first evolved in Brax Ant and Brax Humanoid, and then evaluated on Brax Humanoid on a different set of seeds. We compare these scores against the hyperparameter-tuned warm start SAC. We compute these metrics across 4 seeds and the best result in a _column_ is bolded. | Performance | Generalizability |  Instability† ($\sigma$/$\sigma_{WS}$)
---|---|---|---
RL Algorithm | $f_{perf}$ | $\tilde{f}_{perf}$ | $f_{gen}$ | $\tilde{f}_{gen}$ | Perf. | Gen.
Ant performer | 0.440 | 0.374 | 0.420 | 0.384 | 0.63 | 0.51
Humanoid performer | 0.474 | 0.391 | 0.462 | 0.420 | 0.80 | 0.60
Warm-start SAC (HP tuned) | 0.514 | 0.410 | 0.450 | 0.380 | 1.0 | 1.0

##  Appendix G Additional analyses

In this section we present additional analyses on the evolved algorithms.
Specifically, we take a look at the equations defining some of the best
evolved algorithms and analyze the effect of using algorithms evolved in one
specific environment to train agents in different environments.

###  G.1 Best performer and best generalizer for RWRL Walker and Gym Pendulum

We present the loss equations for both the best performer and best generalizer
when using RWRL Walker and Gym Pendulum as training environments. First, the
best performer for RWRL Walker:

$\displaystyle L_{\pi}^{perf}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[r_{t}+\gamma\left(\underset{i}{\min}\,Q_{targ_{i}}(s_{t+1},\tilde{a}_{t+1})-\mathrm{atan}(\gamma/Q(s_{t},a_{t}))\right)-Q(s_{t},\tilde{a}_{t})\right]$
(11) $\displaystyle L_{Q_{i}}^{perf}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[\left(r_{t}+\gamma\left(\underset{i}{\min}\,Q_{targ_{i}}(s_{t+1},\tilde{a}_{t+1})-\mathrm{atan}(\gamma/Q_{i}(s_{t},a_{t}))\right)-Q_{i}(s_{t},a_{t})\right)^{2}\right]$
(12)

In all cases, $\tilde{a}_{t}\sim\pi(\cdot|s_{t})$,
$\tilde{a}_{t+1}\sim\pi(\cdot|s_{t+1})$, and $\mathcal{D}$ is a dataset of
experience tuples from the replay buffer. Next, the best generalizer for RWRL
Walker:

$\displaystyle L_{\pi}^{gen}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},s_{t+1})\sim\mathcal{D}}\left[\frac{0.2\cdot\log\pi(\tilde{a}_{t+1}|s_{t+1})}{Q_{i}(s_{t+1},\tilde{a}_{t+1})-0.1\cdot\log\pi(\tilde{a}_{t+1}|s_{t+1})}-\underset{i}{\min}\,Q_{i}(s_{t},\tilde{a}_{t+1})\right]$
(13) $\displaystyle L_{Q_{i}}^{gen}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[\left(r_{t}+\gamma\left(Q_{i}(s_{t+1},\tilde{a}_{t+1})-0.1\cdot\log\pi(\tilde{a}_{t+1}|s_{t+1})\right)-Q_{i}(s_{t},a_{t})\right)^{2}\right]$
(14)

Now we present the best performer for Gym Pendulum:

$\displaystyle L_{\pi}^{perf}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t})\sim\mathcal{D}}\left[2\cdot\mathrm{atan}(\log\pi(\tilde{a}_{t}|s_{t}))-\underset{i}{\min}\,Q_{i}(s_{t},\tilde{a}_{t})\right]$
(15) $\displaystyle L_{Q_{i}}^{perf}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[\left(r_{t}+\gamma\left(Q_{targ_{i}}(s_{t+1},\tilde{a}_{t})-\log\pi(\tilde{a}_{t}|s_{t})\right)-Q_{i}(s_{t},a_{t})\right)^{2}\right]$
(16)

Finally, the equations for the best generalizer when using Gym Pendulum are:

$\displaystyle L_{\pi}^{gen}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t})\sim\mathcal{D}}\left[\log(\log\pi(\tilde{a}_{t}|s_{t}))-\underset{i}{\min}\,Q_{i}(s_{t},\tilde{a}_{t})\right]$
(17) $\displaystyle L_{Q_{i}}^{gen}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[\left(r_{t}+\gamma\left(Q_{targ_{i}}(s_{t+1},\tilde{a}_{t})-\log(\log\pi(\tilde{a}_{t}|s_{t}))\right)-Q_{i}(s_{t},a_{t})\right)^{2}\right]$
(18)

###  G.2 Best performer for Brax Ant and Brax Humanoid

We present the loss equations (policy loss and critic loss) for the best
performer algorithms evolved in Brax Ant and Brax Humanoid; we focus on these
two algorithms in the analyses of this paper. The loss functions for the best
performer for Brax Ant are:

$\displaystyle L_{\pi}^{perf}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t})\sim\mathcal{D}}\left[\log\pi(\tilde{a}_{t+1}|s_{t+1})-\underset{i}{\min}\,Q_{i}(s_{t},a_{t})\right]$
(19) $\displaystyle L_{Q_{i}}^{perf}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[\left|\left(r+\gamma\left(\underset{i}{\min}\,Q_{targ_{i}}(s_{t},\tilde{a}_{t+1})-\gamma\right)-Q_{i}(s_{t},a_{t})\right)^{2}\cdot
C_{1}\right|\right]$ (20)

where

$C_{1}=r_{t}+\gamma\cdot\left(\underset{i}{\min}\,Q_{i}(s_{t+1},\tilde{a}_{t+1})-\gamma\right)$
(21)

In all cases, $\tilde{a}_{t+1}\sim\pi(\cdot|s_{t+1})$ and $\mathcal{D}$ is a
dataset of experience tuples from the replay buffer. Then, the equations for
the best performer evolved in Brax Humanoid are:

$\displaystyle L_{\pi}^{perf}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t})\sim\mathcal{D}}\left[\log\pi(\tilde{a}_{t+1}|s_{t+1})-\underset{i}{\min}\,Q_{i}(s_{t},a_{t})\right]$
(22) $\displaystyle L_{Q_{i}}^{perf}$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[\left(C_{2}-Q_{i}(s_{t},a_{t})\right)^{2}\cdot
C_{2}\right]$ (23)

where

$C_{2}=r_{t}+\gamma\left(\underset{i}{\min}\,Q_{targ_{i}}(s_{t},\tilde{a}_{t+1})-\log\pi(\tilde{a}_{t+1}|s_{t+1})\right)$
(24)

###  G.3 Additional analysis on evolved algorithms for RWRL Cartpole

Figure 17 shows the entropy and norm of the gradients of the actor for the
RWRL Cartpole best generalizer. We also show these same metrics for the warm-
start algorithm. This is an extension of Figure 7; in both cases, we let the
agents train for more episodes than those in the experimental setup. We see
that ignoring this fixed number of training episodes and letting run for
longer makes this type of metrics converge to similar values across
algorithms. We acknowledge that training until convergence is usually
preferred; however, in certain applications the number of training episodes
might be a constraint, so we find MetaPG’s ability to exploit this kind of
constraints beneficial in those setups.

(a) Average entropy of the policy during training for RWRL Cartpole.

(b) Average gradient norm of the actor loss during training for RWRL Cartpole.

Figure 17: Analysis of the entropy and gradient norm of the actor when
evaluating the best generalizer from RWRL Cartpole in comparison to the warm-
start.

###  G.4 Hyperparameter tuning for transfer and benchmark for RWRL and Gym

In our experiments with the RWRL Environment suite and Gym Pendulum, once an
evolution experiment is over and the evolved algorithms are meta-validated, we
compare them against: 1) ACME SAC (Hoffman et al., 2020), and 2) other RL
algorithms that have been evolved in a different environment. To that end, for
each ACME benchmark and evolved algorithm transfer, we tune the
hyperparameters of the algorithms. Since we consider two fitness scores in
this work (performance and generalizability adjusted for stability), we select
the two hyperparameter configurations that lead to the best stability-adjusted
performance and best stability-adjusted generalizability scores, respectively.
We denote these two configurations as the best performer and best generalizer,
respectively. To that end, we do a grid search across the sets of
hyperparameters listed in Table 10.

Table 10: Hyperparameter values considered during the tuning process. Hyperparameter | Values
---|---
Discount factor $\gamma$ | 0.9, 0.99, 0.999
Batch size | 32, 64, 128
Learning rate |  $1\cdot 10^{-4}$, $3\cdot 10^{-4}$, $1\cdot 10^{-3}$
Target smoothing coeff. $\tau$ | 0.005, 0.01, 0.05
Reward scale | 0.1, 1.0, 5.0, 10.0

This process is only carried out once the evolution is over; the warm-start
algorithm is not hyperparameter-tuned before evolution.

###  G.5 Hyperparameter tuning for transfer and benchmark for Brax

In our experiments in Brax Ant and Brax Humanoid, given they are more costly
environments, we do not meta-validate all algorithms in the population.
Instead, we choose the best algorithms during meta-training and directly meta-
test them with additional hyperparameter tuning. To that end, we do a grid
search across the hyperparameters listed in Table 11 and select the
configuration that maximizes the score we are interested in for each case, as
described in the previous subsection.

Table 11: Hyperparameter values considered during the tuning process. Hyperparameter | Values
---|---
Discount factor $\gamma$ | 0.95, 0.99, 0.999
Batch size | 128, 256, 512
Learning rate |  $1\cdot 10^{-4}$, $6\cdot 10^{-4}$, $1\cdot 10^{-3}$
Gradient updates per learning step | 32, 64, 128
Reward scale | 0.1, 1.0, 10.0, 100.0

###  G.6 Transferring algorithms evolved in RWRL and Gym

We present the results of carrying out transfer experiments in which we take
the best performer and best generalizer obtained after evolving in a specific
environment (RWRL Cartpole, RWRL Walker, or Gym Pendulum) and test them in the
two other environments considered. To that end, we follow the hyperparameter
tuning procedure described above and therefore, for each different RL
algorithm, we obtain the hyperparameter configurations that lead to the best
stability-adjusted performance and best stability-adjusted generalizability,
respectively. For example, taking the best performer from RWRL Walker (Walker
Perf.) and testing it on RWRL Cartpole leads to two sets of fitness scores
(best performer and best generalizer). The transfer results for RWRL Cartpole,
RWRL Walker, and Gym Pendulum can be observed in Tables 12, 13, and 14,
respectively.

Table 12: Transfer results on RWRL Cartpole. The row highlighted in gray corresponds to the results of the evolution experiment in RWRL Cartpole. The rest correspond to the best stability-adjusted performance and stability-adjusted generalizability configurations that result from doing hyperparameter tuning to the best performer and best generalizer evolved in different environments. | RWRL Cartpole
---|---
| Best performance | Best generalizability
RL Algorithm | $\tilde{f}_{perf}$ | $\tilde{f}_{gen}$ | $\tilde{f}_{perf}$ | $\tilde{f}_{gen}$
Cartpole | 0.868 | 0.459 | 0.756 | 0.551
Walker Perf. | 0.839 | 0.403 | 0.802 | 0.415
Walker Gen. | 0.576 | 0.335 | 0.576 | 0.335
Pendulum Perf. | 0.856 | 0.478 | 0.846 | 0.523
Pendulum Gen. | 0.804 | 0.438 | 0.687 | 0.469
ACME SAC | 0.864 | 0.312 | 0.833 | 0.478
Table 13: Transfer results on RWRL Walker. The row highlighted in gray corresponds to the results of the evolution experiment in RWRL Walker. The rest correspond to the best stability-adjusted performance and stability-adjusted generalizability configurations that result from doing hyperparameter tuning to the best performer and best generalizer evolved in different environments. | RWRL Walker
---|---
| Best performance | Best generalizability
RL Algorithm | $\tilde{f}_{perf}$ | $\tilde{f}_{gen}$ | $\tilde{f}_{perf}$ | $\tilde{f}_{gen}$
Cartpole Perf. | 0.955 | 0.477 | 0.952 | 0.521
Cartpole Gen. | 0.029 | 0.031 | 0.024 | 0.034
Walker | 0.961 | 0.526 | 0.950 | 0.554
Pendulum Perf. | 0.466 | 0.230 | 0.466 | 0.230
Pendulum Gen. | 0.905 | 0.465 | 0.903 | 0.483
ACME SAC | 0.965 | 0.430 | 0.918 | 0.498
Table 14: Transfer results on Gym Pendulum. The row highlighted in gray corresponds to the results of the evolution experiment in Gym Pendulum. The rest correspond to the best stability-adjusted performance and stability-adjusted generalizability configurations that result from doing hyperparameter tuning to the best performer and best generalizer evolved in different environments. | Gym Pendulum
---|---
| Best performance | Best generalizability
RL Algorithm | $\tilde{f}_{perf}$ | $\tilde{f}_{gen}$ | $\tilde{f}_{perf}$ | $\tilde{f}_{gen}$
Cartpole Perf. | 0.862 | 0.329 | 0.857 | 0.349
Cartpole Gen. | 0.821 | 0.323 | 0.821 | 0.323
Walker Perf. | 0.860 | 0.324 | 0.813 | 0.375
Walker Gen. | 0.849 | 0.319 | 0.679 | 0.388
Pendulum | 0.877 | 0.349 | 0.834 | 0.424
ACME SAC | 0.865 | 0.380 | 0.865 | 0.391