# Curriculum Reinforcement Learning via Morphology-Environment Co-Evolution

Shuang Ao1, Tianyi Zhou2, Guodong Long1, Xuan Song3, Jing Jiang1
University of Technology Sydney1;
University of Maryland, College Park2;
Southern University of Science and Technology3
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{guodong.long<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Throughout long history, natural species have learned to survive by evolving
their physical structures adaptive to the environment changes. In contrast,
current reinforcement learning (RL) studies mainly focus on training an agent
with a fixed morphology (e.g., skeletal structure and joint attributes) in a
fixed environment, which can hardly generalize to changing environments or new
tasks. In this paper, we optimize an RL agent and its morphology through
“morphology-environment co-evolution (MECE)”, in which the morphology keeps
being updated to adapt to the changing environment, while the environment is
modified progressively to bring new challenges and stimulate the improvement
of the morphology. This leads to a curriculum to train generalizable RL, whose
morphology and policy are optimized for different environments. Instead of
hand-crafting the curriculum, we train two policies to automatically change
the morphology and the environment. To this end, (1) we develop two novel and
effective rewards for the two policies, which are solely based on the learning
dynamics of the RL agent; (2) we design a scheduler to automatically determine
when to change the environment and the morphology. In experiments on two
classes of tasks, the morphology and RL policies trained via MECE exhibit
significantly better generalization performance in unseen test environments
than SOTA morphology optimization methods. Our ablation studies on the two
MECE policies further show that the co-evolution between the morphology and
environment is the key to the success.

##  1 Introduction

Deep Reinforcement learning (RL) has achieved unprecedented success in some
challenging tasks (Lillicrap et al., 2016; Mnih et al., 2015). Although
current RL can excel on a specified task in a fixed environment through
massing training, it usually struggles to generalize to unseen tasks and/or
adapt to new environments. A promising strategy to overcome this problem is to
train the agent on multiple tasks in different environments (Wang et al.,
2019a; Portelas et al., 2019; Gur et al., 2021; Jaderberg et al., 2017) via
multi-task learning or meta-learning (Salimans et al., 2017; Finn et al.,
2017). However, it increases the training cost and the space for possible
environments/tasks can be too large to be fully explored by RL. So how to
select the most informative and representative environments/tasks to train an
RL agent to evolve generalizable skills becomes an critical open challenge.

Figure 1: MECE vs. previous methods. (a) Evolution strategy (ES)-based method
for morphology optimization. It starts from agents with different morphology,
eliminates those with poor performance, and then applies random mutation to
the survivors. (b) Transform2Act, which trains an RL policy to modify the
morphology of an agent in a fixed environment. (c) MECE (ours). We train two
policies to optimize the morphology and change the environment to form a
curriculum over the course of training a control policy.

Unlike RL agents that do not actively seek new environments to improve their
learning capability, natural species have full motivations to do so to survive
in the competitive world, and one underlying mechanism to drive them is
evolution. Evolution is a race with the changing environment for every
species, and a primary goal is to accelerate its adaptation to new
environments. Besides merely optimizing its control policy, evolution more
notably changes the morphology of species, i.e., the skeletal structure and
the attributes for each part, in order to make them adapt to the environment.
In fact, the improvement on morphology can be more critical because there
could exist a variety of actions or skills an agent cannot do (no matter how
the control policy is optimized) without certain structures, e.g., more than
one leg, a long-enough limb, or a 360-degree rotation joint. For RL, we claim
that A good morphology should improve the agent’s adaptiveness and
versatility, i.e., learning faster and making more progress in different
environments. From prehistoric person to modern Homo sapiens, there is a
definite association between the rise of civilization and the Homo sapiens’
optimization of their physical form for improved tool use. Unfortunately, the
morphology in many RL researches are pre-defined and fixed so it could be sub-
optimal for the targeted environments/tasks and may restrain the potential of
RL. Although morphology can be optimized using RL (Sims, 1994; Wang et al.,
2019b; Kurin et al., 2021; Yuan et al., 2022) to improve the final performance
for a specific task or environment, it was not optimized for the adaptiveness
to varying environments. Moreover, instead of re-initializing the control/RL
policy after every morphology modification or environment change, the millions
of years of evolution demonstrate the effectiveness of continual and
progressive learning of the control policy over generations of morphology
evolution along with a sequence of varying environments. How to optimize the
morphology and control policy of an RL agent for the above goal is still an
underexplored problem.

Given that the agent has the freedom to improve its morphology and control
policy for faster adaptation to different environments/tasks, the remaining
question is: what environment can improve this learning process and
incentivize the agent to keep finding better morphology (rather than staying
with the same one)? The learning environment plays an essential role in RL
since it determines the data and feedback the agent can collect for its
training. In this paper, we adopt a simple criterion: A good environment
should accelerate the evolution of the agent and help it find better
morphology sooner, which will enhance the agent’s adaptiveness and versatility
over new environments/tasks.

The interplay between morphology and environment has lasted for billions of
years. In the endless game, species which find the appropriate environments to
evolve its morphology and policy may survive and keep being improved via
adaptation to new environments/tasks. Inspired by this, we propose an
morphology-environment co-evolution (MECE) scheme that automatically generates
a curriculum of varying environments to optimize the agent’s morphology and
its RL policy so they can be generalized to unseen environments and adapted to
new morphology, respectively. In MECE, we train two RL policies to modify the
morphology and change the environment, which create a sequence of learning
scenarios to train the control policy continuously. This differs from previous
work that select the morphology or the environment from a pre-defined set of
candidates. Specifically, unlike zeroth-order optimization approaches (Wang et
al., 2019b; Hejna et al., 2021; Wang et al., 2018), a morphology policy
network can generate modifications based on all the morphology evaluated in
diverse environments from history. On the other hand, an environment policy
can produce a sequence of progressively modified environments instead of
randomly selected or sampled ones (Portelas et al., 2019; Wang et al., 2019a).
In Fig. 1, we illustrate the co-evolution process of MECE and compare it with
evolution-strategy (ES) based method (Wang et al., 2019b) and Transform2Act
(Yuan et al., 2022), which uses RL to optimize the morphology in a fixed
environment.

Inspired by the former discussion of “good morphology” and “good environment”,
we optimize the morphology policy to improve the agent’s
adaptiveness+versatility and optimize the environment policy to improve the
morphology’s evolution. To this end, we design two rewards for the two
policies based on the learning progress of the control policy, which is
estimated by comparing its concurrent performance on a validation set and its
historical trajectories. Therefore, the three policies are complementary and
mutually benefit each other: (1) the morphology and environment policies
create a curriculum to improve generalizability of the control policy; (2) the
environment policy creates a curriculum to optimize the morphology policy; (3)
the control policy provides rewards to train the other two policies. In MECE,
we train the control policy and automatically determine when to apply the
other two policies to modify the morphology or change the environment: they
are triggered by the slow progress on the current morphology and environment,
respectively. In experiments over different tasks, the morphology and control
policy learned through MECE significantly outperforms the existing method in
new environments. Moreover, MECE exhibits much faster-learning speed than
baselines. We further provide an extensive ablation study to isolate each main
component of MECE’s contribution and compare it with other options.

##  2 Related Work

Continuous Design Optimization. A line of research in the community has
studied optimizing an agent’s continuous design parameters without modifying
its skeletal structure, and they commonly optimize on a specific kind of
robots. Baykal & Alterovitz (2017) studies an annealing-based framework for
cylindrical robots optimization. Ha et al. (2017); Desai et al. (2017)
optimize the design of legged robots by trajectory optimization and implicit
function. Applying deep RL into design optimization becomes more popular
recently. Chen et al. (2020) uses computational graphs to model robot hardware
as part of the policy. CMA-ES (Luck et al., 2019) optimizes robot design via a
learned value function. Ha et al. (2017) proposes a population-based policy
gradient method. Another line of work (Yu et al., 2019; Exarchos et al., 2021;
Jiang et al., 2021) search for the optimal robot parameters to fit an incoming
domain by RL. Different from the prior works, our approach learns to find the
most general skeleton of an agent while maintaining its continuous design
parameters optimization to diverse environments or tasks.

Combinatorial morphology Optimization. One closely related line of work is the
design of modular robot morphology spaces and developing algorithms for co-
optimizing morphology and control (Sims, 1994; Liao et al., 2019; Gupta et
al., 2022; Trabucco et al., 2022) within a design space to find task-optimized
combinations of controller and robot morphology. When the control complexity
is low, evolutionary strategies have been successfully applied to find diverse
morphologies in expressive soft robot design space (Cheney et al., 2013,
2018). For more expressive design spaces, GNNs have been leveraged to share
controller parameters (Wang et al., 2019b) across generations or develop novel
heuristic search methods for efficient exploration of the design space (Zhao
et al., 2020). In contrast to task specific morphology optimization, Hejna et
al. (2021) propose evolving morphologies without any task or reward
specification. Hejna et al. (2021) employ an information-theoretic objective
to evolve task-agnostic agent designs.

Generalization to environments. Several recent works (Wang et al., 2019a;
Portelas et al., 2019; Gur et al., 2021; Parker-Holder et al., 2022) show that
the efficiency and generalization of RL can be improved by training the policy
in different environments. By sampling the parameters of environmental
features, they Portelas et al. (2019); Wang et al. (2019a) teach the RL agent
using a curriculum of various environments. To estimate the distribution of
environments, however, requires evaluating the RL policy on numerous sampled
environments, which might be expensive and inaccurate for intricate ones. The
modified environments may also be unfeasible or too difficult for RL
exploration. In MECE, we train an environment policy to change the
environments adaptive to the evolved agent’s learning progress, and it is
possible to regulate the level of difficulty of the modified environments.

##  3 Formulations of the Three Policies in MECE

In MECE, we have three RL policies, a control policy $\pi$ that learns to
control the agents of evolved morphology, a morphology policy $\pi_{m}$ that
learns to modify the agent’s morphology for better robustness in diverse
environments, and an environment policy $\pi_{e}$ that learns to change the
environment to boost the morphology evolution. During the training phase,
$\pi_{m}$ and $\pi_{e}$ are alternately applied to evolve the agent’s
morphology and the training environment, respectively. Taking into account
that $\pi$ might require various environment steps for training in distinct
morphologies or environments, we propose a dynamic time-scaling for $\pi_{m}$
and $\pi_{e}$ that is adaptive to $\pi$’s learning process.

Agent control. The problem of controlling an agent can be modeled as a Markov
decision process (MDP), denoted by the tuple
$\\{\mathcal{S},\mathcal{A},p,r,\gamma\\}$, where $\mathcal{S}$ and
$\mathcal{A}$ represent the set of state space and action space respectively,
$p$ means the transition model between states, and $r$ is the reward function.
An agent of morphology $m_{i}\in M$ in state $s_{t}\in\mathcal{S}$ at time $t$
takes action $a_{t}\in\mathcal{A}$ and the environment returns the agent’s new
state $s_{t+1}$ according to the unknown transition function
$p(s_{t+1}|s_{t},a_{t},m_{i})$, along with associated reward
$r_{t}=r(s_{t},a_{t})$. The goal is to learn the control policy
$\pi^{*}:\mathcal{S}\rightarrow\mathcal{A}$ mapping states to actions that
maximizes the expected return $\mathbb{E}[\mathcal{R}_{t}]$, which takes into
account reward at future time-steps
$J(\pi)=\mathbb{E}[\mathcal{R}_{t}]=\mathbb{E}\left[\sum_{t=0}^{H}\gamma^{t}r_{t}\right]$
with a discount factor $\gamma\in[0,1]$, where $H$ is the variable time
horizon.

We use a graph neural network (GNN) for the control policy because (1) it can
be generalized to different morphologies and learn fundamental control skills;
(2) it captures the skeleton structure and encodes physical constraints.
Specifically, we represent the agent’s skeleton by a graph $G=(V,A,E)$ where
each node $u\in V$ represents a joint and each edge $e\in E$ represents a bone
connecting two joints. Each joint $u$ is associated with a representation
vector $z_{u}$ storing attributes in $A$, e.g., bone length, motor strength,
size. The GNN takes $z_{u}$ as inputs and propagate messages across nodes on
the graph to produce a hidden state for each node.

Morphology evolution. We model the morphology evolution as an MDP problem,
denoted by the tuple $\\{\mathcal{S}_{m},\mathcal{B},p_{m},r_{m},\rho\\}$.
$\mathcal{S}_{m}$ represents the set of state space, and a state $s_{\alpha
t}^{m}\in\mathcal{S}_{m}$ at time-step ${\alpha t}$ is defined as
$s^{m}_{\alpha t}=G_{\alpha t}$, where $G_{\alpha t}$ is the skeleton
structure of agent. The action $b_{\alpha t}$ sampled from the set of action
space $\mathcal{B}$ can modify the topology of the agent, that has three
choices in $\\{AddJoint,DelJoint,NoChange\\}$. The transition dynamics
$p_{m}(s_{{\alpha t}+1}^{m}|s_{{\alpha t}}^{m},b_{\alpha t})$ reflects the
changes in state and action. We define the reward function for $\pi_{m}$ as
the average improvement of training $\pi$ with the current evolved morphology
in different environments. The reward $r_{m}$ at time-step $\alpha t$ is
denoted as

$r_{\alpha t}^{m}=\frac{1}{|E|}\sum_{\theta^{E}\in
E}\left(\mathcal{R}(H,\theta^{E},G_{\alpha
t})-\mathcal{R}(H,\theta^{E},G_{\alpha t-1})\right)\\\ -\lambda\|b^{m}_{\alpha
t}\|^{2}_{2},$ (1)

note that the first term in Eq. 1 indicates the average performance of the
current morphology in diverse environments, and the second term is a cost for
each action taken by $\pi_{m}$ that modifies the morphology of the agent. $E$
is the set of environments used for evaluation.

Environment evolution. We model environment evolution as an MDP problem,
denoted by the tuple $\\{\mathcal{S}_{e},\mathcal{C},p_{e},r_{e},\rho\\}$. The
state $s^{e}_{\beta t}=(G_{\beta t},\theta^{E}_{\beta t})$ in the set of state
space $\mathcal{S}_{e}$ includes the agent’s topology $G_{\beta
t}=(\mathcal{V}_{\beta t},\mathcal{E}_{\beta t})$ and the parameter of
environment $\theta_{E}\in\Theta_{E}$. The transition dynamics
$p_{e}(s^{e}_{\beta t+1}|s^{e}_{\beta t},c_{\beta t})$ reflects the changes in
the state by taking an action at time-step $\beta t$. The action $c_{\beta t}$
will directly change the environment parameters (more detailed in Appendix.A).
We define the reward for $\pi_{e}$ as the learning progress of multiple
morphologies in the current environment. Thus, training $\pi_{e}$ will
encourage $\pi_{m}$ to optimize the morphology of better generalization
sooner. The reward $r_{e}$ at time-step $\beta t$ is denoted as

$\displaystyle r^{E}_{\beta t}$ $\displaystyle=\mathcal{P}_{\beta
t}-\mathcal{P}_{\beta t-1},$ (2) $\displaystyle where,~{}~{}\mathcal{P}_{\beta
t}$ $\displaystyle=\mathcal{R}(H,\theta_{\beta t}^{E},G_{\beta
t})-\mathcal{R}(H,\theta_{\beta t-1}^{E},G_{\beta t}).$

in Eq. 2, $\mathcal{P}_{k}$ is the learning progress of the RL agent in an
environment defined on $\mathcal{R}(H,\theta_{\beta t}^{E},G_{\beta t})$,
which denotes the expected return of the RL agent $G_{\beta t}$ of $H$ time-
steps evaluated in environment $\theta_{k}^{E}$.

Short evaluation window. Since $r_{m}$ and $r_{e}$ are respectively based on
the training process of $\pi$ on different morphologies or environments, it is
cost but necessary to rollout $\pi$ periodically to collect data. However,
Hejna et al. (2021) demonstrates that evaluating with short horizon is enough
to provide informative feedback to reflect the training process of the current
policy. In light of this, we run a short evaluation window for $\pi$ after
every period of environment steps taken by the agent. In each evaluation
window, we rollout multiple morphologies of the best performance by $\pi$ in
several most recent environments, and then we can calculate $r_{m}$ and
$r_{e}$ based on the evaluation results. More details refer to Appendix.C.

In this paper, we use a standard policy gradient method, PPO (Schulman et al.,
2017), to optimize these three policies. Note that the time scale of the main
policy, morphology policy $\pi_{m}$ and environment policy $\pi_{e}$ are
different, indicated by the subscript $\alpha$ and $\beta$, which we will
explain more detailed in the following section.

##  4 Algorithm of MECE

Algorithm 1 MECE

1:Initialize: control policy $\pi$, morphology policy $\pi_{m}$, environment
policy $\pi_{e}$, dataset $\mathcal{D}\leftarrow\emptyset$, initial agent
morphology $m_{0}$, initial environment parameter $\theta^{E}_{0}$;

2:while Not reaching max iterations do

3: for $t=0,1,\cdots,\tau_{\max}$ do $\triangleright$ RL exploration

4: Sample control action $a_{t}\sim\pi$;

5: $s_{t+1}\leftarrow\mathcal{T}(s_{t+1}|s_{t},a_{t})$;

6: $r_{t}\leftarrow$ environment reward;

7: $m_{t+1}\leftarrow m_{t}$, $\theta^{E}_{t+1}\leftarrow\theta^{E}_{t};$

8: Store $(s_{t},r_{t},a_{t},s_{t+1},m_{t},\theta_{t}^{E})$ into
$\mathcal{D}$;

9: end for

10: Update $\pi$ with PPO using samples in $\mathcal{D}$;

11: $(m_{t+1},\theta^{E}_{t+1})$ = CO-EVO($m_{t},\pi,\theta^{E}_{t}$);

12:end while

In MECE, we need jointly train three policies: control policy $\pi$ learns to
complete tasks, morphology policy $\pi_{m}$ learns to evolve the agent’s
morphology to be more adaptive to diverse environments, and environment policy
$\pi_{e}$ learns to modify the training environments more challenging. At
first glance, training multiple RL policies may appear more difficult than
training a single RL policy, as it needs the collection of additional data via
interaction. However, MECE enables three policies to help in each other’s
training through the co-evolving mechanism, where each policy is trained on a
curriculum of easy-to-hard tasks utilizing dense feedback from the training
experience of other policies. By iterating this co-evolution process, MECE
considerably improves the training efficiency of each policy, resulting in an
RL agent of morphology that is more generalizable across environments.

In each episode, the control policy $\pi$ starts from training on an initial
agent in a randomly sampled environment (refer to Line.3-9 in Alg. 1). This
initial agent’s skeleton structure is relatively simple and is easy to learn
according to its naive morphology. Even though the naive agent is not
generalized due to its constrained morphology, it can initially provide dense
feedback for training $\pi_{m}$ and $\pi_{e}$. On the other hand, beginning
from an agent with a random morphology is theoretically feasible but
inefficient when initializing a complicated morphology that is difficult to
learn to control and yields uninformative feedback for training the other
policies.

Algorithm 2 CO-EVO

1:Input: Current morphology $m_{\alpha t}$, control policy $\pi$, training
environment $\theta^{E}_{\beta t}$, dataset $\mathcal{D}$.

2:Procedure CO-EVO($m_{\alpha t},\pi,\theta_{\beta t}^{E}$):

3:Evaluate $\pi$ with morphology $m_{\alpha t}$ in $\theta^{E}_{\beta t}$;

4:Calculate reward $r_{m}$ and $r_{e}$ following Eq. 1 and Eq. 2;

5:if $r_{m}\leq\delta_{m}$ then $\triangleright$ Modify the agent’s morphology

6: Apply $\pi_{m}$ to modify $m_{\alpha t}$ to $m_{\alpha t+1}$;

7: Update $\pi_{m}$ with PPO using samples in $\mathcal{D}$;

8:else

9: if $r_{e}\leq\delta_{e}$ then $\triangleright$ Modify the training
environment

10: Apply $\pi_{e}$ to modify $\theta_{\beta t}^{E}$ to $\theta_{\beta
t+1}^{E}$;

11: Update $\pi_{e}$ with PPO using samples in $\mathcal{D}$;

12: end if

13:end if

14:Return ($m_{\alpha t+1},\theta^{E}_{\beta t+1}$)

We assume that the control policy is now mature on the current morphology
after training. Then MECE proceeds to a series of co-evolution phases, where a
phase is associated with achieving robustness across the evolved morphologies
and environments (refer to Line.11 in Alg. 1). In each phase, as shown in
Line.3-4 of Alg. 2, we first evaluate the control policy to collect rewards
for training $\pi_{m}$ and $\pi_{e}$. Note that this step is not cost, as the
RL agent executes environment steps with a short horizon. After that, we
alternately apply $\pi_{m}$ and $\pi_{e}$ to co-evolve the agent’s morphology
and the training environments based on two criteria of $r_{m}$ and $r_{e}$
(refer to Line.5 and 9 in Alg. 2).

Dynamic update window based on the reward criteria. As mentioned in section 3,
$r_{m}$ and $r_{e}$ indicate the learning progress of the current morphology
in the training environment respectively. In particular, a nominal value of
$r_{m}$ corresponds to the adaptability of the current morphology to different
environments and indicates that training with the current morphology cannot
significantly increase $\pi$’s performance. In this instance, we should apply
$\pi_{m}$ to optimize the morphology to increase its generalization to
environments. Similarly, a minor $r_{e}$ indicates that modifying the agent’s
morphology will have a negligible effect on its performance in the current
environment, and the environment should be modified to be more challenging to
boost the morphology evolution. As a result, we employ two independent
criteria for $r_{m}$ and $r_{e}$ to formulate a dynamic update window, which
allows MECE to adapt to the learning progress of morphology and environment
and apply $\pi_{m}$ or $\pi_{e}$ to produce corresponding evolution and
alterations.

We now have an agent of newly evolved morphology or a modified training
environment, and then forward to the next iteration of training the control
policy $\pi$ on them. Using $r_{m}$ and $r_{e}$, MECE has achieved the
alternating evolution of morphology and environment. MECE not only improves
the robustness of the morphology to various environment, but also improves the
learning efficiency of the policies through the use of dense feedback.

##  5 Experiments

We design our experiments to answer the following questions: (1) Given dynamic
environments, does our method outperform previous methods in terms of
convergence speed and final performance? (2) Does our method create agents of
better generalization to diverse environments?

###  5.1 Environments Setup

Diverse environments varying features that require the agent to evolve out
different morphologies, e.g., a bipedal agent has too short legs to go across
a high obstacle, or a agent of four legs navigate on rough road more smoothly
than one of one/two legs. In our experiments, we conduct three environments
that requires various morphologies to complete the tasks. We use a tuple
$\theta^{E}$ of environment parameters to denote the possible physical
features of environments.

In experiments, we have three types of environments: (1) 2d locomotion: In the
2D locomotion environment, agents are limited to movement in the X-Z plane.
The state $s_{T}$ is given by the final $(x,z)$ positions of the morphology
joints. We evaluate morphologies on three tasks: running forwards, running
backwards, and jumping. Environment policy $\pi_{e}$ takes actions to change
the roughness of terrains, that is controlled by $\theta^{E}$. (2) 3d
locomotion: where a 3D agent’s goal is to move as fast as possible along
x-axis and is rewarded by its forward speed along x-axis. Environments have a
fixed number of obstacles and terrains of different roughness which are
controlled by $\theta^{E}$. $\pi_{e}$ can not only change the roughness of
terrains in the environments, also learns to modify the average distance
between obstacles. (3) Gap crosser: Gap Crosser, where a 2D agent living in an
$xz-$plane needs to cross periodic gaps and is rewarded by its forward speed.
The width of the gap is controlled by $\pi_{e}$. In order to avoid unlearnable
environments, the upper and lower limits of the gap width are limited. More
details refer to Appendix.A.

Baselines. We compare our method with the following baselines that also
optimize both the skeletal structure and joint attributes of an agent.
(1)Neural Graph Evolution (NGE) (Wang et al., 2019b), which is an ES-based
method that uses GNNs to enable weight sharing between an agent and its
offspring. (2) Transform2Act (Yuan et al., 2022), that optimizes an RL policy
to control the evolution of agents. Note that both baselines do not have
diverse environments in the original code. For a fair comparison, we apply an
environment set randomly sampled from a uniform distribution in the training
phase for them. (3) enhanced POET (Wang et al., 2020), that pertains the
modification of the training environments that are paired with agents
possessing distinct yet unchanging morphologies. Given the absence of
morphology optimization/evolution in enhanced POET, we conducted a random
sampling of six sets of agents, each comprising five agents with distinct
morphologies, resulting in a total of 30 agents. Subsequently, we selected the
top-performing agent from each set for the test.

Figure 2: Baselines comparison and optimized morphology. In Fig.(a)-(c), we
report the evaluation results of MECE and baseline methods in three
environments, and we plot the accumulated rewards (mean$\pm$ std averaged over
6 random seeds) against the number of simulation steps for all methods. For
fair comparison, we add periodically changing environments that randomly
sampled from a fixed distribution in the training of baseline methods. MECE
has better learning efficiency and final performance than all baseline
methods. In Fig.(d)-(f), we list the optimal morphology that evolved by each
method. Intuitively, the structural symmetry and environmental adaptability of
the MECE-optimized morphology is better. Especially in 3d locomotion, the
agent developed by MECE is better at navigating terrain and avoiding
obstacles.

###  5.2 Main results

In Fig. 2(a)-(c), we report the average performance of MECE and all baseline
methods in test environments for three settings. The accumulated rewards are
averaged over 12 UNSEEN and randomly sampled environments and 6 different
random seeds. In terms of the learning efficiency and final performance, MECE
outperforms baseline methods in all environments. The results show that while
Transform2Act’s final performance can be improved by periodically changing the
training environments, MECE’s learning efficiency is still notably higher than
that of Transform2Act. The best morphologies found by each approach are listed
in Fig. 2(d)-(f) for easier comparison. For 2d locomotion MECE develops a
morphology that resembles a crawling animal that can move fast over terrain.
In 3D locomotion, MECE has developed a spider-like framework. MECE evolves a
more streamlined structure in comparison to Transform2Act so that it can more
possibly avoid obstacles. Finally, the Hopper-like agents developed by
Transform2Act and MECE are comparable in the Gap crosser that can jump across
gaps. Overall, MECE-optimized morphologies have superior structural symmetry
and are more consistent with the characteristics of biological evolution in
various environments.

###  5.3 Ablation Study and Analysis

We product out a series of ablation studies to prove the effectiveness of each
part in MECE. We list the introduction information of each ablation study in
Tab. 1, and report the results in Fig. 3. Note that in each ablation study, we
only change one part of MECE and maintain the other parts the same as the
original MECE, and all results are the test performance evaluated on the same
environment set in Fig. 2 (b).

Ablation Study | Definition of Legends
---|---
Ablation study-I for $\pi_{e}$. Fig. 3(a) |  original: Periodic environments generated by $\pi_{e}$.
periodic envs-random: Periodic environments of randomly sampled.
fixed envs-initial: Initial and easy environment.
fixed envs-final: Fixed environment generated by $\pi_{e}$.
Ablation study-II for $\pi_{m}$. Fig. 3(b) |  original: Dynamic morphology evolved by $\pi_{m}$.
random morph: Randomly mutate the morphology.
fixed morph-initial: Fixed agent of the initial morphology.
fixed morph-final: Fixed agent of the final morphology evolved by $\pi_{m}$.
Ablation study-III for dynamic update window. Fig. 3(c) |  Original: Dynamic evaluation window.
fixed update window: Update $\pi_{e}$ and $\pi_{m}$ at a fixed frequency.
Ablation study-IV for $r_{m}$ and $r_{e}$. Fig. 3(d) |  original: Follow the reward setting of $\pi_{m}$ ($\pi_{e}$) in Eq. 1 (Eq. 2).
reward-i: Remove the reward for $\pi_{m}$.
reward-ii: change $r_{m}$ to Eq. 1.
reward-iii: change $r_{e}$ to Eq. 2.
Table 1: List of ablation studies. This list includes definitions for each set
of controls as well as the objectives of ablation study I-IV. Since we only
validated one design of MECE in each ablation study, only one design from each
control group was different from MECE (original).

Ablation Study I. In this ablation study, we focus on the effectiveness of
$\pi_{e}$ for the training and report the results in Fig. 2(a). Note that the
initial environment is relatively simple, while the final environment that
evolved by $\pi_{m}$ is more challenging. The results show that the evolved
morphology and control policy trained in the diverse environments are more
general than trained in the fixed environment, no matter the environment is
simple or not. On the other hand, compare the performance of MECE with MECE
(periodic envs), we can find that $\pi_{e}$ helps the training in terms of the
efficiency and the final performance. This is because training in the
environment modified by $\pi_{e}$, compared to the randomly generated one,
avoids the learning gap between two extremely different environments, and is
smoother for $\pi$ and $\pi_{m}$ to learn. On the other hand, $\pi_{e}$ learns
to generate environment to accelerate the learning progress of $\pi$ and
$\pi_{m}$, which is shown clearly in the learning curves of 10 - 20 million
simulation steps in Fig. 3(a).

Figure 3: Ablation study. We design four ablation studies to illustrate the
advantages of each part in MECE. The results of ablation study I and II show
the effectiveness of $\pi_{e}$ and $\pi_{m}$ on the learning efficiency and
generalization. Ablation study III proves that applying $\pi_{e}$ and
$\pi_{m}$ adaptive to the training process of the control policy can improve
its robustness. In ablation study IV, we try different setting of the reward
function of $\pi_{e}$ and $\pi_{m}$, and the results prove that the original
reward setting is optimal. More details of each ablation study can be found in
Tab. 1.

Ablation Study II. The purpose of this ablation study is to demonstrate how
$\pi_{m}$ can produce morphology that is more adaptable to diverse
environments and increase learning effectiveness. The results are shown in
Fig. 3(b). When comparing the learning curves for MECE (random morph) and MECE
(original), the former has a far higher early learning efficiency than the
latter. This is due to $\pi_{e}$ remaining constant and the environment not
changing significantly, which prevents the morphology from being adequately
adapted to a variety of habitats, even some comparable environments.
Theoretically, the "fixed morph-final" morphology should be able to be more
adaptive to environments and, with training, reach competitive performance.
Although it performs better than "random morph" and "fixed morph-initial,"
"MECE(original)" is still far behind. The findings indicate that training an
agent with a complex morphology (i.e., MECE(fixed morph-final)) in diverse
environments is difficult and ineffective. Therefore, it is imperative to
implement an automated curriculum that encompasses diverse environments and
morphologies in MECE to effectively train an adaptable agent.

Ablation Study III. Through a co-evolutionary process of adaptive criterion on
$r_{m}$ and $r_{e}$, $\pi_{m}$ and $\pi_{e}$ learn to evolve the morphology
and environment. To determine whether this design is effective, we perform an
ablation experiment. The results are shown in Fig 3(c). For “MECE (fixed
evaluation window)”, every fixed number of environment steps, $\pi_{m}$ and
$\pi_{e}$ take actions to change the agent’s morphology and environment. The
results indicate that by removing the dynamic window, MECE’s learning
effectiveness has drastically decreased. It is challenging to coordinate the
training developments of the three policies, particularly the control policy,
without a dynamic evaluation window. Only a somewhat steady control policy, as
was already mentioned, can give the other two reliable feedback for training.
Moreover, the performance of Transform2Act is taken into account for easy
comparing. Even though "MECE(fixed evaluation window)" is less effective, a
competitive final performance to Transform2Act is still feasible.

Ablation Study IV. We designed this ablation investigation to verify the
reward of $\pi_{m}$ and $\pi_{e}$ in MECE, and the findings are displayed in
Fig. 3(d). Note that just one incentive is altered in each scenario in this
experiment; all other rewards remain unchanged. We explore two scenarios for
$\pi_{m}$: the first involves removing the reward (“MECE (reward-i)”), same to
the setting of transform2Act), and the second involves substituting the formal
of $r_{e}$ (“MECE (reward-ii)”), i.e., accelerating learning progress. We take
into account employing learning progress (in the form of $r_{m}$, “MECE
(reward-iii)”) for $\pi_{e}$. The results show that the original design has
obvious advantages in both learning efficiency and final performance. For the
issues addressed by MECE, $\pi_{m}$ can be thought of as the meta learner of
$\pi$, whereas $\pi_{e}$ is the meta learner of the former. In order to help
$\pi$ perform better on various tasks, $\pi_{m}$ learns to modify the agent’s
morphology during training, and $\pi_{e}$ speeds up both of their learning
processes. Therefore, in both theoretical and practical tests, the original
design is the most logical.

###  5.4 Case study: Co-evolution between morphology and environment

To further comprehend the co-evolution of morphology and environment in MECE,
we perform a case study in 2d locomotion and present results in Fig. 4. The
y-axis of the point plot indicates the training environment’s roughness. Fig.
4(a)-(f) are schematic descriptions of the current morphology and the training
environment corresponding to it.

Figure 4: Case study results. This figure more vividly illustrates a schematic
diagram of the co-evolution of morphology and environment during training. The
ordinate in the illustration shows the average environments’ roughness
changing as the training processes (6 random seeds). Although the training
environments generated by $\pi_{e}$ are similar to the randomly sampled at the
beginning, it can be observed from the pointplot that $\pi_{e}$ can guarantee
more challenging and stable training environments. Fig.(a)-(f) compare the
effectiveness of $\pi_{e}$ that correspond to changes in the training
environments (blue indicates original MECE, and orange indicates random
environments). Contrarily, it is evident that no $\pi_{e}$ is likely to result
in exceptionally difficult environments (Fig.(d)), while the environment
produced by $\pi_{e}$ is challenging but learnable (Fig.(c)).

Generally, an effective training environment should meet two conditions
simultaneously. The first is learnable, and the RL agent can collect training
data via exploration. The second factor is the level of difficulty;
environments that are either too easy or too challenging may reduce the
learning efficiency of the RL agent. Therefore, in 2D locomotion, a reasonable
environment is one in which the environment’s roughness is quite large
(challenging), but not so large as to render the agent unconquerable
(learnable) during exploration. Using the results of the pointplot, the
environment policy can ensure that the ratio of environmental unevenness
during the training process to the height of the current agent is
approximately 1, which satisfies the training environment criteria. At
contrast, despite the fact that the randomly generated environment can ensure
the ideal in some phases, it is extremely unstable (the standard deviation is
visibly high), which would undoubtedly lower the learning effectiveness.

Specifically, comparing Fig. 4(a) and (c), the environment generated by
$\pi_{e}$ in the early stage of training is unlearnable for the current agent,
whereas in the middle stage of training, the training environment is
relatively flat, but challenging, especially in terms of the number of
occurrences of a particular feature. The roughness of the environment on the
right fluctuates frequently. Note again compared to Fig. 4(d) that the right
side of the randomly produced environment is unlearnable for the current agent
in the middle of training. This phenomena does not occur in MECE because
$\pi_{e}$ modifies the environment to be morphology-adaptive. Comparing Figs.
4(e) and (f), the environment formed by $\pi_{e}$’s slope exhibits noticeable
but relatively subtle alterations. This is due to the fact that the existing
morphology has a high degree of adaptation to varied contexts, and
environments with more extreme alterations have less effect on the
morphology’s optimization. To better train the control policy, an environment
of moderate difficulty should be adopted.

##  6 Conclusion

In this paper, we offer a framework for morphology-environment co-evolution
that evolves the agent’s morphology and the training environment in
alternation. Compared to previous morphology optimization approaches, ours is
more sample-efficient and learns to develop morphologies that are more robust
across diverse environments. Experiments demonstrate that our approach
outperforms baseline methods in terms of convergence speed and overall
performance. In the future, we are interested in using curriculum RL
approaches to further improve the learning efficiency of our approach so that
it can adapt to more challenging environments and tasks.

## Societal Impact

Our research conducted centers on the co-evolution of an RL agent’s morphology
and the training environments. This study aims to synchronize the instruction
of a control policy, morphology policy, and environment policy. The regulation
of the morphological development of RL agents has the potential to drive
progress in multiple domains, resulting in notable advantages for society. In
the domain of robotics, our investigations have the potential to facilitate
the advancement of adaptable and multifaceted robotic frameworks that can
proficiently traverse volatile surroundings, execute intricate operations, and
provide aid in perilous undertakings. The technological progressions possess
the capability to augment industrial procedures, ameliorate efficacy, and
mitigate hazards to human laborers across diverse domains.

Although the management of morphological evolution presents encouraging
progress, it is imperative to acknowledge potential hazards and apprehensions.
A fundamental issue of utmost importance pertains to the ethical and
responsible utilization of advanced RL agents. It is imperative to ensure that
the behavior of these agents aligns with ethical standards and legal
regulations as their morphology evolves through autonomous learning processes.
It is imperative to develop all-encompassing frameworks and protocols that
regulate the deployment of RL agents featuring dynamic morphologies, thereby
averting any potential instances of misapplication or detrimental outcomes.

## References

  * Baykal & Alterovitz (2017) Baykal, C. and Alterovitz, R.  Asymptotically optimal design of piecewise cylindrical robots using motion planning.  In _Robotics: Science and Systems_ , 2017.
  * Chen et al. (2020) Chen, T., He, Z., and Ciocarlie, M. T.  Hardware as policy: Mechanical and computational co-optimization using deep reinforcement learning.  In _CoRL_ , 2020.
  * Cheney et al. (2013) Cheney, N., MacCurdy, R., Clune, J., and Lipson, H.  Unshackling evolution: evolving soft robots with multiple materials and a powerful generative encoding.  In _GECCO ’13_ , 2013.
  * Cheney et al. (2018) Cheney, N., Bongard, J. C., SunSpiral, V., and Lipson, H.  Scalable co-optimization of morphology and control in embodied machines.  _Journal of The Royal Society Interface_ , 15, 2018.
  * Desai et al. (2017) Desai, R., Yuan, Y., and Coros, S.  Computational abstractions for interactive design of robotic devices.  _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1196–1203, 2017.
  * Exarchos et al. (2021) Exarchos, I., Jiang, Y., Yu, W., and Liu, C. K.  Policy transfer via kinematic domain randomization and adaptation.  _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 45–51, 2021.
  * Fey & Lenssen (2019) Fey, M. and Lenssen, J. E.  Fast graph representation learning with pytorch geometric.  _ArXiv_ , abs/1903.02428, 2019.
  * Finn et al. (2017) Finn, C., Abbeel, P., and Levine, S.  Model-agnostic meta-learning for fast adaptation of deep networks.  _ArXiv_ , abs/1703.03400, 2017.
  * Gupta et al. (2021) Gupta, A., Savarese, S., Ganguli, S., and Fei-Fei, L.  Embodied intelligence via learning and evolution.  _Nature Communications_ , 12, 2021.
  * Gupta et al. (2022) Gupta, A., Fan, L. J., Ganguli, S., and Fei-Fei, L.  Metamorph: Learning universal controllers with transformers.  _ArXiv_ , abs/2203.11931, 2022.
  * Gur et al. (2021) Gur, I., Jaques, N., Malta, K., Tiwari, M., Lee, H., and Faust, A.  Adversarial environment generation for learning to navigate the web.  _ArXiv_ , abs/2103.01991, 2021.
  * Ha et al. (2017) Ha, S., Coros, S., Alspach, A., Kim, J., and Yamane, K.  Joint optimization of robot design and motion parameters using the implicit function theorem.  _Robotics: Science and Systems XIII_ , 2017.
  * Hejna et al. (2021) Hejna, D. J., Abbeel, P., and Pinto, L.  Task-agnostic morphology evolution.  _ArXiv_ , abs/2102.13100, 2021.
  * Jaderberg et al. (2017) Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., Fernando, C., and Kavukcuoglu, K.  Population based training of neural networks.  _ArXiv_ , abs/1711.09846, 2017.
  * Jiang et al. (2021) Jiang, Y., Zhang, T., Ho, D., Bai, Y., Liu, C. K., Levine, S., and Tan, J.  Simgan: Hybrid simulator identification for domain adaptation via adversarial reinforcement learning.  _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 2884–2890, 2021.
  * Kirk et al. (2021) Kirk, R., Zhang, A., Grefenstette, E., and Rocktaschel, T.  A survey of generalisation in deep reinforcement learning.  _ArXiv_ , abs/2111.09794, 2021.
  * Kurin et al. (2021) Kurin, V., Igl, M., Rocktäschel, T., Boehmer, W., and Whiteson, S.  My body is a cage: the role of morphology in graph-based incompatible control.  _ArXiv_ , abs/2010.01856, 2021.
  * Liao et al. (2019) Liao, T., Wang, G., Yang, B., Lee, R. P., Pister, K. S. J., Levine, S., and Calandra, R.  Data-efficient learning of morphology and controller for a microrobot.  _2019 International Conference on Robotics and Automation (ICRA)_ , pp. 2488–2494, 2019.
  * Lillicrap et al. (2016) Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N. M. O., Erez, T., Tassa, Y., Silver, D., and Wierstra, D.  Continuous control with deep reinforcement learning.  _CoRR_ , abs/1509.02971, 2016.
  * Luck et al. (2019) Luck, K. S., Amor, H. B., and Calandra, R.  Data-efficient co-adaptation of morphology and behaviour with deep reinforcement learning.  In _CoRL_ , 2019.
  * Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D.  Human-level control through deep reinforcement learning.  _Nature_ , 518:529–533, 2015.
  * Morris et al. (2019) Morris, C., Ritzert, M., Fey, M., Hamilton, W. L., Lenssen, J. E., Rattan, G., and Grohe, M.  Weisfeiler and leman go neural: Higher-order graph neural networks.  _ArXiv_ , abs/1810.02244, 2019.
  * Narvekar et al. (2020) Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., and Stone, P.  Curriculum learning for reinforcement learning domains: A framework and survey.  _J. Mach. Learn. Res._ , 21:181:1–181:50, 2020.
  * Parker-Holder et al. (2022) Parker-Holder, J., Jiang, M., Dennis, M., Samvelyan, M., Foerster, J. N., Grefenstette, E., and Rocktaschel, T.  Evolving curricula with regret-based environment design.  In _International Conference on Machine Learning_ , 2022.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S.  Pytorch: An imperative style, high-performance deep learning library.  In _NeurIPS_ , 2019.
  * Portelas et al. (2019) Portelas, R., Colas, C., Hofmann, K., and Oudeyer, P.-Y.  Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments.  In _CoRL_ , 2019.
  * Salimans et al. (2017) Salimans, T., Ho, J., Chen, X., and Sutskever, I.  Evolution strategies as a scalable alternative to reinforcement learning.  _ArXiv_ , abs/1703.03864, 2017.
  * Schulman et al. (2016) Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P.  High-dimensional continuous control using generalized advantage estimation.  _CoRR_ , abs/1506.02438, 2016.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _ArXiv_ , abs/1707.06347, 2017.
  * Sims (1994) Sims, K.  Evolving 3d morphology and behavior by competition.  _Artificial Life_ , 1:353–372, 1994.
  * Trabucco et al. (2022) Trabucco, B., Phielipp, M., and Berseth, G.  Anymorph: Learning transferable polices by inferring agent morphology.  _ArXiv_ , abs/2206.12279, 2022.
  * Wang et al. (2019a) Wang, R., Lehman, J., Clune, J., and Stanley, K. O.  Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions.  _ArXiv_ , abs/1901.01753, 2019a.
  * Wang et al. (2020) Wang, R., Lehman, J., Rawal, A., Zhi, J., Li, Y., Clune, J., and Stanley, K. O.  Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions.  In _International Conference on Machine Learning_ , 2020.
  * Wang et al. (2018) Wang, T., Liao, R., Ba, J., and Fidler, S.  Nervenet: Learning structured policy with graph neural networks.  In _ICLR_ , 2018.
  * Wang et al. (2019b) Wang, T., Zhou, Y., Fidler, S., and Ba, J.  Neural graph evolution: Towards efficient automatic robot design.  _ArXiv_ , abs/1906.05370, 2019b.
  * Yu et al. (2019) Yu, W., Liu, C. K., and Turk, G.  Policy transfer with strategy optimization.  _ArXiv_ , abs/1810.05751, 2019.
  * Yuan et al. (2022) Yuan, Y., Song, Y., Luo, Z., Sun, W., and Kitani, K. M.  Transform2act: Learning a transform-and-control policy for efficient agent design.  _ArXiv_ , abs/2110.03659, 2022.
  * Zhao et al. (2020) Zhao, A., Xu, J., Konakovic-Lukovic, M., Hughes, J., Spielberg, A., Rus, D., and Matusik, W.  Robogrammar: graph grammar for terrain-optimized robot design.  _ACM Trans. Graph._ , 39:188:1–188:16, 2020.

##  Appendix A Implementation Details

###  A.1 Control Policy and Morphology Policy

we represent the agent’s skeleton by a graph $G=(V,A,E)$, where each node
$u\in V$ represents a joint and each edge $e\in E$ represents a bone
connecting two joints. Each joint $u$ is associated with a representation
vector $z_{u}$ storing attributes in $A$, e.g., bone length, motor strength,
size. The GNN takes $z_{u}$ as inputs and propagate messages across nodes on
the graph to produce a hidden state for each node. Both the control policy
$\pi$ and the morphology policy $\pi_{m}$ take each node’s hidden state as
their inputs and output their actions.

#### Implementation of morphology poicy $\pi_{m}$

MuJoCo agents are specified using XML strings, during the morphology modify
phase, we represent each agent’s skeleton structure as an XML string and
modify its content based on the morphology actions. The action of $\pi_{m}$
$\left\\{DelJoint,AddJoint\right\\}$ is applied to each node. Once the action
is taken and the skeleton is changed, the graph architecture changes and thus
the nodes’ hidden states changes. More specifically, An agent is initialized
with only one head joint and several (or zero) Lv1 body joints directly
connected to the head joint. In each iteration, the morphology policy will
traverse each joint (using GNN) to decide whether to add an Lv1 body joint
connected to the head joint, or an Lv2 body joint connected to a Lv1 body
joint (i.e., adding a new bone). Then, the morphology policy will further
traverse each joint to modify its attributes (bone length, motor strength,
size) on the morphology.

Modelling the morphology evolution as an MDP. In MECE, we aim to train an
agent $\pi_{m}$ that can design the morphology and modify it when adapting to
new environments. And training $\pi_{m}$ requires interactions with the MDP to
get reward of each modification. This is more efficient than conventional
morphology evolution methods that randomly search for a better morphology
because the policy learns to explore the space more efficiently through (1)
interactions with MDP using different morphologies; and (2) structural
constraint and correlation captured by the GNN.

###  A.2 Environment Details

In this part, we share further information about the three experiment
environments. A simple skeleton structure is used to initialize the agent.
Each joint of the agent is connected to its parent joint via a hinge. The
environment state of each joint contains its joint angle and velocity. In
addition to the height, phase, and global velocity, we also include additional
information for the root joint, such as the phase and height. Zero padding is
employed to ensure that the length of each joint state is same. The attribute
characteristic of each joint includes the bone vector, bone size, and motor
gear value. Using a loosely-defined attribute range, each attribute is
normalized within the interval $[-1,1]$.

#### 2d locomotion

The agent in this environment lives inside an $xz$-plane with a terrain
ground. Each joint of the agent is allowed to have a maximum of three child
joints. For the root joint, we add its height and 2D world velocity to the
environment state. The enviornment reward function is defined as:

$r_{t}=|p_{t+1}^{x}-p_{t}^{x}|/\delta t+1,$ (3)

where $p^{x}_{t}$ denotes the x-position of the agent and $\delta t=0.008$ is
the time step. An alive bonus of 1 is also used inside the reward. The episode
is terminated when the root height is below 1.4.

#### 3d locomotion

The agent in this environment lives freely in a 3D world with an uneven
ground. There are a fixed number of obstacles randomly scattered on the
surface. Each joint of the agent is allowed to have a maximum of three child
joints except for the root. For the root joint, we add its height and 3D world
velocity to the environment state. The reward function is defined as:

$r_{t}=|p_{t+1}^{x}-p_{t}^{x}|/\delta t-\omega\cdot\frac{1}{J}\sum_{u\in
V_{t}}\|a_{u,t}\|^{2},$ (4)

where $\omega=0.0001$ is a weighting factor for the control penalty term. $u$
denotes the each node of the skeleton structure $V_{t}$ of the agent at time-
step $t$. $J$ is the total number of agent’s joints, and the time step $\delta
t=0.04$.

#### Gap crosser

The agent in this environment lives inside an $xz$-plane. The terrain of this
environment includes a periodic gap. The height of the initial terrain is at
0.2. The agent must cross these gaps to move forward. Each joint of the agent
is allowed to have a maximum of three child joints. For the root joint, we add
its height, 2D world velocity, and a phase variable encoding the periodic
$x$-position of the agent to the environment state. The reward function is
defined as:

$r_{t}=|p_{t+1}^{x}-p_{t}^{x}|/\delta t+0.1,$ (5)

where the time step $\delta t=0.08$. An alive bonus of $0.1$ is used inside
the reward. The episode is terminated with the root height is below 1.5.

#### Implementation of environment policy $\pi_{e}$

Environment policy modify the training environment by taking actions to change
the environment parameters. For locomotion environments, we sample terrain
height maps using random Gaussian mixtures. There is an environment tuple of
two parameters to control the generation of terrains. The first environment
parameter is the maximum height of the terrain, which is limited to $2.4(2.7)$
in 2d (3d) locomotion. The second environment parameter is to control the
variance of the sampled environments, which is restricted in $[2.4,7.2]$ for
2d, and $[2.7,5.4]$ for 3d. In 3d locomotion, we have one more environment
parameter in the tuple to control the averaging spacing between the obstacles,
which is restricted in $[1.6,4.4]$.

##  Appendix B Additional Experiments Results

In this section, we report the changing environments and agent’s morphology
during the training in Fig. 5. The results show that when the evolved
morphology is relatively complex, the training environment focuses more on
complex cases than the randomly sampled training environments with environment
generator. On the other hand, the prevalence of easy environments
significantly decreases as morphology evolves (indicating a higher ability to
move in different environments).

Figure 5: This figure shows the changing of the agent’s morphology and
environments in the training phase. The results illustrate that with the
environment generator ($\pi_{e}$), the training environment focuses more on
complex cases (higher roughness). The presence of agents with more complex
morphologies is more noticeable among them. Thus, the environment generator
creates the training environment based on how the morphology changes during
training.

We report the mean and std of the environment roughness over training stages
and during test in the Tab. 2. As shown in the table: (1) the roughness (mean)
increases over time, automatically forming an easy-to-hard curriculum, which
leads to better generalization performance in the unfiromly sampled test
environments, which have relatively high average roughness and also high
variance; (2) the variance of roughness is very small at the beginning of
training, increases in the middle, and reduces a little later. Thereby, the
learning focuses on easier environments at first, encourages exploration for
better generalization in the middle, and keeps improving the agent in the most
challenging ones at last; (3) in addition to the automatic mean and std
scheduling, the increasing speed of roughness is also automatically adjusted
to be adaptive to training stages, e.g., it has a short warming-up stage at
the very beginning when the roughness keeps small and does not change too
much, followed by a critical learning period of quick growing (e.g., 15 - 35
million steps of Gap Crosser), and the growing gradually slows down in the
later stage when close to saturation (e.g., 30 - 50 million steps of 3d
locomotion).

Environment Steps (million) | 2d locomotion | 3d locomotion | Gap Crosser
---|---|---|---
$0-5$ | $2.493\pm 0.123$ | $2.765\pm 0.170$ | $0.541\pm 0.210$
$5-10$ | $2.585\pm 0.256$ | $2.907\pm 0.243$ | $0.978\pm 0.362$
$10-15$ | $3.232\pm 0.551$ | $3.325\pm 0.406$ | $1.128\pm 0.375$
$15-20$ | $3.870\pm 0.951$ | $3.587\pm 0.439$ | $1.306\pm 0.403$
$20-25$ | $4.522\pm 1.768$ | $3.604\pm 0.647$ | $1.775\pm 0.595$
$25-30$ | $5.019\pm 2.027$ | $3.831\pm 0.822$ | $2.393\pm 0.948$
$30-35$ | $5.864\pm 1.116$ | $4.312\pm 1.125$ | $3.028\pm 1.346$
$35-40$ | $6.288\pm 1.447$ | $4.775\pm 1.512$ | $3.242\pm 0.822$
$40-45$ | $6.420\pm 0.857$ | $4.879\pm 0.931$ | $3.445\pm 0.619$
$45-50$ | $6.725\pm 0.699$ | $4.921\pm 0.774$ | $3.612\pm 0.618$
Test Env | $5.425\pm 1.603$ | $4.368\pm 1.020$ | $2.813\pm 1.389$
Table 2: Mean and std of the environment roughness over training stages and
during test.

##  Appendix C Short evaluation window (SEW) in MECE

In Alg. 2, we modify the agent’s morphology and the training environment based
on the control policy’s learning progress. In light of the previous method
(Hejna et al., 2021), we construct a short evaluation window (SEW) on $\pi$ to
collect data for computing $r_{m}$ and $r_{e}$. In accordance with Sec. 3, we
should evaluate the present morphology in many settings for $r_{m}$ in Eq. 1
and several morphologies in the current training environment for $r_{e}$ in
Eq. 2. In this short evaluation phase, we therefore maintain some of the most
recent morphologies and training environments. We attempted to save several
combinations of morphology and number of contexts for this purpose. All
experimental and control data are provided under the assumption that the
optimal combination exists. We list all tried combination in Tab. 3.

##  Appendix D Hyperparameters and Training Procedures

In this section, we present the hyperparameters searched and used for MECE in
Tab. 3 and the hyperparameters for baseline in Tab. 4 and Tab. 5. All models
are implements by PyTorch (Paszke et al., 2019). For control policy and
morphology policy, we use the PyTorch Geometric package (Fey & Lenssen, 2019)
and GraphConv (Morris et al., 2019) as the GNN layer. When training the policy
with PPO, we adopt generalized advantage estimation (GAE) (Schulman et al.,
2016). For the baselines, we employ the released code of NGE and Transform2Act
to build our version.

For fair comparison, we employ the same GNN architecture for MECE and
baselines. In addition, we ensure that the number of simulation steps used for
optimization is the same for both the baseline methods and ours. MECE and
Transform2Act optimizes the policy with a batch size of 50000 for 1000 epochs,
totaling 50 million simulation steps. NGE employs a population of 20 agents,
and each agent is trained with a batch size of 20,000 for 125 generations,
totaling 50 million simulation steps.

Hyperparameter | Values Searched & Selected
---|---
Policy GNN Layer Type | GraphConv
Number of morphologies in SEW | $2,\mathbf{3},4$
Number of environments in SEW | $2,3,\mathbf{4},5$
GNN Size ($\pi_{m}$) | $(32,32,32),(64,64,64),(128,128,128),(\mathbf{256,256,256})$
GNN Size ($\pi$) | $(32,32,32),(\mathbf{64},\mathbf{64},\mathbf{64}),(128,128,128),(256,256,256)$
Hidden layers for $\pi_{e}$ network | $\mathbf{2},3,4$
Hidden units for $\pi_{e}$ network | $\mathbf{200},300,400$
Policy Learning Rate ($\pi$) | $\mathbf{5e-5},1\mathrm{e}-4,3\mathrm{e}-4$
Policy Learning Rate ($\pi_{m}$) | $\mathbf{5e-5},1\mathrm{e}-4,3\mathrm{e}-4$
Policy Learning Rate ($\pi_{e}$) | $\mathbf{3e-4},5\mathrm{e}-4,1\mathrm{e}-4$
Value GNN Layer Type | $\mathbf{Graph}\mathbf{Conv}$
Value Activation Function | Tanh
Value GNN Size | $(\mathbf{64},\mathbf{64,64}),(128,128,128),(256,256,256)$
Value MLP Size | $(256,256),(\mathbf{512},\mathbf{256}),(256,256,256)$
Value Learning Rate | $1\mathrm{e}-4,\mathbf{3e}-\mathbf{4}$
PPO clip $\epsilon$ Pize | $\mathbf{0.2}$
PPO Batch Size | $10000,20000,\mathbf{50000}$
PPO Minibatch Size | $512,\mathbf{2048}$
Num. of PPO Iterations Per Batch | $1,5,\mathbf{10}$
Num. of Training Epochs | $\mathbf{1000}$
Discount factor $\gamma$ | $\mathbf{0.95}$
GAE $\lambda$ | $\mathbf{0.99},0.995,0.997,0.999$
Table 3: Hyperparameters searched and used by MECE. The bold numbers among multiple values are the final selected ones. Hyperparameter | Values Searched & Selected
---|---
Num. of Skeleton Transforms $N_{\mathrm{s}}$ | $3,\mathbf{5},10$
Num. of Attribute Transforms $N_{\mathrm{z}}$ | $\mathbf{1},3,5$
Policy GNN Layer Type | GraphConv
JSMLP Activation Function | Tanh
GNN Size (Skeleton Transform) | $(64,64,64),(\mathbf{128,128,128}),(256,256,256)$
JSMLP Size (Skeleton Transform) | $(\mathbf{256,256}),(512,256),(256,256,256)$
GNN Size (Attribute Transform) | $(64,64,64),(\mathbf{128,128,128}),(256,256,256)$
JSMLP Size (Attribute Transform) | $(\mathbf{256,256}),(512,256),(256,256,256)$
GNN Size (Execution) | $(\mathbf{64},\mathbf{64},\mathbf{64}),(128,128,128),(256,256,256)$
JSMLP Size (Execution) | $(\mathbf{128},\mathbf{128}),(256,256),(512,256),(256,256,256)$
Diagonal Values of $\Sigma^{\mathrm{z}}$ | $1.0,0.04,\mathbf{0.01}$
Diagonal Values of $\Sigma^{\mathrm{e}}$ | $\mathbf{1.0},0.04,0.01$
Policy Learning Rate | $\mathbf{5e-5}$
Value GNN Layer Type | $\mathbf{Graph}\mathbf{Conv}$
Value Activation Function | Tanh
Value GNN Size | $(\mathbf{64},\mathbf{64,64}),(128,128,128),(256,256,256)$
Value MLP Size | $(256,256),(\mathbf{512},\mathbf{256}),(256,256,256)$
Value Learning Rate | $1\mathrm{e}-4,\mathbf{3e}-\mathbf{4}$
PPO clip $\epsilon$ Pize | $\mathbf{0.2}$
PPO Batch Size | $\mathbf{50000}$
PPO Minibatch Size | $\mathbf{2048}$
Num. of PPO Iterations Per Batch | $1,5,\mathbf{10}$
Num. of Training Epochs | $\mathbf{1000}$
Discount factor $\gamma$ | $\mathbf{0.95}$
GAE $\lambda$ | $\mathbf{0.995}$
Table 4: Hyperparameters searched and used by Transform2Act in our version. The bold numbers among multiple values are the final selected ones. Hyperparameter | Values Searched & Selected
---|---
Num. of Generations | $\mathbf{125}$
Agent Population Size | $10,\mathbf{20},50,100$
Elimination Rate | $\mathbf{0.15},0.2,0.3,0.4$
GNN Layer Type | GraphConv
MLP Activation | Tanh
Policy GNN Size | $(32,32,32),(\mathbf{64},\mathbf{64},\mathbf{64}),(128,128,128)$
Policy MLP Size | $(\mathbf{128},\mathbf{128}),(256,256),(512,256)$
Policy Log Standard Deviation | $\mathbf{0.0},-1.6$
Policy Learning Rate | $\mathbf{5e-5}$
Value GNN Size | $(32,32,32),(\mathbf{64},\mathbf{64},\mathbf{64}),(128,128,128)$
Value MLP Size | $(128,128),(256,256),(\mathbf{512},\mathbf{256})$
Value Learning Rate | $\mathbf{3\mathrm{e}-4}$
PPO clip $\epsilon$ | $\mathbf{0.2}$
PPO Batch Size | $\mathbf{20000},50000$
PPO Minibatch Size | $\mathbf{2048}$
Num. of PPO Iterations Per Batch | $\mathbf{10}$
Discount factor $\gamma$ | $0.99,\mathbf{0.995}$
GAE $\lambda$ | $\mathbf{0.95}$
Table 5: Hyperparameters searched and used by NGE in our version. The bold
numbers among multiple values are the final selected ones.