Experiments This section provides additional details on the experiments whose
basic setup and results are reported in sec:experiments in the main paper.

CorrAL feature selection dataset app:corral CorrAL is a synthetic dataset
consisting of six Boolean features (A0, A1, B0, B1, Irrelevant and Correlated)
and 160 samples. The feature names are rather self-explanatory as the target
variable is defined as

$\texttt{class}:=(\texttt{A0}\wedge\texttt{A1})\vee(\texttt{B0}\wedge\texttt{B1})\;.$

The remaining two features are not used—Irrelevant is uniformly random, and
Correlated matches the target label in 75% of all samples. john-1994, who
created the dataset as an illustration for the shortcomings of existing
feature selection algorithms, report that greedy selection mechanisms (both
top-down and bottom-up) have difficulties identifying the correct feature set.
We therefore test on the full dataset and see which features are selected as
part of the minimum-size coalitions with full explanation score.

As causal model for the dataset, we consider the input features to be
independent, while we allow all of them to potentially influence the target.
As threshold for coalitions with full explanation score we use $\alpha=0.998$
and evaluate all coalitions sizes $k\in\\{1,\dots,\text{\\#features}\\}$.

easily identifies the minimal full explanations in all cases without including
the irrelevant or only correlated feature into the coalitions. As an example,
let us consider the first row of the dataset, where all features are False. A
full explanation must consist of at least one of A0 and A1 (to explain why the
first conjunction is False), plus at least one of B0 and B1. Accordingly, the
minimum-size coalitions found by are the four combinations
$\\{\texttt{A0},\texttt{B0}\\}$, $\\{\texttt{A0},\texttt{B1}\\}$,
$\\{\texttt{A1},\texttt{B0}\\}$, and $\\{\texttt{A1},\texttt{B1}\\}$.

Explaining an ML model on the South German Credit dataset app:credit The South
German Credit (SGC) dataset gromping-2019 contains 1,000 loan applications
with 21 features, and the corresponding risk assessment (low-risk or high-
risk) as the binary target variable.

Model In this experiment, we are explaining the predictions of a trained ML
model on a given dataset. As the model, we use a simple random forest
classifier with the pipeline shown in fig:credit-pipeline, trained on the full
dataset with a train/test split of 90/10. The fitted model reaches an accuracy
of 96% for the training set and 80% for the test set. Note that we consider
explanations for this model – not for the data itself. Hence we ignore that
the model is not performing extremely well.

[width=0.6]credit/pipeline.png

Figure : Training pipeline for the model used in the experiment on the SGC
dataset.

###  SHAP values

To obtain the SHAP values for comparison, we make use of the shap package by
lundberg-2017. The use of their explainer is straight-forward; note that the
categorical features need to be converted into integer categories first for
shap to handle the input. In fig:credit-shap we show the SHAP output for the
sample with ID 500, the sample also shown in tab:sample-500 and mentioned in
ex:credit in the main paper.

Table : Feature values and SHAP values for sample 500 of the SGC dataset.  Feature | Value | SHAP value
---|---|---
credit_purpose | others | 0.034
housing | own | 0.020
personal_status | married/widowed | 0.020
other_installment_plans | bank | 0.019
age_groups | 0 | 0.017
sex | male | 0.017
credit_amount | 7308 | 0.015
checking_account_status | … ¡ 0 DM | 0.015
employed_since_years | unemployed | 0.014
job_status | mgmt/self-employed/highly qualified empl./officer | 0.014
property | real estate | 0.013
credit_history | no credits taken/all credits paid back duly | 0.009
telephone | True | 0.007
num_existing_credits | 1 | 0.006
savings | unknown/no savings account | 0.004
credit_duration_months | 10 | 0.003
installment_rate | 25 ¡= … ¡ 35 | 0.002
present_residence_since | ¿= 7 yrs | 0.001
other_debtors_guarantors | none | 0.001
foreign_worker | False | 0.000
num_people_liable_for | 0 to 2 | -0.002
|  |

Originally, SHAP values are designed to explain the difference between the
actual prediction and its expected value. Since the ratio of low-risk
($\texttt{credit\\_risk}=0$) and high-risk ($\texttt{credit\\_risk}=1$) loan
applications in the dataset is $\approx 30/70$, this difference is not always
the same. We therefore normalize the SHAP values such that the feature
contributions add up to 1 for each sample. This does not change the relative
importance of the features, but allows for a better comparison across low-risk
and high-risk samples.

[width=]credit/shap.png

Figure : SHAP values for sample 500 of the SGC dataset.

###  Comparison

We compare the stability of the prediction under three different
randomizations, showing the probability that the prediction remains unchanged
for any number of randomized features. The underlying reasoning is: The better
an explanation, the more stable the model prediction when the explanation
holds (i.e., when the respective feature values are fixed). Therefore, we
start with all features fixed to their observed values and successively
randomize more and more features according to their joint distribution in the
dataset, observing when the prediction stability drops. The three
randomizations only differ in the _order_ of features:

Randomize in ascending order of SHAP values

The features are ordered ascendingly by their SHAP value. This is the order in
which the prediction can be expected to stay stable as long as possible:
Randomizing features with a SHAP value $<0$ means that the _negative_
influence of these features is removed. The features with the highest
(positive) SHAP value, on the other hand, are kept fixed until much longer,
such that their stabilizing influence can be used.

Randomize non-coalition features first

The features of the minimum-size full explanation identified by are randomized
_last_. By doing so, we see whether the prediction is determined by the fixed
values of this feature coalition. The order of the remaining features is
chosen as above to maximize comparability.

Randomize in random order

As a baseline, we randomize the features in random order to see how fast the
prediction stability drops when there are no special features.

In order to obtain the interventional distributions required in , we follow
the reasoning of janzing-2020 and consider the real world to be a confounder
of the input features of the ML model, which therefore do not have direct
causal links among each other. As a consequence, we can intervene directly on
the input features by simply fixing the values of the features belonging to
the considered coalition while sampling the remaining features from their
(joint) marginal distribution. Further, we choose a threshold of
$\alpha=0.9998$ to identify coalitions with (almost) full explanations while
allowing for numerical errors. Additionally, we set the maximum coalition size
to $k=4$ and stop the procedure if we do not find a coalition exceeding the
threshold at this size.

###  Result

Out of the $1,000$ samples in the credit dataset, $426$ have simple
explanations as defined above (most of them have multiple such explanations,
such that the mean and quantiles in fig:credit are composed of 1,803 data
points). On average, the cumulative normalized SHAP value of the minimum-size
full explanations is $53.2\%$. Analyzing fig:credit, we see that, on average,
and SHAP show a similar performance, although ’ performance for the solid-line
sample (sample 500) only drops for the last 3 features while SHAP decreases a
few fetures earlier. This is due to the fact that, for this sample, a
3-feature coalition is actually sufficient to reach an explanation score of
$\approx 1$. Grouping the samples by the size $k$ of their minimum-size full
explanation coalitions, we would see that the stability of is very much
determined by this value, i.e., the prediction stability always drops after
$|\bm{X}|-k$ features. Such a pattern cannot be observed for SHAP. Using a
random order for the randomization of features, the drop in the prediction
stability starts much earlier, usually after $\approx 10$ features. Note that
there is a small number of outliers even after one or two features, causing
the mean to decrease immediately, while the 5-95% range is still at 1.

###  Computation

The experiment was executed on a 10-core MacBook Pro M2. The calculation of
the SHAP values took $\approx 5$ minutes, while the identification of all
minimum-size coalitions up to size $k=4$ took $\approx 6$ hours.

##  Identifying root causes in a cloud computing application

We consider a cloud computing application in which services call each other to
produce a result which is returned to the user at the target service $Y$.
Services can either be stand-alone (like a database or a random number
generator) or call other services and transform their input to produce an
output. When we observe an error at the target service $Y$, we want to know
which service(s) caused this error.

###  Data generation

The causal network of the Cloud Computing Application experiment is based on a
realistic service architecture shown in fig:cloud-services. On top of the
causal dependencies, we have defined parameters $p_{i}$ and $t_{i}$ such that
target errors are a rare, but not impossible phenomenon. This architecture
translates to the causal network shown in fig:cloud-network with services
$X_{1},...,X_{9},Y$, by omitting the _Website_ service and using _www_ as the
target variable. We generate synthetic data representing error counts for the
different services according to X_i=Λ_i ∨(∑_j ∈iX_j ≥t_i) , where Λ_i
∼Bern(p_i) . We generated $10$ million samples, of which $15,373$ samples
produce a target error (i.e., $Y=1$). tab:cloud-number-of-samples shows the
number of elements in the sample set, grouped by the number of original
errors. We see that fewer than two original errors never lead to a target
error.

[width=0.8]cloud/services.pdf

Figure : Cloud service architecture with non-trivial causal dependencies.

[width=0.8]cloud/network.pdf

Figure : Causal graph of the cloud computing application in fig:cloud-services. The parameters $p_{i}$ denote the probability that a node produces an error itself, and $t_{i}$ is the threshold number of parent services that have to encounter an error to also cause the error to propagate to $X_{i}$.  Table : Number of generated samples, grouped by the number of original errors, i.e., noise nodes with value 1.  # errors | # samples with $Y=0$ | # samples with $Y=1$
---|---|---
0 | 8,416,434 | 0
1 | 1,471,666 | 0
2 | 93,532 | 14,009
3 | 2,953 | 1,283
4 | 42 | 78
5 | 0 | 3
|  |

###  Mapping from root causes to minimal coalitions

From the contribution scores produced by the RCA methods
[see][]budhathoki-2022, we find minimal coalitions with the following
procedure:

  1. 1.

Calculate the contribution score $\Phi(i)$ for all features $i\in I$.

  2. 2.

Sort the features by descending score and normalize them so that
$\sum{\Phi_{i}}=1$.

  3. 3.

Select feature via a _cumulative threshold_ (take features in the above order
until the sum of their contribution scores reaches a threshold
$\theta_{c}=0.95$), or via an _individual threshold_ (take all features with
$\Phi(i)\geq\theta_{i}=0.15$).

The thresholds $\theta_{c}$ and $\theta_{i}$ were found through hyperparameter
tuning such that these RCA methods achieve the best performance when compared
to the ground truth. By applying this procedure, we get a coalition of
features from all baseline methods.

###  Traversal RCA

The traversal method for RCA is based on a very simple idea: A node in the
causal graph is identified as a root cause if (a) it shows an error (we say
the node is _anomalous_), and (b) none of its causal parents is anomalous.
This can be directly inferred from an observation, outputting a set of root
causes. A slightly more sophisticated way is to additionally require for a
root cause to have a path of anomalous nodes to the target, therefore
backtracking from the target to possible root causes (called _Backtracking
Traversal RCA_ in fig:cloud-traversal). This can be implemented as a breadth-
first search from the target node which stops at non-error nodes and then
returns the leaves of the search tree. The effect is that a node is only
identified as a root cause if there is a direct error propagation path from
this node to the target, thus reducing false positives.

[t]0.47 [width=]cloud/traversal-accuracy.pdf [t]0.47 [width=]cloud/traversal-
confusion.pdf

Figure : Accuracy of identifying root causes for different numbers of injected
errors.  Figure : True positives, false positives, and false negatives for the
cases with at least 3 injected errors.  Figure : Comparison of Simple
Traversal RCA and Backtracking Traversal RCA for the Cloud experiment. The
accuracy is unaffected, while the false positive rate decreases when using
backtracking.

For completeness, we show in fig:cloud-traversal a comparison between these
two traversal RCA methods. Although the false positive rate, as expected, is
lower for Backtracking Traversal RCA, the accuracy remains the same: The main
failure mode of Traversal RCA is the implicit assumption that a single
anomalous parent is enough to prevent a node from being a root cause. This
weakness is present in both versions of the algorithm, as witnessed by the
high rate of false negatives.

###  Computation

Most of the experiment and analysis was executed on a 10-core MacBook Pro M2.
Identifying all minimum-size coalitions for the 15,000 target error samples
took approximately one hour; the traversal algorithms did not take any
substantial amount of time ($<20$ seconds). However, calculating the anomaly
scores for the RCA approaches using dowhy.gcm bloebaum-2022 turned out to be
much slower, such that this task was transferred to 10 EC2 instances and
finished after about two days.


# Efficient Hyperparameter Optimization for
Physics-based Character Animation

Zeshi Yang  Simon Fraser UniversityCanada<EMAIL_ADDRESS>and  Zhiqi Yin  Simon
Fraser UniversityCanada<EMAIL_ADDRESS>
###### Abstract.

Physics-based character animation has seen significant advances in recent
years with the adoption of Deep Reinforcement Learning (DRL). However, DRL-
based learning methods are usually computationally expensive and their
performance crucially depends on the choice of hyperparameters. Tuning
hyperparameters for these methods often requires repetitive training of
control policies, which is even more computationally prohibitive. In this
work, we propose a novel Curriculum-based Multi-Fidelity Bayesian Optimization
framework (CMFBO) for efficient hyperparameter optimization of DRL-based
character control systems. Using curriculum-based task difficulty as fidelity
criterion, our method improves searching efficiency by gradually pruning
search space through evaluation on easier motor skill tasks. We evaluate our
method on two physics-based character control tasks: character morphology
optimization and hyperparameter tuning of DeepMimic. Our algorithm
significantly outperforms state-of-the-art hyperparameter optimization methods
applicable for physics-based character animation. In particular, we show that
hyperparameters optimized through our algorithm result in at least 5x
efficiency gain comparing to author-released settings in DeepMimic.

Physics-based Character Animation, Bayesian Optimization, Reinforcement
Learning, Curriculum Learning

††copyright: acmlicensed††journal: PACMCGIT††journalyear: 2021††journalvolume:
4††journalnumber: 1††publicationmonth: 5††price: 15.00††doi:
10.1145/3451254††submissionid: 2††ccs: Computing methodologies Animation††ccs:
Computing methodologies Physical simulation††ccs: Theory of computation
Reinforcement learning††ccs: Theory of computation Bayesian analysis

##  1\. INTRODUCTION

Physics-based character animation has made significant progresses recently,
especially with the application of Deep Reinforcement Learning (DRL)
algorithms (Peng et al., 2018a; Bergamin et al., 2019; Won et al., 2020; Park
et al., 2019). For example, DeepMimic-style neural network controllers are
able to synthesize diverse and robust high-quality motor skills (Peng et al.,
2018a). Despite the demonstrated impressive performance, it is usually quite
hard for a novice graduate student to reproduce the performance of such
systems, without knowing the exact value of each hyperparameter involved.
Therefore, more and more authors have released their code for better
reproducibility. However, a single change of one hyperparameter may totally
ruin the performance, and sometimes even the convergence, of such learning
algorithms. How the original authors found the working set of hyperparameters
remains as an art and mystery. Improving upon prior work, even with released
code, thus remains challenging, as any modification to the training algorithm
may require a new set of hyperparameters to work well.

Automatic hyperparameter optimization is thus in great need. Hyperparameters,
in the narrow sense, refer to values that are used to control the learning
process in machine learning algorithms. In contrast, regular parameters are
derived or optimized during the training process. In this paper, we refer to
all parameters external to a learning algorithm that need to be determined
prior to the learning as hyperparameters. For example, morphology parameters
of a virtual character in motor learning are also hyperparameters. To date, it
is a common practice of the field to manually test and select hyperparameters
for various character control algorithms. Behind the scenes, maybe simple grid
search or random search algorithms are used for semi-automatically choosing
hyperparameters. However, in physics-based character control, better search
schemes are needed to handle two key challenges: First, the evaluation of new
hyperparameters typically involves re-learning of the controllers from
scratch, which is usually an expensive black-box function. Second, the number
of hyperparameters can also be large and results in the curse of
dimensionality.

Bayesian optimization (BO) is a promising candidate for hyperparameter
optimizations for physics-based character animation problems. BO is a
sequential design strategy for global optimization of expensive-to-evaluate
black-box functions that do not assume any functional forms (Jones et al.,
1998; Srinivas et al., 2010; Kandasamy et al., 2017). Traditional BO only
evaluates the expensive black-box objective functions themselves (Jones et
al., 1998; Srinivas et al., 2010). We refer to such algorithms as Single
Fidelity BO (SFBO) in this paper. For many physics-based character animation
applications, however, SFBO is inefficient due to the extremely high cost of
function evaluations that rely on physics-based character simulation and motor
learning. Multi-fidelity BO (MFBO) accelerates the optimization by using cheap
approximations of the objective functions in early optimization stages
(Kandasamy et al., 2017; Klein et al., 2017; Song et al., 2019; Swersky et
al., 2013). Such MFBO algorithms typically employ fewer training iterations on
smaller datasets as low-fidelity cheap approximations to the original
objective functions, and work well for supervised learning tasks.

For physics-based character animation, however, existing MFBO methods do not
work well. We analyze in section 3.2 and 3.3 that training iterations are not
a good fidelity criterion for physics-based character control, and multi-
fidelity objective does not have desired properties across fidelity dimension
required by existing MFBO methods like BOCA (Kandasamy et al., 2017) and
FABOLAS (Klein et al., 2017). Therefore, we propose a novel algorithm CMFBO:
Curriculum-based Multi-Fidelity Bayesian Optimization for efficient
hyperparameter optimization of DRL-based character control systems. CMFBO
employs easier motor skill learning tasks as low-fidelity optimization
objectives. Task difficulties are organized and scheduled by a curriculum
(Bengio et al., 2009). For example, a curriculum that gradually increases the
training episode length can help the character perform longer and longer
skills (Peng et al., 2018a; Kostrikov, 2018); and a curriculum that gradually
reduces the hand-of-God assistance forces can help the character learn to
locomote (Yu et al., 2018). Easier tasks are faster to evaluate enabling
efficient search space pruning. Control policies learned at easier tasks could
be transferred to harder tasks to further reduce evaluation cost.
Hyperparameters optimized by CMFBO may significantly outperform those tuned by
experts and optimized by state-of-the-art methods. For instance, at least 5x
efficiency gain is obtained with our optimized hyperparameters on DeepMimic
(Peng et al., 2018a), compared to author-released settings.

To summarize, we

  * •

introduce Bayesian Optimization into physics-based character animation for
principled hyperparameter optimization;

  * •

propose an efficient algorithm CMFBO for hyperparameter optimization of
challenging motor learning tasks based on deep reinforcement learning;

  * •

systematically compare and evaluate SFBO, state-of-the-art MFBO, and CMFBO on
two physics-based character animation problems: morphology optimization, and
automatic tuning of DeepMimic training hyperparameters.

##  2\. RELATED WORK

###  2.1. Physics-based Character Animation

Synthesizing natural human motions is a long-standing challenge in computer
graphics. We classify existing methods roughly into three categories. First,
manually designed controllers, which usually employ finite state machines
(FSMs) and heuristic feedback rules. Human insights and domain knowledge are
usually involved to design and tune the parameters (Hodgins et al., 1995; Yin
et al., 2007; Jain et al., 2009; Wang et al., 2009; Coros et al., 2010; De
Lasa et al., 2010; Coros et al., 2011; Wang et al., 2012; Geijtenbeek et al.,
2013; Felis and Mombaur, 2016). Second, model-based trajectory optimization,
where equations of motions are enforced as optimization constraints. Designing
of an often sophisticated optimization objective function is generally
required along with weights tuning for each term (Mordatch et al., 2012; Tassa
et al., 2012; Wampler et al., 2014; Hämäläinen et al., 2015). Recently, model-
free DRL-based control methods have been demonstrated to reproduce high-
quality complex and robust motor skills (Peng et al., 2015; Lillicrap et al.,
2016; Peng et al., 2016, 2017; Heess et al., 2017; Peng et al., 2018a; Peng et
al., 2018b; Yu et al., 2018; Bergamin et al., 2019; Park et al., 2019; Won and
Lee, 2019; Won et al., 2020; Luo et al., 2020; Merel et al., 2020). DRL-based
methods require designing and tuning of a reward function, as well as related
hyperparameters of DRL algorithms themselves. In this paper, We focus on
hyperparameter optimization for recent DRL-based methods, although our
framework is general enough to be applicable to other categories of methods as
well.

###  2.2. Hyperparameter Optimization

####  2.2.1. Parameter Optimization in Computer Graphics

Parameter tuning and optimization is a common task in computer graphics, such
as weights for SIMBICON-type feedback controllers (Yin et al., 2007; Wang et
al., 2009). Multiple derivative-free optimization algorithms, such as
Covariance Matrix Adaptation (CMA) (Hansen, 2006; Wang et al., 2009) and
Bayesian optimization, have been used to optimize such parameters where the
objective functions are black-box functions and no derivative information is
available. Some methods further combine BO with user evaluations to achieve
desired visual effects, such as bidirectional reflectance distribution
function (BRDF) design (Brochu et al., 2007) and smoke animation tuning
(Brochu et al., 2010). More recently, BO is applied to low-dimensional
subproblems of various graphics applications (Koyama et al., 2017, 2020), such
as color enhancement, and human body geometric modelling (Loper et al., 2015).

We also adopt a Bayesian optimization framework, but for hyperparameter
optimization. To the best of our knowledge, the only previous work on
hyperparameter optimization in graphics is (Tseng et al., 2019), where
parameters of image processing hardware, such as threshold values in the
denoising module, are optimized with a non-Bayesian approach. Hereafter we
will focus on hyperparameter optimization literature from machine learning and
robotics.

####  2.2.2. Hyperparameter Optimization for DRL

State-of-the-art motor learning methods are usually based on DRL. For example,
DeepMimic (Peng et al., 2018a) can produce robust and diverse high-quality
controllers by imitating motion captured reference motions. However, its
performance crucially depends on the choice of its hyperparameters. Minor
changes to one single hyperparameter may result in slow convergence or even
training failure. But so far, no publications on physics-based character
animation have explicitly discussed the issue of hyperparameter tuning or
optimization. In machine learning, hyperparameter optimization for simple
benchmark systems, such as a cartpole model in OpenAI gym (Brockman et al.,
2016), with standard DRL algorithms, such as A2C (Mnih et al., 2016), has been
investigated (Nguyen and Osborne, 2019; Nguyen et al., 2020). However, in
physics-based character animation, we study the control of high dimensional
character models, which is a much more expensive and challenging task
requiring specialized DRL algorithms such as DeepMimic to achieve natural-
looking skills.

####  2.2.3. Morphology Design

Morphology design and optimization is a problem in both computer animation and
robotics (Park and Asada, 1994; Sims, 1994; Pil and Asada, 1996; Lipson and
Pollack, 2000; Paul and Bongard, 2001; Bongard, 2011; Villarreal-Cervantes et
al., 2012; Geijtenbeek et al., 2013; Agrawal et al., 2014; Spielberg et al.,
2017; Ha et al., 2017; Wang et al., 2018; Won and Lee, 2019; Liao et al.,
2019; Huang et al., 2020; Hu et al., 2020; Ma et al., 2021). Traditional
model-based methods require accurate dynamic models, and only work for
specific types of control algorithms such as trajectory optimization
(Spielberg et al., 2017; Ha et al., 2017). Some recent morphology design
methods work with DRL-based control (Schaff et al., 2019; Ha, 2019; Luck et
al., 2020). These methods guide optimization in morphology space based on
gradient estimation (Schaff et al., 2019; Ha, 2019), or performance prediction
of unseen morphology through a learned value network (Luck et al., 2020). We
note that none of these methods address the issue of expensive evaluation per
design. With effective search space pruning through cheap evaluation on easier
tasks, our method significantly outperforms them in terms of sample
efficiency.

###  2.3. Bayesian Optimization

Bayesian Optimization is a class of methods for expensive black-box function
optimization. Since no gradient information is available, functions are
optimized purely through evaluations. A Bayesian statistical model, usually a
Gaussian Process (Rasmussen, 2003), is maintained to estimate value of the
objective function along with the uncertainty. An acquisition function is then
repeatedly applied to query and evaluate the most promising and informative
region based on existing estimation.

With the advancement of deep learning, there is a great appeal for automatic
hyperparameter optimization of deep learning models. BO demonstrates its
potential on such tasks for its promising sample efficiency (Snoek et al.,
2012; Snoek et al., 2015; Nguyen and Osborne, 2019). Recently many MFBO
methods are proposed to further improve efficiency of hyperparameter
optimization for supervised learning tasks (Swersky et al., 2013, 2014;
Kandasamy et al., 2016, 2017; Klein et al., 2017; Takeno et al., 2019; Song et
al., 2019) mainly by utilizing cheap approximations of the objective, such as
neural network validation loss on smaller datasets with fewer training
iterations.

However, most of these methods cannot scale well to motion control tasks. Due
to stochasticity and complexity of DRL training, existing fidelity criterion
such as the number of training iterations mislead the optimization a lot. Low-
fidelity evaluations through early stopping usually provide false estimation
on the relative performance of hyperparameters. Besides the fidelity criterion
issue, existing MFBO acquisition functions have strong assumptions on the
shape of multi-fidelity objective function along the fidelity dimension. For
example, BOCA (Kandasamy et al., 2017) assumes a flat fidelity dimension while
FABOLAS (Klein et al., 2017) assumes it to satisfy quadratic form. These
simplified assumptions are no longer valid in the complex DRL settings. We
detail the analysis and how we resolve the problems with our proposed
curriculum-based fidelity and progressive acquisition function in section 3.2
and 3.3.

###  2.4. Curriculum Learning

Curriculum learning (CL) is a learning method where task difficulty gradually
increases during training(Bengio et al., 2009). Easier tasks are cheaper to
accomplish and serve as good initial solutions for more difficult tasks. CL
has been widely used in character animation to improve the performance and
efficiency of motor learning tasks (Van de Panne and Lamouret, 1995; Yin et
al., 2008; Wu and Popović, 2010; Karpathy and Van De Panne, 2012). (Peng et
al., 2018a; Kostrikov, 2018) employ a time-based curriculum with increasing
episode length to help the character perform longer and longer skills. (Yu et
al., 2018) applies hand-of-God assistance forces at robot torso and decreases
the force gradually during training to encourage the emergence of natural
gaits. More recently, (Xie et al., 2020) uses CL to demonstrate successful
training of stepping-stone locomotion controllers.

##  3\. METHOD

We formulate hyperparameter optimization in a physics-based animation system
as a black-box function optimization problem

(1) $\mathop{\arg\max}\limits_{x\in\mathcal{A}}f(x)$

where $\mathcal{A}$ is the hyperparameter space.
$f(\cdot):\mathcal{A}\to\mathcal{R}$ is the objective function, i.e.
performance of control policy given hyperparameter $x$. The evaluation of
$f(\cdot)$ involves DRL training, thus is costly, noisy and cannot be computed
in closed forms.

Figure 1 illustrates a conceptual overview of our approach based on MFBO. We
first define our multi-fidelity objective function based on a curriculum
(section 3.2). During optimization, we use lower-fidelity cheap approximations
to locate the promising region of hyperparameters through the proposed
progressive acquisition function (section 3.3). Policies learned at easier
tasks are transferred to difficult tasks for more efficient evaluation
(section 3.4).

Figure 1. Conceptual illustration of CMFBO

###  3.1. Background

Figure 2. An example of applying BO to a one-dimensional function. Green
dotted curve represents the unknown black-box function. Red points represent
the queried points. Red curve along with the rosy shaded region indicate the
predicted mean and $95\%$ confidence interval. Blue curve is the acquisition
function (It is scaled for the convenience of visualization). Blue downside
arrows indicate the maximum of the acquisition function which is the point to
be queried next.

In this section we briefly review the technique of general Bayesian
optimization involved in our approach. Given a black-box function $f(x)$, BO
finds its maxima through repeated function evaluations. Since evaluating
$f(x)$ could be expensive, BO is designed to minimize the number of function
evaluations by querying the most promising and informative points. Given a set
of current observations $D_{t}=\\{(x_{i},y_{i})\\}_{i=1}^{t}$, where $y_{i}$
is a noisy measurement of $f(x_{i})$, the acquisition function
$a(x,D_{t}):\mathcal{A}\to\mathbb{R}$ quantifies the utility of an arbitrary
point. Maximizing the acquisition function will give us the point most worth
trying next.

The acquisition function is designed for finding candidate points with both
large values and rich information. Querying a point close to existing ones in
$D_{t}$ is less informative. Gaussian Process Upper Confidence Bound (GP-UCB)
(Srinivas et al., 2010) is a popular acquisition function defined as:

(2) $a(x,D_{t})=\mu_{t}(x)+\beta^{\frac{1}{2}}\sigma_{t}(x)$

where $\mu_{t}(x)$ and $\sigma_{t}(x)$ are approximated posterior mean value
and standard deviation of $f(x)$ respectively. $\mu_{t}(\cdot)$ favors
candidates which are likely to have large values. $\sigma_{t}(\cdot)$
encourages querying informative points with high uncertainty. $\beta$ enables
a trade-off between exploitation and exploration.

Closed-form estimations for $\mu_{t}(\cdot)$ and $\sigma_{t}(\cdot)$ are
available through a Gaussian Process (GP) (Rasmussen, 2003) surrogate model of
the objective function trained on $D_{t}$. A GP contains a prior mean function
$m(x)$ and a kernel function $k(x,x^{\prime})$. $m(x)$ encodes our prior
belief of the objective function value. Kernel function $k(x,x^{\prime})$
measures correlations between $f(x)$ and $f(x^{\prime})$. Given $m(\cdot)$,
$k(\cdot,\cdot)$ and existing observations $D_{t}$, $\mu_{t}$ and $\sigma_{t}$
could be computed as:

(3) $\begin{gathered}\mu_{t}(x)=k(x,X)(K+\eta^{2}I)^{-1}Y\\\
\sigma_{t}^{2}(x)=k(x,x)+\eta^{2}-k(x,X)(K+\eta^{2}I)^{-1}k(X,x)\end{gathered}$

where $Y\in\mathbb{R}^{t},Y_{i}=y_{i}$; $X\in\mathbb{R}^{t\times
d},X_{i}=x_{i}$; $K\in\mathbb{R}^{t\times t},K_{i,j}=k(x_{i},x_{j})$;
$k(x,X)=(k(x,x_{1}),k(x,x_{2}),...k(x,x_{t}))$; $\eta$ is the standard
deviation of the observation noise.

Many choices of kernel functions exist, including Square Exponential kernel
and Matern kernel. In this work we adopt the Square Exponential kernel defined
as:

(4)
$k_{SE}(x,x^{\prime})=\sigma_{f}^{2}exp(-\frac{1}{2}(x-x^{\prime})^{T}\Lambda^{-1}(x-x^{\prime}))$

$\Lambda=diag(\lambda_{1},\lambda_{2},...\lambda{t})$, where $\lambda_{i}$ is
the length scale of the $i$-th component of inputs. $f(\cdot)$ is flat across
the $i$-th component of the input when $\lambda_{i}$ is large.

Figure 2 shows a visual illustration of applying BO to a one-dimensional
function. We refer interested readers to the second chapter of (Rasmussen,
2003) for more in-depth reviews.

###  3.2. Curriculum-based Multi-Fidelity Functions

The main challenge of hyperparameter optimization for physics-based character
control tasks comes from the fact that performance evaluation function $f(x)$
given hyperparameter $x$ is highly computationally expensive. Even if BO is
designed for black-box function optimization with minimum samples, the entire
optimization procedure still requires a prohibitive amount of computational
resources.

We propose to solve this problem with a multi-fidelity approach on top of
Bayesian optimization, where low-fidelity cheap evaluations are utilized for
efficient search space pruning. This procedure crucially depends on an
accurate fidelity criterion being able to distinguish good hyperparameters
from bad ones with lower-fidelity approximations. To the best of our
knowledge, number of training iterations is the only fidelity criterion from
existing literature applicable to our problem. Though number of training
iterations shows its effectiveness for supervised learning tasks (Kandasamy et
al., 2017; Klein et al., 2017; Snoek et al., 2012), performance estimated
through early stopping can be deceptive for challenging DRL character control
tasks. Early stages for DRL training can be noisy (Haarnoja et al., 2018;
Fujimoto et al., 2018), and good hyperparameters do not always guarantee quick
convergence (Nguyen et al., 2020). We illustrate this problem in the context
of finding optimal morphology hyperparameters for character locomotion. We
show the performance of three morphology designs in figure 3(a), where some
good morphology designs show worse performance in early training stages.

To mitigate this problem, we propose the use of curriculum-based task
difficulty as a new fidelity criterion tailored to physics-based character
control settings. Task difficulty can be parameterized by a single continuous
scalar variable $z$. Multi-fidelity objective function is therefore defined to
be $f(x,z)$. Note that $f(x,z_{max})$ is the original objective function.
$f(x,z)$ is more accurate and requires more computational resources as $z$
gets closer to $z_{max}$. Similar to single fidelity BO, we use GP to model
multi-fidelity function $f(x,z)$. The kernel function generalizes to:

(5) $k((x,z),(x^{\prime},z^{\prime}))=k_{SE}(x,x^{\prime})\
k_{SE}(z,z^{\prime})$

The factorized kernel adopts the multiplication form based on the fact that:
if $x$ and $z$ are close to $x^{\prime}$ and $z^{\prime}$ respectively,
$f(x,z)$ correlates strongly with $f(x^{\prime},z^{\prime})$.

We validate the effectiveness of using task difficulty as fidelity criterion
through an example shown in figure 3(b), where ten morphology hyperparameters
are evaluated on two different levels of task difficulty parameterized by the
strength of virtual hand-of-God assistance forces. With fidelity criterion
defined with task difficulty, relative performance of hyperparameters across
different fidelity levels are well preserved. We detail our specific
curriculum settings for different tasks in section 4.

(a)

(b)

Figure 3. (a): Learning curves of locomotion controllers with different
morphology designs. (b): Performance of ten morphology designs on two
difficulty levels. Task difficulty is determined by the strength of hand-of-
God assistance forces parameterized by proportional gain kp of the assistance
stable PD controller.

###  3.3. Progressive Acquisition Function

Given a properly designed multi-fidelity objective function, an effective
acquisition function is still in need to achieve efficient hyperparameter
optimization in physics-based character control tasks. In each iteration,
multi-fidelity acquisition function $a((x,z),D_{t})$ determines which point
$x$ in hyperparameter space is going to be evaluated next, and which fidelity
level $z$ the evaluation is going to be performed on. An effective multi-
fidelity acquisition function is expected to generate queries such that the
original objective can be optimized with much fewer samples by inferring
optimality information through low-fidelity cheap approximations.

Existing MFBO methods like BOCA (Kandasamy et al., 2017) and FABOLAS (Klein et
al., 2017) require a well-fitted surrogate model for accurate estimation of
original objective value from lower-fidelity approximations. Mathematical
assumptions on the shape of fidelity dimension are usually necessary to
achieve this. For example, BOCA assumes the fidelity dimension is flat while
FABOLAS assumes it to satisfy quadratic form. However, due to stochasticity
and complexity of DRL training, performance difference of hyperparameters on
different fidelity levels can be unpredictable, as shown in figure 3(b). A GP
trained on such DRL performance data will learn a small length scale, which
degrades the accuracy of high fidelity value prediction through a reasonable
number of lower-fidelity samples.

We attempt this problem with a different approach. Instead of relying on low
fidelity samples to infer high-fidelity function value, we propose to use them
simply for locating promising regions. The motivation is based on the key
observation that though performance difference of hyperparameters on different
fidelity levels can be unpredictable, optimal regions of objective function on
different fidelity levels highly overlap, as shown in figure 3(b). Search
space for a high-fidelity objective can therefore be greatly pruned by simply
exploring around optimal regions estimated at lower fidelity levels.

Specifically, we propose a novel progressive acquisition function, which
progressively conducts optimization from low fidelity to high fidelity levels.
At iteration $t$, target fidelity $z_{t}$ is chosen to be the least expensive
while most informative one until we reach the highest. This is performed
through an iterative procedure. Initializing $z_{t}$ with $z_{min}$, we
repeatedly increase $z_{t}$ by a small step $\alpha$ and re-compute best
candidate point $x_{t}$ for new $z_{t}$. The procedure terminates when our
model shows insufficient knowledge at $(x_{t},z_{t})$ with uncertainty
$\sigma_{t}(x_{t},z_{t})$ exceeding threshold $\epsilon$. In our
implementation, we choose $\alpha$ to be equal to the length scale of the
fidelity kernel $l_{z}$, since function values from GP surrogate model
strongly correlate along fidelity dimension within $l_{z}$ proximity. $z_{t}$
should be increased only when we have sufficient knowledge at $(x_{t},z_{t})$.
$\epsilon$ can therefore be set conservatively to a small value negligible
comparing to the objective function value range in promising regions. In our
experiments we set $\epsilon$ to $0.02$.

Given a specified fidelity level $z_{t}$, the best candidate point $x_{t}$ is
chosen by optimizing a variant of the upper confidence bound (UCB) utility
function defined as:

(6)
$a((x,z_{t}),D_{t})=\mu_{t}(x,z_{t})+\beta^{\frac{1}{2}}\sigma_{t}(x,z_{max})$

where $\mu_{t}(x,z_{t})$ favors high-performing regions and
$\sigma_{t}(x,z_{max})$ favors uncertain regions of the original objective. We
set $\beta$ to $0.2d\log(2t)$ following suggestions from (Kandasamy et al.,
2017; Srinivas et al., 2010), where $d$ is the dimension of the
hyperparameters and $t$ is the iteration index. The optimization starts from
$z_{t}=z_{min}$ to find initial promising candidates across the hyperparameter
space. When $z_{t}>z_{min}$, optimization of $a((x,z_{t}),D_{t})$ is
restricted to be close to known high-utility regions obtained from lower-
fidelity levels, which enables efficient search space pruning. This can be
achieved by simply applying a gradient-based local optimization method like
L-BFGS (Liu and Nocedal, 1989) on $a((x,z_{t}),D_{t})$, with solutions of
$\mathop{\arg\max}a(\cdot)$ at lower-fidelity levels serving as initial
guesses. The optimization of the acquisition function does not have to be
perfect, since we only care about promising points instead of optimal ones.
Our algorithm works well with our default choices for $\alpha,\beta$ and
$\epsilon$ without manual tuning. We summarize this procedure in algorithm 1
for ease of re-implementation.

###  3.4. Policy Transfer

Progressive acquisition function enables the use of transfer learning to speed
up evaluation on most $(x_{t},z_{t})$ pairs where $z_{t}>z_{min}$. This comes
from the fact that progressive acquisition function queries $z_{t}$
progressively, and only looks for $x_{t}$ within a common promising region.
Therefore, as long as $z_{t}>z_{min}$, a previously queried pair
$(x_{i},z_{i})$ with $i<t$ usually (if not always) exists such that
$(x_{t},z_{t})$ is close to $(x_{i},z_{i})$. Pre-trained policy on
$(x_{i},z_{i})$ can therefore serve as a warm start for training on
$(x_{t},z_{t})$.

In practice, $(x_{i},z_{i})$ can be acquired by traversing all previously
visited $(x,z)$ pairs to find the closest one to $(x_{t},z_{t})$ measured by
kernel function correlation $k((x_{t},z_{t}),(x_{i},z_{i}))$. Theoretically,
negative transfer may happen but we do not observe any in our experiments. We
show in our experiments that the use of transfer learning saves a significant
number of samples during CMFBO optimization.

Algorithm 2 summarizes the pipeline of CMFBO.

Input: Iteration index $t$, fidelity kernel length scale $l_{z}$, observation
dataset $D_{t}$ and uncertainty threshold $\epsilon$

Output: Best candidate pair $(x_{t},z_{t})$

$z_{t}\leftarrow z_{min}$

$x_{t}\leftarrow\mathop{\arg\max}\limits_{x}a((x,z_{t}),D_{t})$ using L-BFGS
with multiple random start points

while  _$z_{t} <z_{max}$ and $\sigma_{t}(x_{t},z_{t})<\epsilon$_ do

1.$z_{t}\leftarrow\mathop{\min}(z_{t}+l_{z},z_{max})$ ;

2.$x_{t}\leftarrow\mathop{\arg\max}\limits_{x}a((x,z_{t}),D_{t})$ using L-BFGS
with $x_{t}$ as an initial solution ;

Return $(x_{t},z_{t})$

Algorithm 1 Progressive Acquisition Function

Input: Multi-fidelity function $f(x,z)$ and the maximum allowed computational
cost $N_{total}$

Output: Observed function maximum $y^{\star}$ and corresponding $x^{\star}$

$N_{current}\leftarrow 0,D_{0}\leftarrow\varnothing$, the Gaussian Process is
initialized randomly.

for  _$t=1,2,...$_ do

1.Choose $(x_{t},z_{t})$ by progressive acquisition function optimization (see
algorithm 1);

2.Evaluate $f(x_{t},z_{t})$ to get performance $y_{t}$ and corresponding
computational cost $N_{t}$;

3.$D_{t}\leftarrow D_{t-1}\cup{((x_{t},z_{t}),y_{t})}$ and update parameters
of the Gaussian Process using $D_{t}$;

4.$N_{current}\leftarrow N_{current}+N_{t}$;

5.if  _$N_{current}\geq N_{total}$_ then

break;

$t^{\star}\leftarrow\mathop{\arg\max}\limits_{t}y_{t}$, $y^{\star}\leftarrow
y_{t^{\star}}$, $x^{\star}\leftarrow x_{t^{\star}}$;

Return $y^{\star}$ and $x^{\star}$

Algorithm 2 Curriculum-based MFBO

##  4\. EXPERIMENTS

We validate CMFBO on two physics-based character control tasks: morphology
optimization and hyperparameter optimization of DeepMimic. For each task, we
discuss its specific curriculum setting and objective function, followed by
experiment results, comparisons and ablation studies.

In our implementation, we use Gaussian Process from GPy(GPy, 2012) and L-BFGS
(Liu and Nocedal, 1989) from Scipy (Virtanen et al., 2020). All physics
simulation is performed on PyBullet(Coumans and Bai, 2019). We report the
performance statistics on a desktop with i7-7800x CPU (12 threads). Each
experiment is run $3$ times independently with different random seeds.

###  4.1. Morphology Optimization

The locomotion capability of a creature is intimately coupled with its
morphology. Elaborately designed morphology of simulated characters encourages
the emergence of natural locomotion (Geijtenbeek et al., 2013). In this
experiment, we optimize character morphology for learning fast and low-energy
locomotion. Our simulated character has $15$ torque-controlled revolute
joints. Morphology hyperparameter $x$ is defined to be a $12$-dimensional
vector that scales the length and radius of character links within a limited
range, as shown in table 1. Link mass is uniformly scaled according to its
volume. Sagittal symmetry is imposed on legs of the character.

####  4.1.1. Environment Setup

We designed a DRL-based character control environment tailored to fast
locomotion, based on humanoid environments in (Coumans and Bai, 2019).
Character state is defined to be orientation and linear velocities of the root
link, along with angles and velocities for all joints. Action is normalized
joint torques. We adopt a reward function specialized for locomotion control
proposed in (Xie et al., 2020), except that task specific and velocity penalty
terms are removed to encourage faster locomotion. Readers can refer to (Xie et
al., 2020) for more details of the reward function. We further adopt the
symmetry loss proposed in (Yu et al., 2018) to encourage symmetric gaits.

We use a feed-forward policy network with three fully connected layers, each
with $128$ units using $\mathop{\mathrm{tanh}}$ activation. The critic network
shares the same architecture with the policy. Policy is trained with Proximal
Policy Optimization (PPO) (Schulman et al., 2017). We set discount factor
$\gamma=0.95$ and $\lambda=0.95$ for both $TD(\lambda)$ and $GAE(\lambda)$
(Schulman et al., 2015). Learning rates of both policy and critic network are
set to $3\times 10^{-4}$. In each training iteration, we sample $4096$ state-
action tuples with $10$ paralleled environments. Batch size for policy update
is set to $256$.

####  4.1.2. Curriculum and Multi-fidelity Objective

We construct our multi-fidelity function based on a curriculum learning
setting proposed in (Yu et al., 2018), where a stable proportional-derivative
(SPD) controller (Tan et al., 2011) is applied to the character root to
provide hand-of-God balancing forces. Task difficulty is parameterized by
stiffness kp and damping kd coefficients of SPD controller. High gain
controllers make the task easier by providing large assistance forces. Given
the normalized task difficulty scalar $z$ ranging from $0$ to $1$, we set
$\textit{kp}(z)=200-200z$. kd is set to be equal to kp as proposed in (Yu et
al., 2018). The policy training starts at $\textit{kp}(0)$. kp gradually
decreases to $\textit{kp}(z)$ during training according to the curriculum
schedule in (Yu et al., 2018). We refer readers to section $4.2.2$ in (Yu et
al., 2018) for more details. The multi-fidelity objective is the normalized
return defined as:

(7) $f(x,z)=\frac{\sum_{t=1}^{T}r(s_{t},a_{t})}{T}$

where $r(s_{t},a_{t})$ is the reward function, and $T$ is the episode length
fixed to $500$ simulation steps ($8.25$ seconds).

Length scale | Range | Radius scale | Range
---|---|---|---
Torso | [0.3, 2] | Torso | [0.5, 1.5]
Waist | [0.3, 2] | Waist | [0.5, 1.5]
Pelvis | [0.3, 2] | Pelvis | [0.5, 1.5]
Thigh | [0.3, 2] | Thigh | [0.5, 1.5]
Knee | [0.3, 2] | Knee | [0.5, 1.5]
Foot | [0.3, 2] | Foot | [0.5, 1.5]
Table 1. Morphology hyperparameters.

(a)

(b)

Figure 4. (a):Results on morphology optimization. We compare best performance
over the number of training samples among CMFBO (Ours), GP-UCB, BOCA with
number of iteration as fidelity and (Schaff et al., 2019). (b): Ablation study
on morphology optimization. We further compare our method with BOCA with
curriculum-based fidelity (BOCA+CL), and CMFBO without policy transfer (w/o
transfer).

####  4.1.3. Comparison

We compare our method with several baseline and state-of-the-art algorithms: a
single fidelity BO with GP-UCB acquisition (Srinivas et al., 2010), a multi-
fidelity method BOCA (Kandasamy et al., 2017) and a recent DRL-based algorithm
(Schaff et al., 2019) (Schaff et al. 2019) specialized for morphology
optimization. GP-UCB and (Schaff et al., 2019) can be directly applied to the
problem. For multi-fidelity method BOCA, We implement it following (Kandasamy
et al., 2017) and construct the multi-fidelity function parameterized by both
number of training iterations and our proposed curriculum-based task
difficulty. Maximum allowed iteration number at fidelity $z$ for BOCA is set
to $500+1500z$ since most training finishes within $2000$ iterations. Maximum
allowed simulation steps $N_{total}$ is set to $10^{8}$. The optimization
process takes about $18$ hours for all methods.

Figure 5. Visualization of morphology and gaits optimized by different
methods.

####  4.1.4. Results

We plot the best performance over number of total simulation steps in figure
4(a) for various methods. Note that for multi-fidelity methods, best
performance is only recorded when the original objective function is
evaluated. Our method finds high-performing morphology hyperparameters much
faster than other methods. BOCA performs worse than single fidelity method GP-
UCB due to a misleading fidelity criterion defined as number of training
iterations, as explained in section 3.2. Figure 5 visualizes the best
morphology and the associated gaits for each method obtained from $3$
individual runs. Our method learns morphology resembling human and shows fast
and stable gaits. We encourage readers to see supplementary videos for more
details. Note that though our method learns different morphology
hyperparameters at different runs, proper values of essential components key
to stable locomotion are consistently learned, like thigh and foot lengths.

Figure 4(b) shows results of our ablation study, where we further compare our
method against BOCA with our proposed curriculum-based fidelity (BOCA + CL),
and our method without policy transfer (w/o transfer). With an effective
fidelity criterion defined by curriculum-based task difficulty, BOCA shows a
large performance gain. However, our method still outperforms BOCA by a large
margin due to effective search space pruning through our progressive
acquisition function. The use of transfer learning further improves efficiency
of our method, with a saving of roughly $2\times 10^{7}$ samples before a
high-performing set of hyperparameters is confirmed on the original fidelity.

###  4.2. DeepMimic Hyperparameter Optimization

DeepMimic is a recent DRL-based framework for diverse motor skill learning.
Many recent physics-based character animation systems are built on DeepMimic,
such as (Park et al., 2019; Bergamin et al., 2019; Won and Lee, 2019; Won et
al., 2020; Luo et al., 2020). As far as we know, this is the first work
towards hyperparameter optimization for high-dimensional physics-based
character control systems. In our experiments, we focus on the optimization of
DRL training hyperparameters listed in table 2. The choices of these
hyperparameters are critical for the learning performance where none of our
randomly sampled $5$ hyperparameters lead to successful policy training. These
hyperparameters correlate with each other implicitly and are hard to be
manually tuned. Some recent works (Ma et al., 2021; Won and Lee, 2019)
therefore simply adopt the default DeepMimic hyperparameters directly into
their systems with minor modifications. We evaluate our method on two motor
skill learning tasks: walk and backflip. Our optimized hyperparameters improve
DeepMimic learning efficiency by a large margin. Hyperparameters optimized for
walking can be reused for other skills as well resulting in superior
performance than original settings.

Parameters | Range | Default
---|---|---
Batch size | [$32$,$1024$] | $256$
| Policy updates
---
per iteration
[$1$,$10$] | $1$
| Learning rate
---
of actor network
[$2.5\times 10^{-6}$,$2.5\times 10^{-4}$] | $2.5\times 10^{-6}$
| Learning rate
---
of critic network
[$1\times 10^{-4}$,$1\times 10^{-2}$] | $1\times 10^{-2}$
Weight decay | [$5\times 10^{-4}$,$5\times 10^{-2}$] | $5\times 10^{-4}$
PPO clip rate | [$0.02$, $0.2$] | $0.2$
| Maximum of
---
the gradient norm
[$1$, $100$] | $100$
Table 2. Hyperparameters of DeepMimic

####  4.2.1. Curriculum and Multi-fidelity Objective

We construct our multi-fidelity function using a time-based curriculum
proposed in (Peng et al., 2018a; Kostrikov, 2018). Given the normalized task
difficulty scalar $z$, task difficulty is given by maximum episode length
$T(z)=30+570z$, which ranges from $1$ to $20$ seconds. Our multi-fidelity
objective considers both policy performance and sample efficiency, which is
defined as:

(8)
$f(x,z)=\frac{\sum_{t=1}^{t=T(z)}r(s_{t},a_{t})}{T(z)}e^{-(\frac{N_{sample}}{C})}.$

$N_{sample}$ is the total number of samples consumed during training. Training
is terminated when normalized expected return reaches a threshold pre-defined
in original DeepMimic, or the normalized expected return does not increase for
$500$ iterations. $C$ is a scaling factor set to $10^{7}$ and $2\times 10^{7}$
for walk and backflip respectively, based on the scale of usual samples
required by DeepMimic on different tasks. Note that original DeepMimic has a
fixed annealing-based schedule for maximum episode length during training.
This schedule can be too slow on easy tasks. We switch to an adaptive schedule
to avoid wasting samples. Starting from $1$ second, episode length is
increased by $0.5$ seconds every iteration until it reaches $T(z)$, as long as
the current fail rate is below $20\%$.

(a) Walk

(b) Backflip

Figure 6. DeepMimic hyperparameter optimization results. We compare best
performance over the number of training samples among CMFBO, GP-UCB and BOCA.

(a) Walk

(b) Backflip

Figure 7. Ablation studies on DeepMimic hyperparameter optimization.

####  4.2.2. Comparison

We compare our method against GP-UCB and BOCA in our experiments. For BOCA
based on number of iterations as its fidelity criterion, maximum iterations on
each fidelity $N(z)$ is set to $500+6500z$ and $500+11500z$ respectively for
walk and backflip. We also show comparison with BOCA using our curriculum-
based fidelity in our ablation study. Maximum allowed simulation steps
$N_{total}$ is set to $5\times 10^{7}$ for all methods. The optimization
process takes about $16$ hours for all methods.

Parameters | Walk | Backflip
---|---|---
Batch size | $172$ | $48$
Policy updates per iteration | $8$ | $10$
Learning rate of actor network | $1.68\times 10^{-5}$ | $4.6\times 10^{-5}$
Learning rate of critic network | $2\times 10^{-3}$ | $0.01$
Weight decay | $5\times 10^{-4}$ | $5\times 10^{-4}$
PPO clip rate | $0.18$ | $0.02$
Maximum of the gradient norm | $47.36$ | $12.30$
Table 3. Optimized hyperparameters for walk and backflip.

####  4.2.3. Results

As shown in figure 6, our method consistently outperforms other methods. CMFBO
find different hyperparameters with different random seeds, but the
performance is consistently good. Table 3 shows the best optimized
hyperparameters, which are quite different from the default ones listed in
table 2. We select best hyperparameters optimized from each method and test
their policy learning efficiency based on $3$ individual runs. Results are
shown in figure 8. GP-UCB fails to find hyperparameters with which policies
could be learned successfully. Since DeepMimic is computationally expensive,
only few evaluations can be performed in such single fidelity methods. BOCA
performs better than GP-UCB and find hyperparameters as good as the default
ones for walk task. It also finds hyperparameters converging $1.7$ times
faster than default ones for backflip. Comparing to GP-UCB and BOCA, our
method finds high-performing hyperparameters highly efficiently.
Hyperparameters optimized by CMFBO requires $5-6$ times less simulation steps
than default settings for successful policy learning. We note that by default
DeepMimic takes about $8$ and $16$ hours respectively to train control
policies until convergence for walk and backflip, while our method can find
high-performing hyperparameters within $8$ and $13$ hours respectively for
these tasks, even before the training of default DeepMimic finishes. We
conduct the same set of ablation studies as in morphology optimization shown
in figure 7, where we compare CMFBO against BOCA with curriculum-based
fidelity and CMFBO without policy transfer. As expected, each component is
essential to achieve the impressive efficiency of CMFBO.

We show that hyperparameters optimized by CMFBO generalize to other tasks as
well. Figure 9 illustrates the large performance gain of policy learning on
run and cartwheel tasks using hyperparameters optimized by CMFBO on walk
comparing with DeepMimic default settings.

CMFBO can also give us insights on the sensitivity of each hyperparameter.
After optimization, the surrogate model shows small length scales along
dimensions corresponding to policy updates per iteration, learning rate of
critic network and weight decay. This implies that the performance is more
sensitive to these hyperparameters within the given ranges.

We briefly analyze the computational cost distribution of CMFBO for DeepMimic
hyperparameter optimization. The optimization of acquisition function usually
finishes within three minutes, which is negligible comparing with the time
required for DRL training. We summarize the number of training samples
consumed on different fidelity levels in table 4, which indicates that most
computational resources are allocated to low fidelity approximations. Easy
tasks require less time to train than difficult ones. For instance, in walk
task of DeepMimic, training at the easiest and the original difficulty level
with our optimized hyperparameters take roughly $20$ minutes and $80$ minutes
respectively.

(a) Walk

(b) Backflip

Figure 8. Learning curves of policies trained with hyperparameters from CMFBO,
DeepMimic default settings, BOCA and GP-UCB.

(a) Run

(b) Cartwheel

Figure 9. Learning curves of policy learning for other tasks with hyperparameters optimized by CMFBO on walk task comparing with DeepMimic default settings. | Walk | Backflip
---|---|---
# of sampled hyperparameters | $20$ | $23$
# of training samples on original tasks | $6.1\times 10^{6}$ | $1.23\times 10^{7}$
# of training samples on low fidelities | $4.39\times 10^{7}$ | $3.77\times 10^{7}$
Table 4. The distribution of training samples in CMFBO

##  5\. CONCLUSIONS

This paper introduces the hyperparameter optimization problem in physics-based
character animation. We present CMFBO, a novel multi-fidelity Bayesian
optimization framework to achieve efficient optimization. Motivated by
curriculum learning, we propose the use of task difficulty as an effective
fidelity criterion, which enables relative performance of different
hyperparameters to be accurately estimated even with low-fidelity cheap
approximations. We enable efficient search space pruning through a progressive
acquisition function focusing only on optimal regions. Transfer learning is
further adopted to reduce evaluation cost of function queries. Through
extensive experiments, we demonstrate that CMFBO is more efficient than state-
of-the-art hyperparameter optimization algorithms. In particular, we show that
hyperparameters optimized through CMFBO result in at least 5x performance gain
comparing to original author-released settings in DeepMimic. We believe that
our method could serve as stepping-stone for automatic tuning of physics-based
character animation systems and free researchers or engineers from laborious
and exhausting tuning works.

There are several interesting future directions worth exploring. First, the
search space grows exponentially with the dimension of hyperparameters. We
could incorporate some recent machine learning and optimization techniques,
like random embedding (Wang et al., 2013), to scale our method to even higher
dimensions requiring much larger computation budgets, such as optimizing the
complex routing of muscles for full-body characters. Second, our current
method is limited to Euclidean search space. It would be interesting to extend
our method to general Non-Euclidean search space such as Riemannian manifolds
in (Jaquier et al., 2020). Last but not least, although our method is
demonstrated on simulated characters, it could possibly be extended to real-
world automatic robot design with the recent success of ’sim to real’ transfer
learning (Tan et al., 2018; Yu et al., 2019).

## References

  * (1)
  * Agrawal et al. (2014) Shailen Agrawal, Shuo Shen, and Michiel van de Panne. 2014\.  Diverse motions and character shapes for simulated skills.  _IEEE transactions on visualization and computer graphics_ 20, 10 (2014), 1345–1355.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009\.  Curriculum learning. In _Proceedings of the 26th annual international conference on machine learning_. ACM New York, NY, USA, 41–48.
  * Bergamin et al. (2019) Kevin Bergamin, Simon Clavet, Daniel Holden, and James Richard Forbes. 2019.  DReCon: data-driven responsive control of physics-based characters.  _ACM Transactions on Graphics (TOG)_ 38, 6 (2019), 1–11.
  * Bongard (2011) Josh Bongard. 2011\.  Morphological change in machines accelerates the evolution of robust behavior.  _Proceedings of the National Academy of Sciences_ 108, 4 (2011), 1234–1239.
  * Brochu et al. (2010) Eric Brochu, Tyson Brochu, and Nando de Freitas. 2010\.  A Bayesian interactive optimization approach to procedural animation design. In _Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation_. 103–112.
  * Brochu et al. (2007) Eric Brochu, Abhijeet Ghosh, and Nando de Freitas. 2007\.  Preference galleries for material design.  _SIGGRAPH Posters_ 105, 10.1145 (2007), 1280720–1280834.
  * Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016.  Openai gym.  _arXiv preprint arXiv:1606.01540_ (2016).
  * Coros et al. (2010) Stelian Coros, Philippe Beaudoin, and Michiel Van de Panne. 2010\.  Generalized biped walking control.  _ACM Transactions on Graphics (TOG)_ 29, 4 (2010), 1–9.
  * Coros et al. (2011) Stelian Coros, Andrej Karpathy, Ben Jones, Lionel Reveret, and Michiel Van De Panne. 2011\.  Locomotion skills for simulated quadrupeds.  _ACM Transactions on Graphics (TOG)_ 30, 4 (2011), 1–12.
  * Coumans and Bai (2019) Erwin Coumans and Yunfei Bai. 2016–2019.  PyBullet, a Python module for physics simulation for games, robotics and machine learning.  http://pybullet.org.
  * De Lasa et al. (2010) Martin De Lasa, Igor Mordatch, and Aaron Hertzmann. 2010\.  Feature-based locomotion controllers.  _ACM Transactions on Graphics (TOG)_ 29, 4 (2010), 1–10.
  * Felis and Mombaur (2016) Martin L Felis and Katja Mombaur. 2016.  Synthesis of full-body 3D human gait using optimal control methods. In _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 1560–1566.
  * Fujimoto et al. (2018) Scott Fujimoto, Herke Van Hoof, and David Meger. 2018\.  Addressing function approximation error in actor-critic methods.  _arXiv preprint arXiv:1802.09477_ (2018).
  * Geijtenbeek et al. (2013) Thomas Geijtenbeek, Michiel Van De Panne, and A Frank Van Der Stappen. 2013.  Flexible muscle-based locomotion for bipedal creatures.  _ACM Transactions on Graphics (TOG)_ 32, 6 (2013), 1–11.
  * GPy (2012) GPy. since 2012\.  GPy: A Gaussian process framework in python.  http://github.com/SheffieldML/GPy.
  * Ha (2019) David Ha. 2019.  Reinforcement learning for improving agent design.  _Artificial life_ 25, 4 (2019), 352–365.
  * Ha et al. (2017) Sehoon Ha, Stelian Coros, Alexander Alspach, Joohyung Kim, and Katsu Yamane. 2017.  Joint Optimization of Robot Design and Motion Parameters using the Implicit Function Theorem.. In _Robotics: Science and systems_.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _arXiv preprint arXiv:1801.01290_ (2018).
  * Hämäläinen et al. (2015) Perttu Hämäläinen, Joose Rajamäki, and C Karen Liu. 2015\.  Online control of simulated humanoids using particle belief propagation.  _ACM Transactions on Graphics (TOG)_ 34, 4 (2015), 1–13.
  * Hansen (2006) Nikolaus Hansen. 2006\.  The CMA evolution strategy: a comparing review.  In _Towards a new evolutionary computation_. Springer, 75–102.
  * Heess et al. (2017) Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. 2017\.  Emergence of locomotion behaviours in rich environments.  _arXiv preprint arXiv:1707.02286_ (2017).
  * Hodgins et al. (1995) Jessica K Hodgins, Wayne L Wooten, David C Brogan, and James F O’Brien. 1995.  Animating human athletics. In _Proceedings of the 22nd annual conference on Computer graphics and interactive techniques_. 71–78.
  * Hu et al. (2020) Sha Hu, Zeshi Yang, and Greg Mori. 2020.  Neural fidelity warping for efficient robot morphology design.  _arXiv preprint arXiv:2012.04195_ (2020).
  * Huang et al. (2020) Wenlong Huang, Igor Mordatch, and Deepak Pathak. 2020\.  One policy to control them all: Shared modular policies for agent-agnostic control. In _International Conference on Machine Learning_. PMLR, 4455–4464.
  * Jain et al. (2009) Sumit Jain, Yuting Ye, and C Karen Liu. 2009.  Optimization-based interactive motion synthesis.  _ACM Transactions on Graphics (TOG)_ 28, 1 (2009), 1–12.
  * Jaquier et al. (2020) Noémie Jaquier, Leonel Rozo, Sylvain Calinon, and Mathias Bürger. 2020.  Bayesian optimization meets Riemannian manifolds in robot learning. In _Conference on Robot Learning_. PMLR, 233–246.
  * Jones et al. (1998) Donald R Jones, Matthias Schonlau, and William J Welch. 1998\.  Efficient global optimization of expensive black-box functions.  _Journal of Global optimization_ 13, 4 (1998), 455–492.
  * Kandasamy et al. (2016) Kirthevasan Kandasamy, Gautam Dasarathy, Junier B Oliva, Jeff Schneider, and Barnabás Póczos. 2016.  Gaussian process bandit optimisation with multi-fidelity evaluations. In _Advances in Neural Information Processing Systems_. 992–1000.
  * Kandasamy et al. (2017) Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, and Barnabás Póczos. 2017.  Multi-fidelity bayesian optimisation with continuous approximations.  _Advances in Neural Information Processing Systems_ (2017), 1799–1808.
  * Karpathy and Van De Panne (2012) Andrej Karpathy and Michiel Van De Panne. 2012.  Curriculum learning for motor skills. In _Canadian Conference on Artificial Intelligence_. Springer, 325–330.
  * Klein et al. (2017) Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. 2017.  Fast bayesian optimization of machine learning hyperparameters on large datasets. In _Artificial Intelligence and Statistics_. PMLR, 528–536.
  * Kostrikov (2018) Ilya Kostrikov. 2018\.  PyTorch Implementations of Reinforcement Learning Algorithms.  https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail.
  * Koyama et al. (2020) Yuki Koyama, Issei Sato, and Masataka Goto. 2020.  Sequential gallery for interactive visual design optimization.  _ACM Transactions on Graphics_ 39, 4 (Jul 2020).  https://doi.org/10.1145/3386569.3392444
  * Koyama et al. (2017) Yuki Koyama, Issei Sato, Daisuke Sakamoto, and Takeo Igarashi. 2017\.  Sequential line search for efficient visual design optimization by crowds.  _ACM Transactions on Graphics (TOG)_ 36, 4 (2017), 1–11.
  * Liao et al. (2019) Thomas Liao, Grant Wang, Brian Yang, Rene Lee, Kristofer Pister, Sergey Levine, and Roberto Calandra. 2019.  Data-efficient learning of morphology and controller for a microrobot. In _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2488–2494.
  * Lillicrap et al. (2016) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2016.  Continuous control with deep reinforcement learning.  _International Conference on Learning Representations_ (2016).
  * Lipson and Pollack (2000) Hod Lipson and Jordan B Pollack. 2000.  Automatic design and manufacture of robotic lifeforms.  _Nature_ 406, 6799 (2000), 974–978.
  * Liu and Nocedal (1989) Dong C Liu and Jorge Nocedal. 1989.  On the limited memory BFGS method for large scale optimization.  _Mathematical programming_ 45, 1-3 (1989), 503–528.
  * Loper et al. (2015) Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015\.  SMPL: A skinned multi-person linear model.  _ACM transactions on graphics (TOG)_ 34, 6 (2015), 1–16.
  * Luck et al. (2020) Kevin Sebastian Luck, Heni Ben Amor, and Roberto Calandra. 2020\.  Data-efficient Co-Adaptation of Morphology and Behaviour with Deep Reinforcement Learning. In _Conference on Robot Learning_. PMLR, 854–869.
  * Luo et al. (2020) Ying-Sheng Luo, Jonathan Hans Soeseno, Trista Pei-Chun Chen, and Wei-Chao Chen. 2020\.  CARL: Controllable Agent with Reinforcement Learning for Quadruped Locomotion.  _ACM Transactions on Graphics (Proceedings of SIGGRAPH 2020)_ 39, 4 (2020), 10 pages.
  * Ma et al. (2021) Li-Ke Ma, Zeshi Yang, Tong Xin, Baining Guo, and KangKang Yin. 2021.  Learning and Exploring Motor Skills with Spacetime Bounds.  _Computer Graphics Forum_ 40, 2 (2021).
  * Merel et al. (2020) Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. 2020.  Catch & Carry: reusable neural controllers for vision-guided whole-body tasks.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 39–1.
  * Mnih et al. (2016) Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016.  Asynchronous methods for deep reinforcement learning. In _International conference on machine learning_. 1928–1937.
  * Mordatch et al. (2012) Igor Mordatch, Emanuel Todorov, and Zoran Popović. 2012\.  Discovery of complex behaviors through contact-invariant optimization.  _ACM Transactions on Graphics (TOG)_ 31, 4 (2012), 1–8.
  * Nguyen and Osborne (2019) Vu Nguyen and Michael A Osborne. 2019.  Knowing the what but not the where in Bayesian optimization.  _arXiv preprint arXiv:1905.02685_ (2019).
  * Nguyen et al. (2020) Vu Nguyen, Sebastian Schulze, and Michael Osborne. 2020\.  Bayesian optimization for iterative learning.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Park and Asada (1994) Jahng-Hyon Park and Haruhiko Asada. 1994.  Concurrent design optimization of mechanical structure and control for high speed robots.  (1994).
  * Park et al. (2019) Soohwan Park, Hoseok Ryu, Seyoung Lee, Sunmin Lee, and Jehee Lee. 2019.  Learning predict-and-simulate policies from unorganized human motion data.  _ACM Transactions on Graphics (TOG)_ 38, 6 (2019), 1–11.
  * Paul and Bongard (2001) Chandana Paul and Josh C Bongard. 2001.  The road less travelled: Morphology in the optimization of biped robot locomotion. In _Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No. 01CH37180)_ , Vol. 1. IEEE, 226–232.
  * Peng et al. (2018a) Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018a.  Deepmimic: Example-guided deep reinforcement learning of physics-based character skills.  _ACM Transactions on Graphics (TOG)_ 37, 4 (2018), 1–14.
  * Peng et al. (2015) Xue Bin Peng, Glen Berseth, and Michiel Van de Panne. 2015\.  Dynamic terrain traversal skills using reinforcement learning.  _ACM Transactions on Graphics (TOG)_ 34, 4 (2015), 1–11.
  * Peng et al. (2016) Xue Bin Peng, Glen Berseth, and Michiel Van de Panne. 2016\.  Terrain-adaptive locomotion skills using deep reinforcement learning.  _ACM Transactions on Graphics (TOG)_ 35, 4 (2016), 1–12.
  * Peng et al. (2017) Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. 2017.  Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning.  _ACM Transactions on Graphics (TOG)_ 36, 4 (2017), 1–13.
  * Peng et al. (2018b) Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. 2018b.  SFV: Reinforcement Learning of Physical Skills from Videos.  _ACM Transactions on Graphics (TOG)_ 37, 6 (2018).
  * Pil and Asada (1996) Anton C Pil and H Haruhiko Asada. 1996.  Integrated structure/control design of mechatronic systems using a recursive experimental optimization method.  _IEEE/ASME transactions on mechatronics_ 1, 3 (1996), 191–203.
  * Rasmussen (2003) Carl Edward Rasmussen. 2003\.  Gaussian processes in machine learning. In _Summer School on Machine Learning_. Springer, 63–71.
  * Schaff et al. (2019) Charles Schaff, David Yunis, Ayan Chakrabarti, and Matthew R Walter. 2019.  Jointly learning to construct and control agents using deep reinforcement learning. In _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 9798–9805.
  * Schulman et al. (2015) John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015.  Trust region policy optimization. In _International conference on machine learning_. 1889–1897.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ (2017).
  * Sims (1994) Karl Sims. 1994\.  Evolving virtual creatures. In _Proceedings of the 21st annual conference on Computer graphics and interactive techniques_. 15–22.
  * Snoek et al. (2012) Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012\.  Practical bayesian optimization of machine learning algorithms. In _Advances in neural information processing systems_. 2951–2959.
  * Snoek et al. (2015) Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. 2015.  Scalable bayesian optimization using deep neural networks. In _International conference on machine learning_. 2171–2180.
  * Song et al. (2019) Jialin Song, Yuxin Chen, and Yisong Yue. 2019.  A general framework for multi-fidelity bayesian optimization with gaussian processes. In _The 22nd International Conference on Artificial Intelligence and Statistics_. 3158–3167.
  * Spielberg et al. (2017) Andrew Spielberg, Brandon Araki, Cynthia Sung, Russ Tedrake, and Daniela Rus. 2017.  Functional co-optimization of articulated robots. In _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 5035–5042.
  * Srinivas et al. (2010) Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. 2010.  Gaussian process optimization in the bandit setting: No regret and experimental design.  _Proceedings of the 27th annual international conference on machine learning_.
  * Swersky et al. (2013) Kevin Swersky, Jasper Snoek, and Ryan P Adams. 2013\.  Multi-task bayesian optimization. In _Advances in neural information processing systems_. 2004–2012.
  * Swersky et al. (2014) Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. 2014\.  Freeze-thaw Bayesian optimization.  _arXiv preprint arXiv:1406.3896_ (2014).
  * Takeno et al. (2019) Shion Takeno, Hitoshi Fukuoka, Yuhki Tsukada, Toshiyuki Koyama, Motoki Shiga, Ichiro Takeuchi, and Masayuki Karasuyama. 2019\.  Multi-fidelity Bayesian optimization with max-value entropy search.  _arXiv preprint arXiv:1901.08275_ (2019).
  * Tan et al. (2011) Jie Tan, Karen Liu, and Greg Turk. 2011.  Stable proportional-derivative controllers.  _IEEE Computer Graphics and Applications_ 31, 4 (2011), 34–44.
  * Tan et al. (2018) Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. 2018\.  Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. In _Proceedings of Robotics: Science and Systems_. Pittsburgh, Pennsylvania.  https://doi.org/10.15607/RSS.2018.XIV.010
  * Tassa et al. (2012) Yuval Tassa, Tom Erez, and Emanuel Todorov. 2012.  Synthesis and stabilization of complex behaviors through online trajectory optimization. In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 4906–4913.
  * Tseng et al. (2019) Ethan Tseng, Felix Yu, Yuting Yang, Fahim Mannan, Karl ST Arnaud, Derek Nowrouzezahrai, Jean-François Lalonde, and Felix Heide. 2019.  Hyperparameter optimization in black-box image processing using differentiable proxies.  _ACM Trans. Graph._ 38, 4 (2019), 27–1.
  * Van de Panne and Lamouret (1995) Michiel Van de Panne and Alexis Lamouret. 1995.  Guided optimization for balanced locomotion.  In _Computer Animation and Simulation’95_. Springer, 165–177.
  * Villarreal-Cervantes et al. (2012) Miguel G Villarreal-Cervantes, Carlos A Cruz-Villar, Jaime Alvarez-Gallegos, and Edgar A Portilla-Flores. 2012\.  Robust structure-control design approach for mechatronic systems.  _IEEE/ASME Transactions on Mechatronics_ 18, 5 (2012), 1592–1601.
  * Virtanen et al. (2020) Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. 2020\.  SciPy 1.0: fundamental algorithms for scientific computing in Python.  _Nature methods_ 17, 3 (2020), 261–272.
  * Wampler et al. (2014) Kevin Wampler, Zoran Popović, and Jovan Popović. 2014\.  Generalizing locomotion style to new animals with inverse optimal regression.  _ACM Transactions on Graphics (TOG)_ 33, 4 (2014), 1–11.
  * Wang et al. (2009) Jack M Wang, David J Fleet, and Aaron Hertzmann. 2009\.  Optimizing walking controllers.  In _ACM SIGGRAPH Asia 2009 papers_. 1–8.
  * Wang et al. (2012) Jack M Wang, Samuel R Hamner, Scott L Delp, and Vladlen Koltun. 2012.  Optimizing locomotion controllers using biologically-based actuators and objectives.  _ACM Transactions on Graphics (TOG)_ 31, 4 (2012), 1–11.
  * Wang et al. (2018) Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. 2018\.  Nervenet: Learning structured policy with graph neural networks. In _International Conference on Learning Representations_.
  * Wang et al. (2013) Ziyu Wang, Masrour Zoghi, Frank Hutter, David Matheson, Nando De Freitas, et al. 2013\.  Bayesian Optimization in High Dimensions via Random Embeddings.. In _IJCAI_. 1778–1784.
  * Won et al. (2020) Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2020\.  A scalable approach to control diverse behaviors for physically simulated characters.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 33–1.
  * Won and Lee (2019) Jungdam Won and Jehee Lee. 2019.  Learning body shape variation in physics-based characters.  _ACM Transactions on Graphics (TOG)_ 38, 6 (2019), 1–12.
  * Wu and Popović (2010) Jia-chi Wu and Zoran Popović. 2010.  Terrain-adaptive bipedal locomotion control.  _ACM Transactions on Graphics (TOG)_ 29, 4 (2010), 1–10.
  * Xie et al. (2020) Zhaoming Xie, Hung Yu Ling, Nam Hee Kim, and Michiel van de Panne. 2020.  ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills. In _Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation_.
  * Yin et al. (2008) KangKang Yin, Stelian Coros, Philippe Beaudoin, and Michiel Van de Panne. 2008.  Continuation methods for adapting simulated skills.  In _ACM SIGGRAPH 2008 papers_. 1–7.
  * Yin et al. (2007) KangKang Yin, Kevin Loken, and Michiel Van de Panne. 2007\.  Simbicon: Simple biped locomotion control.  _ACM Transactions on Graphics (TOG)_ 26, 3 (2007), 105–es.
  * Yu et al. (2019) Wenhao Yu, Visak CV Kumar, Greg Turk, and C Karen Liu. 2019\.  Sim-to-real transfer for biped locomotion.  (2019).
  * Yu et al. (2018) Wenhao Yu, Greg Turk, and C Karen Liu. 2018.  Learning symmetric and low-energy locomotion.  _ACM Transactions on Graphics (TOG)_ 37, 4 (2018), 1–12.