#  The Ka-Band High Power Klystron Amplifier Design Program of INFN

Mostafa Behtouei1, Bruno Spataro1, Franco Di Paolo2 and Alberto Leggieri2
1INFN, Laboratori Nazionali di Frascati, P.O. Box 13, I-00044 Frascati, Italy
2Dipartimento di Ingegneria Elettronica, Università degli Studi di Roma “Tor
Vergata”, Via del Politecnico, 1-00133-Roma, Italia

###### Abstract

In the framework of the “Compact Light XLS”project, a short ultra-high
gradient linearizer working on the third harmonic of the main LINAC frequency
is requested. Increasing gradients and reducing dimensions are requirements
for XLS and all next generation linear accelerators. Actually, ultra- compact
normal conducting accelerating structures, operating in the Ka-band regime
ranging from 100 to 150 MV/m are required to achieve ultra-high gradients for
research, industrial and medical applications. To fulfill these strong
requirements, the R$\&$D of a proper Ka-band klystron with RF power output and
a high efficiency is mandatory. This contribution reports the design of a
possible klystron amplifier tube that provides and output signal at 36 GHz to
feed the phase space linearizer, while receiving at the input, a low level
signal oscillating at 12 GHz; which is the LINAC frequency. The proposed
structure reaches an efficiency of 42$\%$ and a 20 MW RF power output. This
contribution discusses also the high-power DC gun, the beam focusing channel
and the RF beam dynamics.

###### Index Terms:

High Power Klystron, Particle Acceleration, Linear Accelerators, Free Electron
Laser, Accelerator applications, Accelerator Subsystems and Technologies

##  I Introduction

High-brightness electron beams are required for a great number of
applications, including advanced accelerators linear colliders, X-ray Free-
Electron Lasers (FELs) and inverse Compton scattering accelerators for
research, compact or portable devices for radiotherapy, mobile cargo
inspections and security, biology, energy, and environmental applications.
This new generation of linear accelerators is highly demanding in terms of
accelerating gradients. In the framework of the INFN-LNF, SLAC (USA), KEK
(Japan), UCLA (Los Angeles) collaboration, the Laboratori Nazionali di
Frascati (LNF) is involved in the modeling, development and test of short RF
structures devoted to the acceleration with high gradient electric field by
means of metal devices made with hard copper or copper/silver (Cu/Ag) alloys.
A significant fraction of the Research and Development (R$\&$D) has been
devoted to studies of new manufacturing techniques to improve the maximum
sustainable gradients by short normal conducting hard RF structures. This aim
was achieved by minimizing the breakdown and the dark current for extremely
demanding applications and for the construction of millimeters high power high
frequency cavities [1, 2, 3, 4]. In addition, the present R$\&$D using
cryogenic copper technology (working conditions $\sim$77 K) will enable a
variety of new applications, including linear colliders and Free Electron
Laser acceleration, thanks to accelerating gradients over twice the value
achieved with the existing technologies [5, 6].

In the framework of the “Compact Light XLS” project [7], a short ultra-high
gradient linearizer working on the third harmonic (36 GHz) with respect to the
main linac frequency (11.988 GHz) [8, 9, 10]. In order to operate in the MW
power range, an accelerating gradient 150 MV/m (i.e., 15-20 MV integrated
voltage range) is requested [11]. To meet these requirements, a 36 GHz pulsed
Ka-band RF power source with a pulse length of 100 ns and a repetition
frequency in the (1-10) Hz range is necessary. These frequencies can be easily
be reached by solid state devices [12, 13] and innovative techniques [14, 15]
are proposed to increase the RF power levels however, actual solution of this
kind are not able to reach MW range of RF power, and vacuum electron devices,
especially klystrons are the only solution [16]. It is well known that the
klystron amplifier design requires a proper choice of

  * •

perveance

  * •

beam and pipe diameters

  * •

focusing magnetic field

  * •

bunching cavities and output cavity system

  * •

ultra-vacuum system

  * •

coupling coefficient

  * •

plasma frequency reduction factor, and

  * •

beam collector.

In addition, high electron beam power densities, RF electric gradient in
cavity gaps, and stresses on the ceramic window are just few of the most
critical issues to deal with. As a consequence, the choice of the global power
source parameters has to be consistent with the obtainable amplifier
efficiency and its operation stability. One of the more challenging aspects of
the high power klystron design is the electron gun device. The electron beam
perveance is defined as $K=IV^{-3/2}$ where I and V are beam current and
voltage. A high perveance results in a lower electronic efficiency due to the
higher space charge that affects the beam quality: Since the efficiency is
defined as the ratio of the output power to the input one, the voltage is
proportional to the $(P_{out}/(\eta K))^{2/5}$. Accordingly, we have to find
the best perveance to achieve a satisfactory efficiency with a maneageable
voltage [16].

In this paper, a particular high frequency high power source source is
described. In the proposed structure, the bunching section design approach is
almost conventional, while the output cavity is an extended interaction cavity
[16, 17] allowing loss reduction, breakdowns, etc., while achieving the needed
pulsed power [18, 19, 20]. Consequently, the objective is to design an output
circuit, which has the lowest gap fields, consistent with the satisfactory
efficiency and the operational reliability.

A preliminary theoretical efficiency estimation of the third harmonic klystron
operating on the $TM_{01}$ mode is around 18$\%$, about a factor 3 less than
the standard klystron efficiency. A design on the 34 GHz Ka-band klystron tube
amplifier by achieving more than 10 MW output power with a theoretical
efficiency estimation exceeding 22$\%$ has been proposed [21]. Basically, in
this design the RF system consists of a X-band input cavity, two X-band
bunching cavities, a complicated output cavity and a 5.5 mm diameter beam
pipe. In the output coupler, the output cavity is a $TM_{030}$ mode cavity
while others are $TM_{010}$ mode chocke cavities.

The strong motivation of this work is to propose a high power source with a
Ka-band harmonic klystron with the output cavity consisting in an extended
interaction cavity where 36 GHz is oscillating. This strategy allows to
optimize klystron conversion efficiency and provide a peak output power of 20
MW in a 100 ns pulse width, to be used to feed the linearizer while working at
150 MV/m of accelerating gradient as requested by all challenging next
generation projects [6, 7, 22, 23]. For this reason we are also planning to
finalize the traveling wave (TW) or the Standing wave (SW) linearizer design
as well as the RF power source that will be able to produce up to (50-60) MW
input power by using a SLED system [24, 25].

Currently there is no high power klystron operating on 3rd harmonics. It is
known that the theoretical efficiency of this device is estimated to be about
1/3 with respect to the standard klystron. The goal of this program is to
maximize the klystron efficiency in order to get enough power to feed a short
linearizer to be used in Compact Light XlS project [7] and Ultra Compact
Cryogenic FEL [5, 6]. In these applications the linearizers has to be designed
in order to provide an integrated voltage at least of 15 MV. To get this
integrated voltage we need a power in the range 50-60 MW. By using the SLED
system [24, 25] with a compression factor of 4 we can feed the linearizer. The
novelty of our work is to design a high power klystron operating at 36 GHz
operating to provide enough power to feed the linearizer. It should be noted
that with a Gyroklystron, that is able to provide up to 3 MW, also using the
SLED system we can achieve 12 MW which is not enough for the projects (Compact
Light lS project and Ultra Compact Cryogenic XFEL).

The paper is organized as follows: section 1 reports the preliminary analysis
of the design, section 2 discusses the electron gun and the focusing magnet
system, section 3 the interaction structure, the section 4 gives the
conclusions of the paper.

##  II Preliminary analysis on the design approach

Klystrons rely on passing an electron beam through one or more weakly coupled
resonant cavities which behaves like grids that serves the purpose of bunching
the electrons with resulting high frequency oscillation and amplification of
the grid current [21]. The electron beam first interacts with the Buncher
cavity, where it undergoes the force related to a low energy alternate field
that modulates the electron velocity. As the beam has crossed an opportune
distance from the Buncher, the velocity modulation becomes a modulation of the
charge density. This charge modulated beam is forwarded into another cavity
(the Catcher cavity), where it induces an oscillating field stronger than the
field inside the Buncher cavity: from the Catcher cavity the amplified RF
field is extracted.

The motivation of this work is to propose a high power source Ka-band klystron
in the millimeter wavelength domain for providing a power of 10-20 MW. In
designing the klystron amplifier, the choice of a proper electron gun has to
be considered, taking with particular attention to achieve a high compression
factor with a minimum transverse size. The electron gun relies on the Pierce-
type cathode for generating a converging and good laminar electron beam, to be
matched with a focusing magnetic field and manipulated in the interaction
region by klystron cavities to maximize the RF device power output. Assuming a
perveance of 0.3 $10^{-6}A/V^{3/2}$ , a factor of 2 less than other design
[21], with a 480kV pulsed beam voltage, an efficiency of about 42$\%$ can be
achieved.

Since this high voltage could generate breakdowns, an appropriate design in
the electron gun regions to withstand these high voltages is mandatory.
Cathodes used in standard high power klystrons working in ultra-high vacuum
conditions are good candidates. For a long lifetime the cathode loading has to
be limited well below 15 $A/cm^{2}$, too. A cathode diameter in the 8-10 cm
range can be considered to this aim. The interaction between the electron beam
and the RF cavities system and the Brillouin limit [22] are two important
parameters in order to control the beam radius limit for design requirements.
For getting the optimum klystron efficiency, the beam pipe dimension has to be
comparable to the beam radius [16]. Moreover, in order to minimize the beam
potential depression, a stable and reliable electron beam with a minimal
scalloping behavior is also required.

An average accelerating gradient of 150 MV/m needs at least 50 MW peak RF
power. With a 48 MW beam power (480 kV-100 A) and a 100 ns pulse width, 42
$\%$ of klystron efficiency to achieve 20 MW peak RF power on the third
harmonic is necessary. Finally, using a SLED system with a compression factor
4 [24, 25], the RF peak output on the third harmonic for feeding the
linearizer is 80 MW.

##  III Electron gun and focusing magnet system design

The electron gun design has been carried out by using the numerical code CST
[26]. Intense investigations carried out as function of the geometry anode-
cathode, beam current and applied voltage showed a high electrostatic beam
compression but with an excessive electric field on the focusing electrodes.
As an example, with a 480 kV-210 A beam, or a $\mu$-perveance of 0.631
$A/V^{3/2}$, a 24 MV/m electric field on the focusing electrode, is obtained.
Furthermore, in order to obtain a good klystron efficiency, it is also very
difficult to make the beam parameters of the klystron to comply to the design
requirements. A dedicated and detailed report on the electron gun design will
b discussed in a forthcoming paper. However, with a 480 kV beam voltage, 100 A
beam current (or a µ perveance of 0.3 $AV^{-3/2}$), an electrostatic beam
compression ratio of 1488:1, the maximum electric field on the focusing
electrodes is about 200 kV/cm, a reasonable value for assuring a safety
operational margin working with a 100 ns pulse length. Analytical
verifications confirmed the electron gun numerical investigation [27].

Figure 1:  a) 3D model of the gun and beam pipe. b) Beam trajectory along the
propagation direction c) axial magnetic field distribution.

Special attention was given also to the matching among the current generated
by the electron gun, the focusing magnetic system and the beam radius
dimensions to optimize the beam current transport. The external magnetic field
provides both beam focusing and coupling between the electrons and the RF
cavities system. Since the output cavity works on the fundamental $TM_{010}$
mode at 36 GHz and its radius is about 3.6 mm, the beam pipe has to be much
smaller and comparable to the allowed beam radius, giving klystron efficiency
optimization [16].

At the same time, the beam pipe shall place under cut off undesired signals:
The drive and the output frequency shall be placed under cut off in the beam
pipe.

Beam tracking simulations of the 480 kV-100 A gun showed that the electron
beam is confined within 1 mm radius with a maximum magnetic field of about 32
kG, a 1635:1 beam compression area and 2$\%$ scalloping parameter. The beam
pipe reduces the longitudinal space charge, which helps the efficiency in
general, but other issues have to be considered as the Beam stratification. We
have decided to taper the beam pipe up to 1.2 mm radius to confine the beam
radius within 1 mm along the beam pipe to satisfy the beam stratification.

The design parameters of the electron gun, including the magnetostatic beam
compression area, are listed in Table 1 and in Fig.1 the 3D model of electron
gun, beam trajectory along the propagation direction and magnetic field
distribution are reported in Fig.1.

TABLE I: Design parameters of the gun with focusing magnetic field along the beam axis Design parameters |
---|---
Beam power [MW] | 48
Beam voltage [kV] | 480
Beam current [A] | 100
$\mu-$ perveance $[I/V^{3/2}]$ | 0.3
Cathode diameter [mm] | 76
Pulse duration [$\mu$ sec] | 0.1
Minimum beam radius in magnetic system [mm] | 0.98
Nominal radius [mm] | 1.00
Beam pipe radius [mm] | 1.20
Max EF on focusing electrode [kV/cm] | 200
Electrostatic compression ratio | 1488:1
Beam compression ratio | 1635:1
Emission cathode current density [$A/cm^{2}$] | 2.02
Beam Transverse Emittance [mrad-cm] | 1.23 $\pi$

The magnetic field distribution reported in Fig. (1c) shows a small peak of 7
kG and a constant magnetic field of 32 kG along the 300 mm long pipe.This
choice allow to achieve a smaller beam radius suitable to insert the whole
system of RF cavities system as it will be described in the next section. In
the region where the magnetic field is 7 kG the beam radius is $\sim$2.3 mm,
higher by a factor of 4 than the Brillouin limit. In the region where the
field is 32 kG, the beam radius is $\sim$ 1 mm, again larger by a factor about
8 than the Brillouin limit, which is $\sim$ 0.13 mm.

We point out also that with the $\mu$-perveance of 0.657 $A/V^{3/2}$, a common
value for modern klystron, we could obtain 235 A, although with an efficiency
much smaller than with a beam current of $\sim$100 A. Reducing the
$\mu$-perveance from 0.657 $A/V^{3/2}$ to 0.3 $A/V^{3/2}$, we maintained the
cathode-anode shapes [28] while increasing the distance between them in order
to obtain a beam current of 100 A maintaining a satisfactory value of the
electric field on the focusing electrodes [29].

##  IV Interaction Structure Design

The beam spot is injected in the interaction structure with an initial
velocity, given by (1) [16], that is $u_{0}$ = 2.57 $\times 10^{8}ms^{-1}$,
corresponding to a normalized velocity $u_{0}$/c = 0.857, where $c_{0}$ is the
speed of the light in the vacuum.

$u_{0}=c_{0}\sqrt{1-\frac{1}{(1+\frac{eV_{0}}{m_{0}c_{0}^{2}})^{2}}}$ (1)

The interaction structure manipulates the beam produced from the electron gun
above described with a 12 GHz signal and an input power of 800W. Our proposed
structure is composed by 5 cavities, where the last cavity is an extended
interaction cavity integrating 4 gaps [16, 17, 32, 34]. It differs from a
traditional klystron for both arrangement and shape of the cavities and it
uses an extended interaction cavity for the output, where the energy is
distributed inside a large volume with the consequent advantage on the surface
electric fields.

The structure receives a beam of 100 A and 480 kV with a radius of 1 mm in a
drift tube of 1.2 mm radius.

The first interaction element, encountered by the electron beam, is the
Buncher cavity, that operates at 12 GHz and it is followed by gain cavities,
designed to have the higher R/Q without degrading the coupling. In the
Buncher, the inlet electron wavelength, $\lambda_{e}=u_{o}/f$ =21.4 mm,
specifies the beam propagation factor $\beta_{e}=2\pi/\lambda_{e}$ =293.5
$m^{-1}$, while the RF wavelength is $\lambda=c_{o}/f$=25.0 mm, and the wave
propagation factor $k=2\pi/\lambda=\omega/c_{o}$ = 251.5$m^{-1}$.

While the beam crosses the cavity gap, the velocity of the beam is modulated.
The ratio of change in beam voltage is regulated by the beam to wave coupling
coefficient M [16], given by (2). The electric field in the Buncher gap,
experienced by electron that traverse the cavity, determines the electron
energy variation in the gap. The peak $E_{RF\ pk}$ along the gap, mediated by
the coupling factor M, gives a shunt voltage $V_{RF}=M\int_{0}^{d}E_{RF\ pk}\
dz$. The coupling coefficient [16] can be numerically calculated by
integrating the electric field resulting by an Eigen mode simulation.

$M(\beta_{e})=\frac{V_{RF}}{V_{RF\
pk}}=\frac{\int_{-\infty}^{\infty}E_{z}e^{-i\beta_{e}z}dz}{\int_{-\infty}^{\infty}E_{z}dz}.$
(2)

The mediation is performed in the time interval between the instant when the
particle is at the center of the cavity and the instant when the electric
field is maximum.

The beam conductance is $G_{0}=I_{0}/V_{0}$ = 208.33 $\mu$S and, in the
Buncher, the beam-loading conductance, which represents the energy transfer
from the cavity to the beam, given by (4) [16], is $G_{b}=10.28\mu S$, that
represents a compromise between the need sinusoidal bunching and a non-
excessive beam loading.

$G_{b}=\frac{G_{0}}{2}M(M-\cos(\frac{\pi d}{\lambda_{RF}\frac{u_{0}}{c}})$ (3)

Figure 2:  Current modulation depth for the first three harmonics in function
of $J_{n}(X)$ for X = 1.84, 3.05, 4.20 respectively related to the current
modulation depth of $I_{1}$, $I_{2}$ and $I_{3}$.

The external coupling of the Buncher has been chosen to be reasonably low, in
order to use a strong signal at the input, $Q_{ext}$ = 20. This strategy can
allow to reduce the energy spread between beam electrons placed at different
radial coordinates. The distance between the Buncher and the first gain cavity
is chosen to ensure a smooth linear amplification even with a high coupling
factor for both the cavities.

The effect on the power transfer from the fundamental to superior harmonics is
critically dependent to the bunching parameter X. For a two cavity Klystron it
is [16]:

$X_{Cavity1\rightarrow 2}=M\frac{V_{RF}(cavity1)}{2V_{beam}\frac{2\pi
f\ell}{u_{0}}}$ (4)

The bunching parameter X shall provide, over the full klystron, the needed
distortion to enrich the space charge of the 3rd harmonic content, without
producing electron reflection or excessive electric fields. The power transfer
from the fundamental to the harmonic is function of the Bessel function
$J_{n}(X)$, where n identifies the harmonic index of the considered spectral
component of the output signal, as shown in Fig.2. A roughly estimation for
the efficiency of the two-cavity klystron operating on the third harmonic is
around 19$\%$, about a factor 3 less than the maximum klystron theoretical
efficiency operating at the fundamental frequency that is 58$\%$ [16].

(b)

Figure 3: Applegate diagram (distance-time plot) is superimposed to the cavity
axial electric field normalized to the maximum value given by the cavity
eigenmode calculation.

(a)

(b)

Figure 4:  a) The velocity variation; b) The modulation depth of the 1st
harmonic current at different beam substrates as a function of the
longitudinal coordinate z. In the vertical axis, $J_{z1}$ is for the 1st
harmonic current normalized to the dc current, $J_{z0}$.

After the Input cavity, the Buncher, three gain cavities are placed with the
opportune drift spaces in order to operate a smooth amplification and to
produce the desired signal compression before the output cavity. The gap of
the gain cavities is enough large to have a good coupling with the beam and
ensure the desired signal growth. The Output circuit is an extended
interaction output cavity that integrates four gaps [32, 34]. The multiple
gaps are designed to be synchronous with the beam for the mode chosen during
the passage of the beam inside the whole cavity. This output circuits do not
obey this rule and make possible wider bandwidths than that of a traditional
cavity and/or more power extraction from the beam. The beam, that traverses
the extended interaction cavity, is subjected to a continuous extraction of
energy during the whole path. Such a cavity has not only a larger surface from
which to remove the heat, but also a higher R/Q, which leads to a lower
eternal quality factor and a higher circuit efficiency. Within the extended
interaction cavity, the restrictions of single or gridded gaps are removed and
the beam interaction takes place with a generalized field. Such a field may
consist of several discrete regions, corresponding to cavities placed closely
together, or it may be the continuous field of a shorted helix or a coupled
cavity structure [16].

The most important aspect is that these structures for R/Q values that cannot
be attained with single-gap cavities. As clearly explained by Caryotakis [16],
this is a direct result of the fact that these circuits are no longer driving-
point impedances as single-gap cavities are and hence not subject to the
bandwidth limitation. Extended Interaction klystrons can attain bandwidths
modestly superior to ordinary klystrons. High R/Q output circuits also imply
low external $Q_{ext}$, and higher circuit efficiency. Extended interaction
cavities can distribute the necessary retarding RF voltage at an output cavity
over several gaps, thereby reducing the RF gradient and the danger of RF
breakdown. This is the principal reason for the use of extended cavities in
pulsed klystrons, though circuit efficiency was also a consideration [16]. In
the presented structure, from the Extended Interaction Cavity, the signal at
the output frequency is provided.

The R/Q, coupling coefficient and $G_{b}/G_{0}$ for an extended interaction
cavity must be chosen carefully and the beam loaded conductance must be
positive for the operating mode. This approach required a certain value of
micro-perveance to produce a satisfactory efficiency. The gap to gap period is
dimensioned to ensure the synchronism between the beam and wave phase
velocity, while the beam traverse the cavity. Depending on the beam voltage,
the operative cavity mode is selected [16]. An inconvenient of the extended
interaction cavities is the possibility to lead oscillations at lower
frequencies than the operating.

Interesting solutions are being proposed in the domain of extended interaction
techniques to be used in frequency conversion mostly by using higher order
mode output cavities, as proposed by G. Burt [30] and D. Consatble [31] at
Lancaster University and J. Cai and I. Syratchev[32] and CERN.

An important aspect concerns the dimension of the structure that determines
the possibility to be manufactured.

Figure 5: The modulation currents normalized to the dc current, $J_{z0}$ as
function of the longitudinal coordinate z. In the vertical axis $J_{zn}$ is
the higher harmonic current, which is calculated for different harmonic
(n=1,2,..,12).

The klystron Buncher is a cylindrical cavity with a radius of 8.43 mm,
operating in a quasi $TM_{010}$ mode. The beam pipe has a radius aperture of
1.2 mm. The extended interaction [17] output cavity has a radius of 3.37mm can
be realized using Electrostatic Discharge Machining (EDM). The size of the
component composing the designed structure allows for manufacturing processes
based on Control Numeric Computer (CNC) machining. Klystron performances are
limited by the beam voltage reduction effect and by the more invasive large
signal phenomena, including plasma resonances; which introduces oscillations
below the drive frequency, as given by (6) [16] that apply a plasma reduction
factor on the interaction frequency.

$f_{p}=\frac{1}{2\pi}\sqrt{\frac{e}{m_{0}}\frac{J_{0}}{\epsilon_{0}u_{0}\gamma^{3}}}$
(5)

where $e/m_{0}=1.759\times 10^{11}$ is the electron charge to rest mass ratio,
$\epsilon_{0}=8.85\times 10^{-12}$ F/m the permittivity of vacuum and $\gamma$
= 1+ V0 / 511kV is the Lorenz factor and u0 the beam velocity. In presence of
deep modulation $V_{RF}>>V_{0}$, the space charge effect due to the plasma
frequency reduces the bunching parameter. A numerical approach for a large
signal deign is mandatory.

Numerical behavioral characterization of high efficiency klystron amplifier in
large signal regimes requires specialized computer codes. 1D codes like AJDisk
[33], and KlypWin, operate in an idealized environment, allow for a very fast
(1-10 min) computation time. 2D codes like TESLA, Klys2D and FCI, account for
all radial effects and economize on computation time (10 min - 1 hour).
Particle-In-Cell (PIC) codes like MAGIC2D/3D and CST Particle Studio provide
complete and self-consistent field solution therefore the computation time
could be very large (10-100 hours) [34]. A new 1D/2D complex simulation code
has been developed by Dr. Cai and Syratchev at CERN to operate with a
precision close to that of a PIC code with the benefit of discrete approach
that reduces the computation time to 5-30min. The used approach assumes that
radial beam expansion has little effect on the RF power production; therefore,
only longitudinal beam dynamic is enough to be considered in the klystron
optimization routine. In such approximation, PIC approach can be successfully
replaced by the model based on the generic large signal theory. Such a
theoretical model was developed and implemented into new large signal 1D and
2D klystrons simulation code called KlyC [34]. It has been demonstrated by a
large number of examples on high efficiency klystron simulations, operating in
very difficult computational condition needed to get high efficiency that the
KlyC code is equivalent to a PIC code when cavity geometries or cold fields
are used as source data for the beam dynamics computation. The KlyC
simulations were evaluated by comparison with the CST PIC code, and both codes
showed a good (within 1$\%$) agreement, while KlyC was significantly (about
100 times) faster [34].

The software KLyC [34]. has been used for the computer aided design of the
structure. The best effort has been done to obtain a sufficiently short
structure suitable to accommodate a considerably narrow focusing magnet and
obtain the perveance needed for the required beam power output ensuring the
maximum efficiency.

The presented simulations have been obtained by setting the following KlyC
parameters [34].: Number of beams: 8, Space charge field order: 12, Electron
wavelength division number: 256, RF wavelength division number: 128, Iteration
residual limit:10-4, Max iterations: 64, Iteration relaxation: 0.25.

At the same time, a minimal velocity, below -0.1c, has been considered to
avoid electron reflections in the output cavities. The phase grouping is shown
in Fig. 3, where the Applegate diagram is superimposed to the cavity axial
electric field normalized to the maximum value given by the cavity Eigen-mode
calculation. In order to get high efficiency, the electron beam is
significantly decelerated in the last cavity of the output coupler down to a
velocity of -0.09 c along the z direction. This effect occurs in the last
cavity, where the beam, has a considerably reduced energy. The velocity
variations are showed in Fig. (4a). The modulation depth of the $1^{st}$
harmonic current at different beam substrates is shown in Fig. (4b). In this
figure, $J_{z1}$ is the $1^{st}$ harmonic current normalized to the dc
current, $J_{z0}$. The 1st harmonic content has a specific distribution on
different substrates of the beam. The inner substrates have a slightly reduced
fundamental content and this is one of the reason for which the last gain
cavity is a low R/Q cavity.

The superior harmonic content grows while the signal is sufficiently
compressed; this operation is demanded mostly to the $4^{th}$ cavity. In the
output cavity, the $3^{rd}$ harmonic phasor of the drive frequency, reaches
the amplitude 0.77 of the amplitude of the fundamental. The output signal
oscillates at 36 GHz. In Fig. (5), $J_{zn}$ is the higher harmonic current,
which is calculated for different harmonic (n=1,2,..,12). The output power is
20 MW at 36 GHz. The max gap voltage is 471 kV, which is produced in the
extended interaction output cavity, where is also located the maximum electric
field (222 MV/m).

The minimum velocity level is $-0.09^{*}c$; which fulfils the requirement on
$v_{min}>-0.1^{*}c$. A summary of the klystron output is reported in Table II.
The considered efficiency is the Balanced Efficiency computed by KlyC. This
quantity accounts for the ohmmic losses that, a traditional discrete method
can’t.

TABLE II: Resume of Klystron Output Simulated quantity | Output data
---|---
$P_{out}$ [kW] | 19983.804
Gain [dB] | 43.976
RFefficiency | 0.416
Min v/c | -0.094

The design software KLyC has been used for the computer aided design of this
structure. With this tool efforts have been made to design a sufficiently
short structure to be accommodated inside a considerably narrow focusing
magnet, characterized by a high perveance as required by the beam power
output, while ensuring the maximum efficiency. At the same time, a minimal
velocity, below -0.1 c, has been obtained to avoid electron reflections in the
output cavities. The output element, a waveguide iris facing on a dielectric
window, can be designed. Since the Klystron produces a higher signal at the X
band; the window can be optimized at the Ka band, in order to not reduce power
in this range.

Two dielectric windows could be also designed following standard rules [35].
The max gap voltage is 302.7 kV and the max electric field is 138.4 MV/m that
for a 200ns pulse avoid electric discharges.

##  V Conclusions

This paper proposes a novel Klystron structure that provides an output signal
to energize a phase space linearizer while receiving at the input the LINAC
frequency.

In order to linearize the longitudinal space phase of the Compact Light XLS
project a Ka-Band accelerating structure operating on the $\pi$ mode at the
third RF harmonic with respect to the main LINAC RF frequency has been
considered. The linearizer can work with a high accelerating gradient around
up to 150 MV/m. In this contribution, a klystron amplifier has been also
investigated to feed this linearizer structure. We presented the design of the
high-power DC gun, of the beam focusing channel and of the RF beam.

The numerical model has been applied to the needs of the Compact Light XLS
project, to design a high power Klystron multiplier optimized to provide a 36
GHz output while receiving a 12GHz input.

The proposed structure can be supplied by same Low Level Radio Frequency
driver of an X-band klystron, offering the possibility to control the phase of
the two power source outputs. Lower parasitic and phase noise than using a
single higher frequency driver would be also present. The proposed structure
shows an efficiency of 42$\%$ and a gain of 44dB. At the output of the device
a signal at the 3rd harmonic of the drive frequency is generated with a power
of 20 MW.

The max gap voltage is 302.7 kV and the max electric field is 138.4 MV/m that
for a 200ns pulse avoid electric discharges. The minimum velocity level is
-0.094 c, which respect the common design requirement on $v_{min}>$-0.1c, to
prevent electron reflections. This device is particularly suitable to produce
pulsed signals ranging from 100 to 200 ns.

This feature is in line with requirements of high gradient accelerators
operating in Ka band, which are foreseen to achieve gradients around 150 MV/m.

The dimension of the cavities, will allow using a complete CNC process without
the use of electro erosion technology, needed to machine the cavities.

The application of the proposed design principle is very large: In addition to
the 12 GHz to 36 GHZ application, it can be efficiently used in a frequency
manipulation from 4 to 12 GHz. In order to extend the applicability to higher
frequencies of the proposed principle or to reduce machining complexity,
structures using higher order mode cavities can be engineered

## Acknowledgment

The authors would like to thank Cai Jinchi, Igor Syratchev, and Zening Liu for
their support in the development of the RF beam dynamic.

This work was partially supported by the Compact Light XLS Project, funded by
the European Union’s Horizon 2020 research and innovation program under grant
agreement No. 777431.

## References

  * [1] Dolgashev, V. A., et al. Innovative compact braze-free accelerating cavity. Journal of Instrumentation 13.09 (2018): P09017.
  * [2] Dolgashev, V. A., et al. “Materials and technological processes for High-Gradient accelerating structures: new results from mechanical tests of an innovative braze-free cavity.”Journal of Instrumentation 15.01 (2020): P01029.
  * [3] B. Spataro et al., “High Gradient RF Tests of Welded X-Band Accelerating Structure”, Manuscript submitted for publication in PRAB.
  * [4] Othman, Mohamed AK, et al. “Experimental demonstration of externally driven millimeter-wave particle accelerator structure”. Applied Physics Letters 117.7 (2020): 073502.
  * [5] Rosenzweig, J. B., et al. “Next generation high brightness electron beams from ultrahigh field cryogenic rf photocathode sources.”Physical Review Accelerators and Beams 22.2 (2019): 023403.
  * [6] Rosenzweig, J. B., et al. “An ultra-compact x-ray free-electron laser.”New Journal of Physics 22.9 (2020): 093067.
  * [7] http://www.compactlight.eu/Main/HomePage
  * [8] Behtouei, M., et al. “A Ka-band linearizer TW accelerating structure for the Compact Light XLS project.”Journal of Physics: Conference Series. Vol. 1596. No. 1. IOP Publishing, 2020.
  * [9] Behtouei, Mostafa, et al. “New Analytical derivation of Group Velocity in TW accelerating structures.”Journal of Physics: Conference Series. Vol. 1350. No. 1. IOP Publishing, 2019.
  * [10] Behtouei, Mostafa, et al. “A SW Ka-Band Linearizer Structure with Minimum Surface Electric Field for the Compact Light XLS Project.”Nuclear Instruments and Methods in Physics Research Section A. Vol. 984, 2020 .
  * [11] Nezhevenko, O. A., and V. P. Yakovlev. “Traveling-wave accelerating test structure at 34.3 GHz.”Proceedings of the 1999 Particle Accelerator Conference (Cat. No. 99CH36366). Vol. 5. IEEE, 1999.
  * [12] Passi, Davide, et al. “Design of high power density amplifiers: application to ka Band.”Journal of Infrared, Millimeter, and Terahertz Waves 38.10 (2017): 1252-1263.
  * [13] Leggieri, Alberto, et al. “Multiphysics design of a spatial combiner predisposed for thermo-mechanically affected operation.”Journal of Electromagnetic Waves and Applications 28.17 (2014): 2153-2168.
  * [14] Passi, D., Leggieri, A., Di Paolo, F., Tafuto, A., Bartocci, M.“ Spatial power combiner technology”, Progress in Electromagnetics Research Symposium, 2015, 2015-January, pp. 932?937.
  * [15] Leggieri, Alberto, et al. “ The Squarax spatial power combiner.”Progress In Electromagnetics Research 45 (2013): 43-55.
  * [16] Caryotakis, George. “High power klystrons: Theory and practice at the Stanford linear accelerator center.”Stanford Linear Accelerator Center, SLAC-PUB 10620 (2004).
  * [17] Zhang, Haiyu, Ding Zhao, and Jirun Luo. “The beam-wave interaction calculation of a Ka-band Extended Interaction Klystron.”2016 IEEE International Vacuum Electronics Conference (IVEC). IEEE, 2016.
  * [18] Lee, T. G., et al. “The design and performance of a 150-MW klystron at S band.”IEEE transactions on plasma science 13.6 (1985): 545-552.
  * [19] Eppley, Kenneth. “Design of a 100 MW X-band klystron.”Proceedings of the 1989 IEEE Particle Accelerator Conference, Accelerator Science and Technology. IEEE, 1989.
  * [20] Fazio, Michael V., and K. Habiger. “Design for a One-Gigawatt Annular-Beam Klystron.”No. LA-UR-00-3793. Los Alamos National Lab., Los Alamos, NM (US), 2000.
  * [21] Chang, X., et al. “Ka-Band High Power Harmonic Amplifier for Bunch Phase-Space Linearization.”North American Particle Accelerator Conf.(NAPAC’19). 2019.
  * [22] Yakovlev, V. P., and O. A. Nezhevenko. “Limitations on area compression of beams from pierce guns.”AIP Conference Proceedings. Vol. 474. No. 1. American Institute of Physics, 1999.
  * [23] Ferrario, M., et al. “EuPRAXIA SPARC-LAB Design study towards a compact FEL facility at LNF.”Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 909 (2018): 134-138.
  * [24] Ivanov, O. A., et al. “Active quasioptical Ka-band rf pulse compressor switched by a diffraction grating.”Physical Review Special Topics-Accelerators and Beams 12.9 (2009): 093501.
  * [25] Ivanov, O. A., et al. “Active microwave pulse compressor using an electron-beam triggered switch.”Physical review letters 110.11 (2013): 115002.
  * [26] CST studio suite 2018. www.cst.com.
  * [27] Behtouei, Mostafa, et al. “Initial design of a high-power ka-band klystron.”Journal of Physics: Conference Series. Vol. 1596. No. 1. IOP Publishing, 2020.
  * [28] Behtouei, Mostafa, et al. “Electron Gun and Magnetic Systems Studies for a 36 GHz Klystron Amplifier”, Technical report, INFN-Publication, INFN-20-10/LNF.
  * [29] Behtouei, Mostafa, et al. “Initial electromagnetic and beam dynamics design of a Klystron amplifier for Ka-Band Accelerating Structures.”INFN-19-12/LNF, 2019. Available online: http://www. lnf. infn. it/sis/preprint/detail-new. php, 2019.
  * [30] Burt, Graeme, et al. “A millimeter-wave klystron upconverter with a higher order mode output cavity.”IEEE Transactions on Electron Devices 64.9 (2017): 3857-3862.
  * [31] Egorov, R. V., et al. “The Physical Limitations of the Efficiency of High Power Multibeam Klystrons.”Moscow University Physics Bulletin 75.5 (2020): 451-458.
  * [32] Cai, J. C., and I. Syratchev. “Modeling of coupled cell output structures for the klystrons.”IEEE Transactions on Electron Devices 66.11 (2019): 4964-4969.
  * [33] Jensen, Aaron, et al. “Developing sheet beam klystron simulation capability in AJDISK.”IEEE Transactions on Electron Devices 61.6 (2014): 1666-1671.
  * [34] Cai, Jinchi, Igor Syratchev, and Zening Liu. “KlyC: Large Signal Simulation Code for Klystrons.”submitted to IEEE TED (2017).
  * [35] A. Variola, private communication.


# An Adaptable Approach to Learn Realistic
Legged Locomotion without Examples

Daniel Ordonez-Apraez1, Antonio Agudo1, Francesc Moreno-Noguer1 and Mario
Martin2,3

###### Abstract

Learning controllers that reproduce legged locomotion in nature has been a
long-time goal in robotics and computer graphics. While yielding promising
results, recent approaches are not yet flexible enough to be applicable to
legged systems of different morphologies. This is partly because they often
rely on precise motion capture references or elaborate learning environments
that ensure the naturality of the emergent locomotion gaits but prevent
generalization. This work proposes a generic approach for ensuring realism in
locomotion by guiding the learning process with the spring-loaded inverted
pendulum model as a reference. Leveraging on the exploration capacities of
Reinforcement Learning (RL), we learn a control policy that fills in the
information gap between the template model and full-body dynamics required to
maintain stable and periodic locomotion. The proposed approach can be applied
to robots of different sizes and morphologies and adapted to any RL technique
and control architecture. We present experimental results showing that even in
a model-free setup and with a simple reactive control architecture, the
learned policies can generate realistic and energy-efficient locomotion gaits
for a bipedal and a quadrupedal robot. And most importantly, this is achieved
without using motion capture, strong constraints in the dynamics or kinematics
of the robot, nor prescribing limb coordination. We provide supplemental
videos for qualitative analysis of the naturality of the learned gaits444
Supplemental videos: https://bit.ly/30NtYAu.

---

Figure 1: Control policies for the biped robot Bolt ($1.05[m/s]$) and
quadruped robot Solo ($0.96[m/s]$) learned with model-free RL learning guided
with the spring-loaded inverted pendulum model. All gaits across training
seeds feature periodicity, limbs coordination and natural spring-mass center
of mass dynamics.

††
1 Institut de Robòtica i Informàtica Industrial, CSIC-UPC, Barcelona, Spain
[dordonez, aagudo<EMAIL_ADDRESS>
2 Departament de Ciències de la Computació, Universitat Politècnica de
Catalunya. 3 Barcelona Supercomputing Center (BSC<EMAIL_ADDRESS>
This work’s experiments were run at the Barcelona Supercomputing Center in
collaboration with the HPAI group. This work is supported by the Spanish
government with the project MoHuCo PID2020-120049RB-I00 and the ERA-Net
Chistera project IPALM PCI2019-103386.

##  I INTRODUCTION

The study and generation of natural legged locomotion is a problem that has
been approached both by the robotics and computer animation communities. In
robotics, the aim is to enable mobile platforms to locomote in various
terrains with the flexibility, energy efficiency, and robustness observed in
animals in nature. Alternatively, computer graphics focus on generating
animations of legged locomotion that appear natural and are compliant to
physical laws and interactions with the virtual environment.

Reinforcement Learning, especially in computer graphics, has shown remarkable
progress in generating physics-based controllers for complex animated
characters in simulation that showcase highly complex and dynamic motion
skills [24, 29, 30]. While demonstrating impressive results, the scalability
of these methods to robots of different morphologies (i.e., body size, mass,
legs anatomy, and number of legs) is somewhat compromised. Mainly because, to
ensure realism, they rely on: (1) Motion capture references [23, 30]. (2)
Precise modeling of the kinematic constraints and approximate dynamics of the
musculoskeletal system [20, 35]. (3) Learning environments that feature
elaborate curriculum learning techniques [35] or meta-learning of multiple
motion skills [18].

These approaches aim to aid the learning process by guiding or constraining
solution space (i.e., the feasibility region of the optimal policy [24]) to
avoid converging to unnatural motions. The necessity to do so comes from the
fact that for most legged systems, there exist an extensive set of feasible
control policies that maintain locomotion; For instance, a human can move
forwards at a speed of $2.0\,[m/s]$ by jogging naturally but also by jumping,
limping, using a single leg, and so on. The low variance of gait types in
nature across individuals of the same species is a consequence of the tendency
of animals to choose the properties of their gaits to maximize stability and
robustness while minimizing energy expenditure [1].

For most animals in nature, the selection of the optimal gait that minimizes
the Cost of Transport (CoT) (i.e., energy used over distance traveled) is a
complex and not well-understood function of the forward velocity, terrain
properties, and the dynamics of the animal’s musculoskeletal system [1, 26].
For this reason, replicating this optimization function in simulation is still
an open problem, especially since accurate models of the body dynamics are
hard to attain or require considerable design efforts. However, bio-inspired
models of legged locomotion are able to approximate valuable properties of the
dynamics of locomotion in nature [13, 19], providing substantial queues of the
gait properties and gait selection mechanisms in animals.

This work shows that realistic high-speed locomotion can be learned by guiding
the control policy with a Spring-Loaded Inverted Pendulum (SLIP) template
model. To achieve this, we design an RL environment that employs controlled
trajectories of SLIP as reference motions. These are then used to loosely
constrain the allowed robot Center of Mass (CoM) trajectories in time,
effectively limiting the policy’s exploration space to full-body state-action
trajectories that feature SLIP CoM dynamics.

This learning environment is designed to mimic several properties of full-body
CoM trajectory optimization techniques, as the policy is encouraged to learn
full-body control that results in energy-efficient CoM trajectories in the
vicinity of the reference motion. The main difference is the use of RL to
address the optimal control problem instead of traditional convex trajectory
optimization methods [3, 7, 8]. With the use of RL we lose the assurances of
convex optimization algorithms but gain flexibility in experimentation by
avoiding relying on simplifying assumptions of the gait and body dynamics
(e.g., centroidal dynamics, zero CoM angular momentum, knowledge of Centers of
Pressure (CoP) during contacts, contact timings). Furthermore, our learning
setup is compatible with both model-based and model-free RL and with the use
of any feed-forward controller.

Having a generic approach to learning realistic legged locomotion in
simulation can be a valuable tool for gait analysis and robot and control
design. Some immediate practical applications are the estimation of design
requirements (e.g., joint and actuator profiles), identification of passive
components, or generation gait information (e.g., centroidal or CoM linear and
angular momentum trajectories) to improve the performance of trajectory
optimization methods [2]. This work takes a small step towards such a generic
system.

##  II BACKGROUND

###  II-A Spring Loaded Inverted Pendulum

SLIP is a low-order template model for the dynamics of legged locomotion in
the sagittal plane. In its simplest form, the model comprises of a point-mass
$m$ representing the body’s center of mass and a massless spring-leg that
models compliance during contact with the ground (see Fig. 2-left) [12].

Although the model highly simplifies the dynamics of locomotion, it can
predict the CoM dynamics’ low-frequency components in the sagittal plane (see
Fig. 2-right) [12, 13, 19]. The potential of SLIP as a generic model for
animal locomotion in nature was proven by Full et al. [11] in a comparative
study of the dynamics of high-speed locomotion across different animal species
and body sizes. The study revealed that most animals experience similar
profiles of Ground Reaction Forces (GRFs) while running and that these forces
could be approximated with SLIP models. Furthermore, there was evidence of
patterns in the parametrization of the model for each animal and gait type,
allowing to approximate the animal’s GRF during locomotion with a SLIP model
parameterized with the animal’s mass $m$, hip height $r_{0}$, and spring
constant $k$ given by:

$k=\frac{k_{rel}\,m\,g}{r_{0}}\;\;\;\;\textrm{with}\;\;\;\;k_{rel}=\frac{\frac{F}{mg}}{\frac{\Delta
r}{r_{0}}},$ (1)

where $F$ is the maximum GRF measured for each animal in the stance phase,
$\Delta r$ indicates an approximated maximum vertical displacement of the CoM
during the stance phase, $k_{rel}$ represents a dimensionless spring constant
used to compare animals across species, and $g$ is the gravity constant. The
parametrization of SLIP models’ spring constant is usually guided by the
distribution of $k_{rel}$ values observed in nature (see further details in
[11]).

Information encoded in SLIP dynamics. The SLIP model dynamics provides insides
on the expected time of contact during a stance phase (as a function of the
number of legs in contact, the spring constant, and velocity at touch-down),
the expected ground reaction forces, and gait period. Additionally, it models
the natural fluctuations of the animal’s kinetic, potential, and elastic
potential (of muscles) energies during locomotion [11, 12].

|
---|---
Figure 2: SLIP Model parameters and stance cycle. Left: SLIP gait cycle with
flight and stance phases. $m$ and $k$ are the model parameters mass and spring
stiffness constant, respectively. Right: SLIP predicted ground reaction forces
as a function of the stance time is plotted in red. Traces of ground reaction
forces observed in human running experiments in black. Right image adapted
from [12].

###  II-B Maximum Entropy Reinforcement Learning

Maximum entropy RL is a framework that exploits the concept of entropy from
information theory [4, 17] and introduces it to the classical RL policy
objective to address the known challenge of exploration-exploitation [33]. In
this framework, agent’s policies are modeled by parametric probability
distributions, and the agent is tasked with learning to succeed at a given
task by maximizing the expected reward while at the same time acting as
randomly as possible (i.e., maximize the policy distribution entropy) [17].
The augmented policy objective is therefore defined as:

$\pi^{*}=\underset{\pi}{\textrm{argmax}}\sum_{t}\mathbb{E}_{(\mathbf{s}_{t},\mathbf{a}_{t})\sim\rho_{pi}}\left[r(\mathbf{s}_{t},\mathbf{a}_{t})+\alpha\mathcal{H}(\cdot|\mathbf{s}_{t})\right],$
(2)

where $\pi$ represents the RL agent policy,
$\mathcal{H}(\cdot|\mathbf{s}_{t})$ refers to the entropy of the policy given
the present state, $\alpha$ the temperature parameter for entropy
maximization, and $r(\mathbf{s}_{t},\mathbf{a}_{t})$ represents the
environment reward as a function of the agent present state $\mathbf{s}_{t}$
and action $\mathbf{a}_{t}$.

Maximum entropy RL has been wildly adopted in the robotics community [16, 18,
21, 30] as its reduced sample complexity is promising for real-life
applications. In this work, we use Soft Actor-Critic (SAC) [17], an off-policy
actor-critic algorithm that has shown remarkable sample complexity,
exploration capacities and requires almost no hyper-parameter tuning. This
algorithm was used for real-world locomotion learning with success in [16,
22].

###  II-C Space-Time Bounds

Space-Time bounds is a technique introduced in [24] to ensure that a control
policy learns dynamic skills while remaining in the close neighborhood of a
reference motion. The technique uses an early-termination criteria to stop the
RL episodes whenever the agent’s state at a given time $\mathbf{s}_{t}$
violates a fixed or dynamical state constraint (e.g., detection of robot fall
or a velocity limit violation). In our work, the reference SLIP motions are in
CoM space rather than in joint space or cartesian/system space (see [28] for
details).

By constraining the episode state dynamics with a space-time bound
$\mathcal{B}$, we effectively ensure that the control policy learned through
experimentation generates causal state trajectories that remain within
$\mathcal{B}$. This set of possible causal state trajectories allowed by
$\mathcal{B}$ is referred to as the feasible region (or solution space)
$\mathcal{F}(\mathcal{B})$ of the control policy.

##  III RELATED WORK

###  III-A Learning Realistic Legged Locomotion

Previous locomotion learning approaches use various forms of curriculum
learning techniques to guide the learning process and reduce exploitation of
simulation inaccuracies or unmodeled dynamics by the learned policy. The
benchmark tasks on locomotion train an agent to progressively maximize its
forward velocity during learning, using the increasing velocity as a mechanism
for pruning unstable and undesired gaits [9]. Other approaches rely on
modeling more realistically musculoskeletal system dynamics; for instance, in
[35], the authors tune the damping and friction coefficients of the DoF of
each robot to encourage realism, and aid the training process with a
curriculum learning that stabilizes the robot during the early stages of
training. Another work [20], approximates realistic state-dependent joint
torque limits of humanoid robots to ensure the emergence of realistic motions
using both trajectory optimization techniques and model-free RL. Lastly, an
alternative technique to encourage realism is to train the robots in parallel
on multiple motion tasks in a meta-learning environment, reducing the capacity
of the policy to exploit unnatural gaits that are functional for single-tasks
environments [18].

|  |  | Revolute joints limits
---|---|---|---
|  $r_{0}$[cm] | m[kg] | Pos[rad] | Vel[rad/s] | Torque[N/m]
Bolt | 35 | 1.3 | $[-\pi,\pi]$ | $[-4\pi,4\pi]$ | [-2.7, 2.7]
Solo | 24 | 2.2 | $[-\pi,\pi]$ | $[-4\pi,4\pi]$ | [-2.7, 2.7]

TABLE I: Bolt and Solo hip height $r_{0}$ and mass $m$ alongside their joints
position, velocity and torque limits.

###  III-B Guiding controllers with low-order template models

Using low-order template models to design or guide controllers is not a novel
idea. The assumption of centroidal dynamics is a common technique to enable
the convex trajectory optimization methods that fuel most state-of-the-art
platforms [3, 7]. These controllers are sensitive to initialization [2],
require prescribed contact timings, and struggle to perform in unstructured
environments [21].

In the context of RL, a bipedal extension of the SLIP model was used to
successfully guide the learning process of a walking gait for the Cassie robot
[14].

##  IV METHODOLOGY

In this work, we evaluate the potential use of the dynamics of the SLIP
template model in the problem of learning controllers for legged locomotion
that achieve natural and realistic behavior. The motivations to study this
problem are the following:

  * •

The SLIP model provides a flexible mechanism for approximating the CoM
dynamics of high-speed legged locomotion observed in nature, making it an
ideal candidate for generating reference motions that bias the learned control
policy to appear natural.

  * •

A well-structured RL environment offers the ideal learning flexibility to
learn control policies that fill in the information gap between the template
model CoM dynamics and the robot full-body dynamics while making no
assumptions about the legged system morphology.

  * •

Realistic and energy-efficient gaits of high-speed locomotion learned in
simulation can become valuable assets in studying the role of individual legs
and CoM momentum during gaits.

Our experiments focus on learning control policies that maintain a constant
forward velocity through a periodic, realistic, and energy-efficient gait. We
test our approach on the bipedal robot Bolt (6 DoF) and a quadrupedal robot
Solo (8 DoF) [15]. The robots’ kinematics can resemble those of an Ostrich and
Cat, but in reality, they have dynamics that differ significantly from any
animal in nature, as they are abnormally lightweight and possess joints that
are greatly under-constrained and overpowered (see table I).

For each robot, we independently learn control policies for moving at constant
velocities of medium and high magnitudes (i.e., 3 and 6 times the robot’s hip
height per second). For Bolt we consider $\dot{x}_{des}=[1.05,2.10]\,[m/s]$
and for Solo $\dot{x}_{des}=[0.60,0.96]\,[m/s]$.

###  IV-A Enforcing SLIP dynamics

Figure 3: Reference SLIP gait CoM trajectory in the sagittal plane for Bolt
maintaining a forward velocity of $2.1[m/s]$ with an Space-Time Bound
$\mathcal{B}$ with $\epsilon=0.75$ (shaded area). Gray and yellow areas
represent flight and stance phases of the SLIP gait, respectively.
Experimental trajectories in the early stages of training are plotted to
illustrate the boundary violation scenarios and episode terminations.

The reference motions for each robot and forward velocities are obtained
through the optimal control of a SLIP model [5, 27] that is parameterized with
the total robot mass ($m$), the hip height ($r_{0}$) [32], and a relative
spring constant $k_{rel}=10.7$ according to [11, 13]. The number of legs
expected to contact the ground at the same time during a stance phase is 1 for
the bipedal robot Bolt and $2$ for the quadrupedal Solo.

Considering that the SLIP model only approximates the low-frequency CoM
dynamics during legged locomotion [11, 19] (see Fig. 2 right), during
learning, we loosely constrain the dynamics of the robot CoM with the
reference motion by terminating the learning episode when the robot violates a
space-time bound around the reference SLIP CoM trajectory. This condition is
evaluated by considering the bound $\mathcal{B}(t)$ at every time step $t$ in
simulation as:

$\displaystyle\mathcal{B}(t)$
$\displaystyle=\\{\mathbf{s}_{G}(t)\;|\;\rho>|\mathbf{s}_{G}(t)-\mathbf{s}_{\scriptscriptstyle
SLIP}(t)|\\},$ (3) $\displaystyle\mathbf{s}_{G}$
$\displaystyle=[x_{G},y_{G},z_{G},\dot{x}_{G},\dot{y}_{G},\dot{z}_{G}],$ (4)
$\displaystyle\mathbf{s}_{{\scriptscriptstyle SLIP}}$
$\displaystyle=[x_{S},y_{S},z_{S},\dot{x}_{S},\dot{y}_{S},\dot{z}_{S}],$ (5)

where $\mathbf{s}_{G}$ represents the robot’s CoM state in the sagittal plane,
$\mathbf{s}_{{\scriptscriptstyle SLIP}}$ is the reference SLIP trajectory
state, and $\rho$ is a vector that controls the size of the bound (see Fig.
3).

To independently avoid tuning the size of each dimension of the bound for each
robot and forward velocity, we parameterize $\rho$ utilizing the robot
dimensions and the present gait cycle of the reference trajectory as:

$\begin{split}\rho=\epsilon\,\left[\infty,\frac{r_{0}}{2},\frac{r_{0}}{4},\hat{\dot{x}}_{S},\frac{\hat{\dot{z}}_{S}}{2},\frac{\hat{\dot{z}}_{S}}{2}\right]\end{split},$
(6)

where $\hat{\dot{x}}_{S}$ and $\hat{\dot{z}}_{S}$ represent the difference
between the maximum and minimum forward and vertical velocities of the SLIP
trajectory in the entire present gait cycle, respectively. Lastly, $\epsilon$
is a scalar coefficient that defines the size of the space-time bound.

Space/Energy-Time bounds. The bound $\mathcal{B}$ in Eq. (5) and its
implications have a clearer intuition in terms of the system energy: By
constraining the robot’s CoM height and velocity in the sagittal plane, we
effectively bound the robot’s potential and CoM kinetic energies in time.
Therefore, the imposed feasibility region of the control policy
$\mathcal{F}(\mathcal{B})$ constrain the possible locomotion gaits to those
that behave energetically similarly to the template model, i.e., gaits that
have variations in phase of the potential and CoM kinetic energy. If the bound
is sufficiently tight, this encourages the robot to use its legs to mimic the
energy storage and release mechanisms observed in legged systems in nature
during high-speed locomotion [12, 19].

###  IV-B Reinforcement Learning Environment

The Markov Decision Process (MDP) used in this work is defined by the tuple
$\langle\mathcal{S},\mathcal{A},r,\mathcal{T},\gamma,s_{0}\rangle$.
$\mathcal{S}$ and $\mathcal{A}$ are the state and action spaces, respectively.
$r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}+$ is a positive reward
function defined for every state action pair.
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$ represents
the environment dynamics transition function, governed by the physics
simulator Bullet [6]. Moreover, $\gamma$ is the discount factor, and $s_{0}$
indicates the initial state distribution.

As in [24], the reward is a positive real function so that the policy is
encouraged to remain within $\mathcal{F}(\mathcal{B})$. The reward can be
represented by $r=r_{S}\cdot r_{P}$, where $r_{S}\in\\{0,1\\}$ is a survival
reward taking the value of $1$ whenever the robot state is within the bound
$\mathcal{B}$ and $0$ otherwise, and $r_{P}\in[0,1]$ is a multiplicative
energy and torque minimization reward, similar to the ones used in [9],
defined as:

$r_{P}=\left(1-\cfrac{\lVert\mathbf{\tau}\dot{q}\rVert}{\lVert\mathbf{\tau}_{max}\dot{q}_{max}\rVert}\right)\cdot\left(1-\cfrac{\lVert\mathbf{\tau}\rVert}{\lVert\mathbf{\tau}_{max}\rVert}\right),$
(7)

where $\mathbf{\tau}$ is the joint torques and $\dot{\mathbf{q}}$ the joint
velocities.

Intuitively, the early episode termination using the bound $\mathcal{B}$
constrains the exploration of the RL agent to remain close to the reference
SLIP trajectory in CoM space, i.e., the agent will only accumulate experiences
of causal state-action trajectories that maintain the robot’s CoM within the
$\mathcal{F}(\mathcal{B})$. At the same time, due to the positive reward, as
learning progresses, the agent will try to survive by following state-action
trajectories of a longer duration, while the energy minimization reward
penalizes inefficient trajectories.

This learning environment was designed as a CoM full-body trajectory
optimization: The RL exploration (in state-action space) is constrained to the
local vicinity of the reference trajectory (in CoM space) and the cost/reward
function is composed solely of an energy minimization term.

###  IV-C Maximum-Entropy RL and Control architecture

The learning setup of this work is framed as a Maximum Entropy Inverse
Reinforcement Learning (IRL) task [25, 37], where the SLIP dynamics are used
as the demonstrations and the objective of the optimization is to find the
optimal control policy that maximizes the expected reward and policy
distribution entropy (see Eq. (2)). The algorithm used is SAC with automatic
temperature adjustment [17].

The policy $\pi_{\phi}(\mathbf{a}|\mathbf{s})$ and Q functions
$Q_{\psi}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)$ of SAC are parameterized
with Fully-Connected Neural Networks $\phi$ and $\psi$ with two hidden layers
of 256 neurons and ReLu activation functions. As in [17], the control policy
is modeled as a state conditional $tanh$ transformed normal distribution
$\pi_{\phi}(\mathbf{a}|\mathbf{s})\sim\tanh{\mathcal{N}(\mathbf{\mu}_{\phi},\Sigma_{\phi})}$
with mean and diagonal covariance matrix parameterized by $\phi$. All
experiments are run with $\gamma=0.995$ and a learning rate of $lr=3e^{-4}$.
Lastly, the control frequency is set to $200Hz$, considering the reactive
nature of the policy.

###  IV-D State and Action space

We use a reactive policy that directly controls the torques of the robot
joints $\mathbf{\tau}$, by scaling the action
$\mathbf{a}\sim\pi_{\phi}(\mathbf{a}|\mathbf{s})$ by each DoF torque limits
(see table I). The MDP state space $\mathbf{s}$ is defined as:

$[\mathbf{h}_{G},z_{G},R,\mathbf{q},cos(\phi_{{\scriptscriptstyle
SLIP}}),sin(\phi_{{\scriptscriptstyle
SLIP}}),\dot{x}_{des}-\dot{x}_{G},\mathbf{c}],$ (8)

where $\mathbf{h}_{G}=A_{G}(\mathbf{q})\mathbf{\dot{q}}$ is the robot’s CoM
momentum expressed in the robot base coordinates, computed using the
Centroidal Momentum Matrix (CMM) $A_{G}(\mathbf{q})$ [28]. $z_{G}$ is the
robot’s base height. $R$ is the 6D continuous representation of the robot’s
base orientation in world coordinates [36]. As in [21],
$\phi_{{\scriptscriptstyle SLIP}}$ is a gait phase signal increasing linearly
from $0$ to $\pi$ during the flight phase and from $\pi$ to $2\pi$ on the
stance phase of the SLIP gait cycle. Lastly, $\dot{x}_{des}$ is the desired
forward velocity, and $\mathbf{c}$ is a binary vector indicating contact of
the robot’s feet with the ground as in [35]. Note that the state observation
is invariant w.r.t the direction of forward motion.

###  IV-E Directional and Sagittal Symmetry

A relevant property of gaits in nature is their symmetry: Animals, including
humans, tend to control their body’s left and right parts similarly when not
injured. In the context of RL, [38] showed that when MDPs have symmetry, the
optimal policy, Value, and $Q$ functions are also symmetrical. While in legged
locomotion learning [35] reported enhanced performance and realism of gaits
when the symmetry was encouraged in the control policy.

In this work, we extended the policy symmetry loss in [35] to function for
SAC’s policies. The main difference is that [35] uses a Proximal Policy
Optimization (PPO) policy which has fixed variance in their action
distribution, while in SAC, the variance for the action distribution is also a
function of the input state ($\Sigma_{\phi}$). Therefore our policy symmetry
loss is defined as:

$\displaystyle\mathbb{L}_{sym}=$
$\displaystyle\lVert\bar{\pi}_{\phi}(\mathbf{s})-\Psi_{a}(\bar{\pi}_{\phi}(\Psi_{s}(\mathbf{s})))\rVert+$
$\displaystyle\lVert\hat{\pi}_{\phi}(\mathbf{s})-|\Psi_{a}(\hat{\pi}_{\phi}(\Psi_{s}(\mathbf{s})))|\,\rVert,$
(9)

where $\bar{\pi}_{\phi}=\mathbb{E}[\pi_{\phi}(\mathbf{a}|\mathbf{s})]$
represents the mean value of the policy distribution, and
$\hat{\pi}_{\phi}=\mathbb{E}[\pi_{\phi}(\mathbf{a}|\mathbf{s})-\bar{\pi}_{\phi}]$
the variance of the action policy distribution. Lastly, $\Psi_{s}$ and
$\Psi_{a}$ are functions mapping states and actions to their mirrored
equivalents.

Furthermore, influenced by [38], we encouraged symmetry in the Critic
Q-functions by augmenting each batch sampled from the replay buffer with its
symmetric equivalent.

Note that encouraging symmetry in control actions merely requires the control
policy to have similar reactions to symmetric states, but not necessarily
symmetric gaits trajectories.

##  V EXPERIMENTAL RESULTS

We evaluated our approach by learning control policies for both robots with
and without enforcing the SLIP dynamics and symmetry. The control policies
trained without enforcing the SLIP dynamics use a space-time bound
$\mathcal{B}_{const}$ where only the CoM forward velocity and lateral position
are bounded. The bound for the forward velocity is centered around the desired
constant speed $\dot{x}_{des}$, with $\epsilon=2.0$ to allow for the expected
oscillation around the target velocity proper of animal gaits. Therefore the
policy feasibility region $\mathcal{F}(\mathcal{B}_{const})$ contains any
locomotion gait that can remain close to the target velocity. In the rest of
the document, we refer to the policies trained using the SLIP space-time bound
$\mathcal{B}$ as with-SLIP, and without-SLIP for the ones using
$\mathcal{B}_{const}$.

(a)

(b)

(c)

(d)

Figure 4: Reward curves and gait’s Cost of Transport for high speed locomotion
of Solo $0.96[m/s]$ (a and b) and for Bolt $2.1[m/s]$ (c and d). All training
curves show the step average, maximum and minimum (shaded region) across 5
training seeds. Exploiting symmetry results in more robust control policies
(i.e., higher and faster survival rates) across all training variants. For
Bolt exploiting symmetry reduces considerable the sample-complexity. Although
quantitatively the variants with-SLIP and symmetry appear not to perform
better, we highly encourage the reader to compare the gait properties
qualitatively in the supplemental video 4.

###  V-A Qualitative and quantitative effects of imposing SLIP dynamics

The control policies without-SLIP for both robots find stable gait
trajectories that maintain forward velocity within the imposed bound
$\mathcal{B}_{const}$. However, realistic or natural behavior is not present
in any of the training seeds (see comparison video 444 Videos with-SLIP vs.
without-SLIP: youtu.be/_0Y3sJ9w9F8). Due to the large feasibility region, each
training seed rapidly converges to unnatural aperiodic gaits that exploit the
unmodeled dynamics (e.g., actuator dynamics, joint-space friction, and
damping) and the simulation inaccuracies in contact dynamics (specifically the
transition from static to dynamic friction regimes). Furthermore, because the
elastic properties of muscles are not modeled, the energy minimization reward
$r_{P}$ leads both robots to locomote with short and rapid stepping with
nearly fully extended legs (since such a gait minimizes the required energy
and torques for compensating gravity).

On the contrary, policies trained with-SLIP showcase natural limb coordination
and spring-mass dynamics, resulting in realistic and periodic gaits. Because
there is no penalization on the torso’s linear and angular momentum, the
robots learn to locomote with graceful and natural torso motions that sync
with leg contacts. Both robots perform irregular footstep sequences when
needed to stabilize but tend to converge to near-periodic footstep sequences
once they achieve to remain in a limit cycle. In all experiments, Solo
converges to use a trot gait, coordinating diagonally opposed legs, while Bolt
uses a running gait.

From a quantitative perspective, the control policies without-SLIP showcase
similar or better performance in sample complexity, gait robustness, and
energetic cost-of transport (see Fig. 4). Therefore, for the RL agent, the
unnatural gaits represent suitable and close to optimal control policies. This
minor difference in quantitative performance highlights the challenges of
obtaining realistic behavior with learning techniques without introducing
inductive biases in the learning process to encourage realism.

Gait variance across seeds. Perhaps the most relevant impact of enforcing SLIP
gait dynamics through space-time bounds is the reduction of the variance of
gait properties across training seeds, meaning that final policies for each
seed converges to locomotion gaits that feature similar limb coordination and
CoM trajectories (see all seed’s gaits in supplemental video 4).

Utility of learned gaits. Figure 5 presents the robot base centroidal
trajectory during the SLIP gait cycle for the Bolt. This trajectory and those
of all training seeds alongside other gait properties can be exploited by
centroidal or CoM trajectory optimization methods [8, 28, 31, 34].

Figure 5: With-SLIP and symmetry linear and angular gait centroidal velocities
of the torso of Bolt, at medium and high forward velocities. The gaits
showcase periodicity and spring-mass dynamics proper of the SLIP model. For
the lateral velocity $\dot{y}$, roll $\dot{r}$ and yaw $\dot{\mathrm{y}}$ the
two trajectory modes correspond to gait cycles with left and right leg contact

###  V-B Impact on symmetry enforcing

The training results in Fig. 4 show a considerable increase in performance by
exploiting the symmetry of the robot control and the MDP for both with-SLIP
and without-SLIP variants, as there are signs of improved sample complexity
and control robustness (i.e., ability to maintain stable gaits). The impact of
symmetry in the realism of the gaits is best observed qualitatively in the
comparison video 555 Videos with-symmetry vs. without-symmetry:
youtu.be/qN25-XQBqKM.

##  VI CONCLUSIONS AND FUTURE WORK

In this work, we show the potential use of the SLIP model as a mechanism for
measuring and asserting realism in gaits of high-speed locomotion. Taking a
step towards locomotion learning without MoCap references. To achieve this, we
designed an RL environment capable of producing realistic full-body periodic
locomotion gaits for any robot morphology, using a simple energy minimization
reward. We present results with a bipedal and quadrupedal robot in simulation
for the model-free control case and a reactive policy acting directly on joint
torques.

The advantage of our approach relies on the use of controlled trajectories of
the SLIP model as a reference to loosely constrain the policy exploration
space. This model can be parameterized to approximate the dynamics of high-
speed locomotion of most animals/robots. Enabling applications where no motion
capture data is available.

In future work we want to: (1) Enforce symmetry as a hard constraint in the
optimization process by using equivariant function approximators for the actor
and critic functions [10]. (2) Expand the experiments scope to embrace a
larger suite of robot morphologies (and sizes), terrain properties, and energy
penalization schemes (based on various models of the cost of transport [1]).
(3) Explore the use of extended versions of the SLIP model to address the
realism of walking gaits of different robot morphologies. (4) Adapt our
approach for Computational Design: By studying the learned robot dynamics at
different speeds, we theorize, we can test potential passive components for
each DoF to optimize gait energy expenditure, control robustness, and/or the
robot’s kinematics and dynamics.

## References

  * [1] R McNeill Alexander  “Models and the scaling of energy costs for locomotion”  In _Journal of Experimental Biology_ 208.9 Company of Biologists, 2005, pp. 1645–1652
  * [2] Gerardo Bledt and Sangbae Kim  “Extracting legged locomotion heuristics with regularized predictive control”  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 406–412  IEEE
  * [3] Gerardo Bledt et al.  “MIT Cheetah 3: Design and control of a robust, dynamic quadruped robot”  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018, pp. 2245–2252  IEEE
  * [4] Tom Carter  “An introduction to information theory and entropy”  In _Complex systems summer school, Santa Fe_ Citeseer, 2007
  * [5] Hua Chen, Patrick M Wensing and Wei Zhang  “Optimal control of a differentially flat two-dimensional spring-loaded inverted pendulum model”  In _IEEE Robotics and Automation Letters_ 5.2 IEEE, 2019, pp. 307–314
  * [6] Erwin Coumans and Yunfei Bai  “PyBullet, a Python module for physics simulation for games, robotics and machine learning”, http://pybullet.org, 2016–2021
  * [7] Jared Di Carlo et al.  “Dynamic locomotion in the mit cheetah 3 through convex model-predictive control”  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018, pp. 1–9  IEEE
  * [8] Yanran Ding, Abhishek Pandala and Hae-Won Park  “Real-time model predictive control for versatile dynamic motions in quadrupedal robots”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 8484–8490  IEEE
  * [9] Yan Duan et al.  “Benchmarking deep reinforcement learning for continuous control”  In _International conference on machine learning_ , 2016, pp. 1329–1338  PMLR
  * [10] Marc Finzi, Max Welling and Andrew Gordon Wilson  “A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups”  In _International Conference on Machine Learning_ , 2021, pp. 3318–3328  PMLR
  * [11] Robert J Full, Claire T Farley and Jack M Winters  “Musculoskeletal dynamics in rhythmic systems: a comparative approach to legged locomotion”  In _Biomechanics and neural control of posture and movement_ Springer, 2000, pp. 192–205
  * [12] H Geyer, U Saranli, A Goswami and P Vadakkepat  “Gait based on the spring-loaded inverted pendulum”  In _Humanoid Robotics: A Reference, A. Goswami and P. Vadakkepat, Eds. Springer Netherlands_ , 2018, pp. 1–25
  * [13] Hartmut Geyer  “Simple models of legged locomotion based on compliant limb behavior — Grundmodelle pedaler Lokomotion basierend auf nachgiebigem Beinverhalten”, 2005
  * [14] Kevin Green et al.  “Learning Spring Mass Locomotion: Guiding Policies with a Reduced-Order Model”  In _IEEE Robotics and Automation Letters_ 6.2 IEEE, 2021, pp. 3926–3932
  * [15] F. Grimminger et al.  “An Open Torque-Controlled Modular Robot Architecture for Legged Locomotion Research”  In _IEEE Robotics and Automation Letters_ 5.2, 2020, pp. 3650–3657
  * [16] Tuomas Haarnoja et al.  “Learning to walk via deep reinforcement learning”  In _arXiv preprint arXiv:1812.11103_ , 2018
  * [17] Tuomas Haarnoja et al.  “Soft actor-critic algorithms and applications”  In _arXiv preprint arXiv:1812.05905_ , 2018
  * [18] Roland Hafner et al.  “Towards general and autonomous learning of core skills: A case study in locomotion”  In _arXiv preprint arXiv:2008.12228_ , 2020
  * [19] Philip Holmes, Robert J Full, Dan Koditschek and John Guckenheimer  “The dynamics of legged locomotion: Models, analyses, and challenges”  In _SIAM review_ 48.2 SIAM, 2006, pp. 207–304
  * [20] Yifeng Jiang, Tom Van Wouwe, Friedl De Groote and C Karen Liu  “Synthesis of biologically realistic human motion using joint torque actuation”  In _ACM Transactions On Graphics (TOG)_ 38.4 ACM New York, NY, USA, 2019, pp. 1–12
  * [21] Joonho Lee et al.  “Learning quadrupedal locomotion over challenging terrain”  In _Science robotics_ 5.47 Science Robotics, 2020
  * [22] Tianyu Li et al.  “Learning generalizable locomotion skills with hierarchical reinforcement learning”  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 413–419  IEEE
  * [23] Ying-Sheng Luo, Jonathan Hans Soeseno, Trista Pei-Chun Chen and Wei-Chao Chen  “Carl: Controllable agent with reinforcement learning for quadruped locomotion”  In _ACM Transactions on Graphics (TOG)_ 39.4 ACM New York, NY, USA, 2020, pp. 38–1
  * [24] Li-Ke Ma et al.  “Learning and Exploring Motor Skills with Spacetime Bounds”  In _Computer Graphics Forum_ 40.2, 2021, pp. 251–263  Wiley Online Library
  * [25] Tien Mai, Kennard Chan and Patrick Jaillet  “Generalized Maximum Causal Entropy for Inverse Reinforcement Learning”  In _arXiv preprint arXiv:1911.06928_ , 2019
  * [26] AE Minetti and R McN Alexander  “A theory of metabolic costs for bipedal gaits”  In _Journal of Theoretical Biology_ 186.4 Elsevier, 1997, pp. 467–476
  * [27] Daniel Felipe Ordoñez Apraez  “Learning to run naturally: guiding policies with the Spring-Loaded Inverted Pendulum”, 2021
  * [28] David E Orin and Ambarish Goswami  “Centroidal momentum matrix of a humanoid robot: Structure and properties”  In _2008 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2008, pp. 653–659  IEEE
  * [29] Xue Bin Peng, Pieter Abbeel, Sergey Levine and Michiel Panne  “Deepmimic: Example-guided deep reinforcement learning of physics-based character skills”  In _ACM Transactions on Graphics (TOG)_ 37.4 ACM New York, NY, USA, 2018, pp. 1–14
  * [30] Xue Bin Peng et al.  “Learning agile robotic locomotion skills by imitating animals”  In _arXiv preprint arXiv:2004.00784_ , 2020
  * [31] Brahayam Ponton et al.  “On time optimization of centroidal momentum dynamics”  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 5776–5782  IEEE
  * [32] Herman Pontzer  “Effective limb length and the scaling of locomotor cost in terrestrial animals”  In _Journal of Experimental Biology_ 210.10 Company of Biologists, 2007, pp. 1752–1761
  * [33] Richard S Sutton and Andrew G Barto  “Reinforcement learning: An introduction”  MIT press, 2018
  * [34] Zhaoming Xie et al.  “GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model”  In _arXiv preprint arXiv:2104.09771_ , 2021
  * [35] Wenhao Yu, Greg Turk and C Karen Liu  “Learning symmetric and low-energy locomotion”  In _ACM Transactions on Graphics (TOG)_ 37.4 ACM New York, NY, USA, 2018, pp. 1–12
  * [36] Yi Zhou et al.  “On the continuity of rotation representations in neural networks”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 5745–5753
  * [37] Brian D Ziebart  “Modeling purposeful adaptive behavior with the principle of maximum causal entropy”  Carnegie Mellon University, 2010
  * [38] Martin Zinkevich and Tucker Balch  “Symmetry in Markov decision processes and its implications for single agent and multi agent learning”  In _In Proceedings of the 18th International Conference on Machine Learning_ , 2001  Citeseer