Learning Quadruped Locomotion using Bio-Inspired Neural Networks with
  Intrinsic Rhythmicity
Biological studies reveal that neural circuits located at the spinal cord
called central pattern generator (CPG) oscillates and generates rhythmic
signals, which are the underlying mechanism responsible for rhythmic locomotion
behaviors of animals. Inspired by CPG's capability to naturally generate
rhythmic patterns, researchers have attempted to create mathematical models of
CPG and utilize them for the locomotion of legged robots. In this paper, we
propose a network architecture that incorporates CPGs for rhythmic pattern
generation and a multi-layer perceptron (MLP) network for sensory feedback. We
also proposed a method that reformulates CPGs into a fully-differentiable
stateless network, allowing CPGs and MLP to be jointly trained with
gradient-based learning. The results show that our proposed method learned
agile and dynamic locomotion policies which are capable of blind traversal over
uneven terrain and resist external pushes. Simulation results also show that
the learned policies are capable of self-modulating step frequency and step
length to adapt to the locomotion velocity.
Chuanyu Yang, Can Pu, Tianqi Wei, Cong Wang, Zhibin Li
# Learning Quadruped Locomotion using Bio-Inspired Neural Networks with Intrinsic Rhythmicity

###### Abstract

Biological studies reveal that neural circuits located at the spinal cord called central pattern generator (CPG) oscillates and generates rhythmic signals, which are the underlying mechanism responsible for rhythmic locomotion behaviors of animals. Inspired by CPG's capability to naturally generate rhythmic patterns, researchers have attempted to create mathematical models of CPG and utilize them for the locomotion of legged robots. In this paper, we propose a network architecture that incorporates CPGs for rhythmic pattern generation and a multi-layer perceptron (MLP) network for sensory feedback. We also proposed a method that reformulates CPGs into a fully-differentiable stateless network, allowing CPGs and MLP to be jointly trained with gradient-based learning. The results show that our proposed method learned agile and dynamic locomotion policies which are capable of blind traversal over uneven terrain and resist external pushes. Simulation results also show that the learned policies are capable of self-modulating step frequency and step length to adapt to the locomotion velocity.

## I Introduction

Animals are able to adapt their locomotion gait pattern to suit the locomotion velocity and ground condition. Efforts have been put into discovering the underlying mechanism of animal locomotion, and have obtained evidence that legged locomotion is rhythmic in nature. Findings have revealed evidence of special neurons called central pattern generators in the animal spinal cord. CPGs are a series of coupled oscillatory neurons capable of producing rhythmic signals internally without external sensory input. The rhythmic signals produced by CPGs are necessary for activities that involve periodic movement, such as breathing, walking, running, swimming, flying, etc. [1, 2].

In recent years, there has been significant progress in using deep reinforcement learning (DRL) to train artificial neural networks for legged locomotion. Multi-expert neural networks can be designed to perform various different locomotion tasks [3]. Robust control policy has also been trained to successfully traverse over extreme terrain in the wild [4]. The neural networks from both works rely on an external periodic phase input to produce rhythmic motor patterns. Apart from a single global phase, multiple local phases can be provided, e.g. providing four phases as input to match the four individual legs of the quadruped [5].

Instead of providing external periodic phase input, there have been attempts in designing network structures capable of self-generating phases. One approach is to add an additional phase increment output to the policy. The phase increment is added to the current phase and is used as the phase for the next timestep in a bootstrap manner [6, 7, 4]. To reproduce the agile locomotion behaviors of animals in legged robots, researchers have looked into animals for inspiration. Inspired by findings of CPGs within the animal nervous system, there have also been attempts in utilizing the intrinsic rhythmicity of CPGs to generate rhythmic motor patterns for locomotion controllers in robotics, eliminating the need for external phase input [8, 9, 10].

This work aims to utilize the natural rhythmicity of CPGs to generate agile and dynamic locomotion behaviors for quadruped robots. While CPGs by themselves are capable of generating open-loop rhythmic signals, sensory feedback has to be integrated to enable the CPGs to react to external environment. We propose a framework that combines MLP networks and CPGs, where the CPGs are responsible for generating rhythmic patterns, and the MLP receives sensory information and provides feedback to modulate the CPGs' behavior. Our proposed framework reformulates CPGs by exposing the internal states as inputs and outputs of the network, essentially turning CPGs into a fully-differentiable stateless network. The parameters of the MLP and CPGs are optimized in conjunction via DRL.

The contributions of this paper are:

* We proposed a network architecture that combines CPGs and MLP network to generate rhythmic motions intrinsically without relying on external phase input. We name the network architecture as MLP-CPG.
* We proposed an approach that reformulates CPGs into

Fig. 1: Depiction of the agile and dynamic locomotion learned by the MLP-CPG network architecture.
fully-differentiable networks. The CPG parameters and MLP parameters within the MLP-CPG network architecture can be jointly optimized with DRL.
* We implemented the network architecture on a simulated quadruped robot and demonstrated the policy's capability to track user targets, and the robustness to environmental uncertainties.
* The proposed network architecture is interpretable. Locomotion characteristics such as step frequency, step length, and gait pattern can be inferred from the CPG parameters.

## II Related Work

### _CPG-Based Locomotion for Legged Robots_

CPGs have gained interest within the robotics field due to their oscillatory and rhythmic nature. To study the rhythmic mechanism of biological CPGs, researchers have proposed different CPG models with varying levels of complexity, from detailed biophysical models, connectionist models, to abstract models based on mathematical oscillators [2, 11].

Researchers have attempted to leverage the natural rhythmic behaviors of CPGs by incorporating CPGs into the controller to achieve animal-like locomotion for all kinds of legged robots, including bipeds, quadrupeds, and hexapods. Various abstract models of mathematical oscillators such as SO(2) oscillator, Matsuoka oscillator, and Hopf oscillator have been employed within the locomotion controller for different robots [12, 13, 14].

### _Learning CPG-Based Controllers with DRL_

DRL has proven to be effective in learning robust and complex locomotion skills for legged robots in simulation and real world. There are many works that have combined CPGs together with MLP networks and trained using the DRL paradigm. However, due to the intrinsic recurrent nature of CPG networks, back-propagation through time (BPTT) is required to obtain the parameters of CPGs while trained with gradient-based learning, which is computationally more expensive [15, 10].

One approach of dealing with the recurrent nature of CPGs is to treat the CPG controller as part of the environment dynamics. Such approach is referred to as CPG-Actor-Critic, where the CPG controller, the robot, and the environment are treated as a single dynamic system called CPG-coupled system. The action space of the policy is the CPG parameters. The policy controls the robot by adjusting the CPG parameters of the CPG-coupled system. [8, 9, 10].

Another approach of dealing with the recurrent nature of CPGs is to optimize the parameters of CPGs and MLP network separately. The parameters of the neural network can be optimized with DRL, while the parameters of CPG are optimized using non-gradient based approaches such as evolutionary strategy (ES) [16], genetic algorithm (GA) [17], or a biologically plausible learning rule [18, 19]. Wang et al. proposed a hierarchical control structure with a high-level neural network and a low-level CPG controller. The high-level network generates latent variables that are fed into the CPGs to regulate their behaviors. The CPG controller is optimized using GA, while the neural network is trained using DRL [17]. Shi et al. used ES to train a CPG-based foot trajectory generator. A separate neural network is trained using DRL to generate residual joint angles that correct the generated foot trajectory [16].

Campanaro et al. proposed an approach that dealt with the recurrent properties of CPGs by reformulating the CPG into a fully-differentiable stateless network, and thus allowing end-to-end training of CPGs alongside MLP. Campanaro et. al. have only validated the approach on a simple example of a single two degrees-of-freedom (DoF) hopping leg [15]. We extended upon the idea of reformulating CPG network into a stateless network and proposed our MLP-CPG framework. Furthermore, we have successfully implemented on a 12-DOF simulated quadruped robot.

## III Method

### _Robot Platform and Simulation setup_

The Jueying quadruped robot is used as the robot platform (Table I). Pybullet is chosen as the physics engine for the simulation environment.

### _CPG Controller_

The CPGs within this work are modelled using Hopf oscillators [14].

\[\begin{split}\dot{\theta}_{i}^{t}&=2\pi f^{t}+\sum _{j}\epsilon_{ij}sin(\theta_{j}^{t-1}-\theta_{i}^{t-1}-\phi_{ij})+\xi_{i}^{t} \\ \dot{r}_{i}^{t}&=\gamma r_{i}^{t-1}\big{(}(\eta_{i} +\kappa_{i}^{t})^{2}-(r_{i}^{t-1})^{2}\big{)}\\ \psi_{i}^{t}&=r_{i}^{t}cos(\theta_{i}^{t})+\chi_{i} ^{t}+o_{i},\end{split} \tag{1}\]

w

Fig. 2: An example consisting of 2 coupled CPG oscillators. The values within \(\epsilon_{ij}\), \(\gamma\), and \(dt\) are set to 6.0, 5.0, and 0.01. The values within \(\eta,\chi,o,\phi\) are set to 0. (A) and (B) illustrate how the phase and amplitude of the CPG can be adjusted by feedback component \(\xi\) and \(\kappa\).
where \(\theta_{i}^{t}\) and \(r_{i}^{t}\) represent the phase and amplitude of the i-th oscillator at timestep \(t\), \(\dot{\theta}_{i}^{t}\) and \(\dot{r}_{i}^{t}\) are their corresponding derivative, \(\epsilon_{ij}\) and \(\phi_{ij}\) are the coupling weight and phase bias between i-th and j-th oscillator, \(\eta_{i}\) is the desired amplitude, \(o_{i}\) is a constant offset of the oscillation setpoint, \(\gamma\) is a constant that determines the rising time for \(r\), \(\psi_{i}^{t}\) represents the output signal. Finally, \(\kappa_{i}^{t}\), \(\chi_{i}^{t}\), \(\xi_{i}^{t}\), \(f^{t}\) are the feedback components, where \(\kappa_{i}^{t}\) adjusts the amplitude, \(\xi_{i}^{t}\) adjusts the phase, \(\chi_{i}^{t}\) adjusts the oscillation setpoint, \(f^{t}\) adjusts the frequency of the oscillator at timestep \(t\).

The phase and amplitude \(\theta_{i}^{t}\) and \(r_{i}^{t}\) are obtained from their derivative values \(\dot{\theta}_{i}^{t}\) and \(\dot{r_{i}^{t}}\) using the following equation.

\[\theta_{i}^{t}=\theta_{i}^{t-1}+\dot{\theta}_{i}^{t}dt,\ r_{i}^{t}=r_{i}^{t-1}+ \dot{r}_{i}^{t}dt, \tag{2}\]

where \(dt\) is the duration of the timestep.

### _Reformulating Hidden States in CPGs_

Within Recurrent Neural Networks (RNN), the hidden states of the previous timestep has to be passed to the current timestep and combined with the input of the current timestep to compute the current output. RNN requires BPTT to be trained. CPGs can be viewed as a type of RNN, as they require hidden states to propagate information through time. The transition between consecutive hidden states of a CPG is determined by eq (1) and eq (2).

We dealt with the recurrent nature of CPGs by proposing a reformulation approach that exposes the hidden states through network inputs and outputs. The previous hidden states will be passed as network inputs, and the current hidden states will be passed as network outputs. With our proposed reformulation approach, the CPG network is essentially transformed into a fully-differentiable stateless feedforward network (Fig. 3C).

In contrast to previous work where the parameters of the CPG controllers and MLP network are optimized with separate algorithms, our reformulated CPG network is differentiable and can be optimized jointly with MLP network within the same DRL paradigm without BPTT [17, 16].

### _Network Architecture_

The MLP-CPG network consists of two separate modules, one rhythmic module responsible for generating rhythmic signals, and another non-linear feedback module responsible for processing sensory feedback (Fig. 4).

#### Iv-D1 Rhythmic module

The rhythmic module contains 12 CPG neurons, each neuron represents a joint. The values of the CPG parameters are shown in Table II.

#### Iv-D2 Feedback module

The feedback module is an MLP network with 2 hidden layers of size 256 with \(\tanh\) activation function. The MLP receives state inputs and regulates the CPG network via feedback components \(f\), \(\mathbf{\kappa}\), \(\mathbf{\xi}\), \(\mathbf{\chi}\). The MLP also outputs the covariance of the stochastic policy \(\mathbf{\sigma}\).

The output of the MLP-CPG network \(\mathbf{\psi}\) is bounded within \((-1,1)\) using \(\tanh\), and re-scaled to the corresponding joint limits to be used as the target joint angles.

### _Training Setup_

#### Iv-E1 Soft Actor Critic

We select Soft Actor Critic (SAC) as the DRL algorithm for the training of our policy. SAC learns

Fig. 4: Control framework overview. The MLP-CPG network generates target joint positions at 25Hz. The PD controller functions at 1000Hz and converts the joint angles to joint torques.

Fig. 3: Illustration of RNN and CPGs. (A) Unfolded RNN. (B) Unfolded CPGs. (C) Our reformulation approach transforms CPGs into stateless networks by exposing the hidden states as network inputs and outputs.
a policy by maximizing the expected return and the entropy. The optimization objective can be expressed as follows:

\[J_{SAC}(\pi)=\sum_{t=0}^{T}\mathbb{E}_{(s_{t},a_{t})\sim\rho_{\pi}}[r(s_{t},a_{t}) +\alpha H(\pi(\cdot|s_{t}))] \tag{3}\]

where \(\pi\) is the policy, \(\rho_{\pi}\) is the sample distribution, \(r\) is the reward, \(s_{t}\) and \(a_{t}\) are the state and action at time step \(t\) within the sample distribution, \(\alpha\) is the temperature parameter, and \(H(\pi)\) is the entropy. The temperature parameter determines the stochasticity of the policy and is tuned automatically during training to balance exploration and exploitation. The hyperparameters can be seen in Table III

#### Iii-B2 Generating smooth action

We introduce temporal and spatial regularization terms to encourage the learning of smoother actions [20]. We only regularize the setpoint feedback component \(\chi\) of the network output.

\[L_{T}=\left\|\chi(s_{t})-\chi(s_{t+1})\right\|_{2}^{2},L_{S}=\left\|\chi(s_{t} )-\chi(\hat{s}_{t})\right\|_{2}^{2} \tag{4}\]

The spatial regularization loss minimizes the difference between the action generated under observed state \(s_{t}\) and perturbed state \(\hat{s}_{t}\sim\mathcal{N}(s_{t},\delta)\), where \(\delta\) is the standard deviation. The temporal regularization loss minimizes the difference between the action under current state \(s_{t}\) and the next state \(s_{t+1}\).

#### Iii-B3 Step frequency

Multiple researches have shown that animals adjust their step frequency for different locomotion velocities [21, 22]. Prior knowledge of the step frequency can be embedded into the controller. Lee et al. manually designed a state machine that switches between different base frequencies depending on user velocity command [4].

We provided prior knowledge of step frequency using the curve function \(f_{ref}=a+b\ln v\) calculated from real animal data, where \(f_{ref}\) is the step frequency, \(v\) is the locomotion velocity, \(a=1.314\) and \(b=0.762\) are the parameters [21]. We fitted a new curve that passes through the origin, as we want zero frequency at zero velocity (Fig. 5):

\[\begin{split} f_{ref}&=a+b\cdot\ln{(v+c)},a=1.066,b= 0.876,c=0.289\\ v&=(\hat{v}_{x}^{2}+\hat{v}_{y}^{2})^{0.5}\end{split} \tag{5}\]

where \(\hat{v}_{x}\) and \(\hat{v}_{y}\) are the target forward and lateral velocity, respectively. The step frequency \(f\) is bounded within \([0,3]\). We introduce loss function \(L_{f}\) to encourage the MLP network to track the reference frequency \(f_{ref}\).

\[L_{f}=\left\|f(s_{t})-f_{ref}\right\|_{2}^{2} \tag{6}\]

The frequency loss \(L_{f}\), the temporal regularization loss \(L_{T}\), and spatial regularization loss \(L_{S}\) are added to the loss function of the SAC \(L_{SAC}=-J_{SAC}\).

\[L=L_{SAC}+\lambda_{T}L_{T}+\lambda_{S}L_{S}+\lambda_{f}L_{f}. \tag{7}\]

Fig. 5: Red dashed curve is obtained from animal data [21]. Green solid curve is obtained by fitting a new curve that passes through the origin.
### _Reward Design_

Radial basis function (RBF) kernels are used in the reward design. The output range of RBF is bounded within \((0,1)\). The formulation of the RBF kernel is shown as follows:

\[K(x,\widehat{x},c)=\exp\left(c(\hat{x}-x)^{2}\right) \tag{8}\]

where \(x\) is the physical quantity for the evaluation, \(\hat{x}\) is the desired value, and \(c\) is the parameter that controls the width of the RBF.

The reward is the weighted sum of individual reward terms, each governing a different physical aspect (Table IV and Table V). The Swing and stance foot reward penalizes the movement of the stance foot close to the ground while encouraging the movement of the swing foot far from the ground. The body placement reward encourages the Center of Mass (COM) of the robot to be placed close to the center of the support polygon. The foot placement reward encourages the feet to be placed close beneath the hip joint.

### _State Space and Action Space_

The state space consists of the observations from the environment \(S_{O}\), the internal states of the CPGs \(S_{I}\), and the goal states provided by the user \(S_{G}\).

The robot observation states consist of base linear velocity in robot heading frame \(\mathbf{v}_{xyz}\), base angular velocity \(\mathbf{\omega}_{rpy}\), orientation vector \(\mathbf{\varphi}_{b}\), joint position \(\mathbf{q}\), and joint velocity \(\hat{\mathbf{q}}\). The goal states consist of the target forward velocity \(\hat{v}_{x}\), target lateral velocity \(\hat{v}_{y}\), and the target yaw turning rate \(\hat{\omega}_{z}\). The internal states consist of the amplitude \(\mathbf{r}\) and phase \(\mathbf{\theta}\) of the 12 CPG oscillators. A mod operation is used to bound the phase within the range \([0,2\pi]\). The observation states \(S_{O}\) are filtered by a low-pass Butterworth filter with a cut-off frequency of 10\(Hz\).

The action space \(a_{t}\) contains the target joint angles \(\hat{q}\). A low-pass Butterworth filter with a cut-off frequency of 5\(Hz\) is applied to the target joint angles to encourage smooth actions. PD controllers are used to calculate joint torques \(\tau\) from the target joint angles \(\hat{q}\), measured joint angles \(q\), and measured joint velocities \(\hat{q}\) using the following equation \(\tau=K_{p}(\hat{q}-q)+K_{d}(0-\hat{q})\), where \(K_{p}=300\), \(k_{d}=10\).

### _Exploration Setup_

#### Iii-H1 Initialization

The initial forward velocity, lateral velocity, and yaw rate are sampled from \(U(-1,5)\), \(U(-1,1)\), and \(U(-\pi,\pi)\) respectively. The initial joints angles are sampled from \(\mathcal{N}(\tilde{q},\frac{\pi}{4})\), where \(\tilde{q}\) is the joint angle during nominal standing posture. The CPG amplitude and phase are initialized from \(U(0,\frac{\pi}{4})\) and \(U(0,2\pi)\), respectively. The target forward velocity, target lateral velocity, and target turning rate are initialized from \(U(-1,5)\), \(U(-1,1)\), \(U(\frac{\pi}{2},\frac{\pi}{2})\), respectively.

#### Iii-H2 Early termination

We terminate and restart the training episode when the robot reaches undesirable fail states. We define the fail state as the state where the body of the robot comes into contact with the ground or when the robot has a large body tilt. We also set a time limit of 10\(s\) to terminate the episode early.

## IV Results

### _Effect of Step Frequency_

Step frequency modulation behaviors have been observed within animals, where the step frequency increases alongside the locomotion velocity. We conduct a comparison study to investigate the effect of step frequency modulation on locomotion performance by comparing the following configurations:

1) _Fixed 1.5Hz_: The intrinsic frequency of the 12 CPGs are fixed to 1.5Hz.

2) _Fixed 3.0Hz_: The intrinsic frequency of the 12 CPGs are fixed to 3Hz.

2) _Adaptive Curve_: Prior knowledge of frequency is provided to encourage the policy to track the reference frequency calculated from eq (5).

3) _Adaptive Free_: No prior knowledge of frequency is provided.

#### Iv-B1 Locomotion velocity

Policies from all four configurations are able to perform successful locomotion while tracking user commanded velocity (Fig. 6A). However, the policy from _Fixed 1.5Hz_ configuration fails to track 4m/s target velocity, which might be due to the relatively low step frequency (Fig. 6D1).

Figure 6D2 shows the frequency modulation signal \(f\) provided by the MLP network. Both policies from _Adaptive Curve_ and _Adaptive Free_ configurations learned to increase their step frequency to reach higher locomotion velocity (Fig. 6C). While frequency modulation behavior naturally emerges without providing prior knowledge, providing prior knowledge of step frequency through a loss function leads to a smoother frequency modulation behavior.

#### Iv-B2 Energy efficiency

We utilize Cost of Transport (CoT) to measure the energy efficiency of the policies under different locomotion velocities. After comparing the policies from four configurations, we observed that policies with fixed frequency exhibit significantly higher CoT within the low-velocity region compared to policies capable of self-modulating frequency (Fig. 6B). This is due to unnecessarily high-frequency stepping.

#### Iv-B3 Step length

Animals adjust their step length to regulate their locomotion velocity [22]. We observed similar emergent behavior of step length modulation among all four policies. All policies have learned to increase the step length to reach higher locomotion velocities (Fig. 7A). Policies modulate the step length via the adjustments of the amplitude of the joint CPGs. The amplitude of the hip and knee CPG oscillators within all policies changes accordingly with locomotion velocity (Fig. 6D3 & D4). The change in amplitude will have a direct impact on the step length as can be seen in Fig. 7B and Fig. 7C, where larger CPG oscillation amplitude results in a larger step length. The step length for policy _Fixed 1.5Hz_ is higher compared to the other three policies to compensate for the relatively lower step frequency.

### _Interpretability of MLP-CPG Architecture_

A side benefit of our MLP-CPG network architecture is the additional interpretability compared to vanilla neural
networks. The frequency, amplitude, and phase of the CPG neurons are parameters that can be directly retrieved from the MLP-CPG network. As can be seen from Fig. 7 and Fig. 8, the frequency, amplitude, and phase of the CPGs correlate with the frequency of the neurons. The frequency of the neurons is \(2.
with step frequency, step length, and foot contact pattern, which are three important aspects of legged locomotion. By analyzing the values of the frequency, amplitude, and phase of the CPG neurons, we can easily interpret the locomotion characteristics of the robot.

### _Performance Benchmark_

#### Iv-C1 Target following

We design 4 sets of trajectories with varying difficulties for the robot to follow (Fig. 9). The robot is tasked to follow the trajectory by tracking a moving target. The relative position of the moving target is translated into velocity command for the robot. The policy from _Adaptive Curve_ is evaluated.

Given the target position and robot base position, we calculate the velocity commands as follows:

\[\hat{v}_{x}=-x_{target}^{base},\hat{v}_{y}=-y_{target}^{base},\hat{\omega}_{ yaw}=-\arctan(\frac{y_{target}^{base}}{x_{target}^{base}}), \tag{9}\]

where \(x_{target}^{base}\) is the x coordinate of the target in robot base frame, and \(y_{target}^{base}\) is the y coordinate of the target in robot base frame. The velocity targets are bounded within \([-1,4]\)\(m/s\), \([-0.75,0.75]\)\(m/s\), \([-1.0,1.0]\)\(rad/s\) for forward velocity, lateral velocity, and yaw angular rate. The learned _Adaptive Curve_ policy is able to follow all four trajectories.

### _Robustness Tests_

We design four sets of test scenarios to evaluate the robustness of the learned policies (Fig. 10). 1) Blind traversal over Uneven terrain. The uneven terrain consists of small bumps with height ranging from 0\(m\) to 0.1\(m\). 2) Persistent perturbation. External perturbation forces are applied to the robot every 4s through the impact of small cubes with mass of 5\(kg\). 3) Blind traversal over stairs. The stair consists of steps with height of 0.05m. 4) Carrying external load. A load with mass of 20\(kg\) (50% of the robot's own mass) is placed on the robot.

Policies from all four training configurations are able to traverse over uneven terrain and stairs, carry external load, and resist external perturbations (See accompanying video). The policies exhibit highly dynamic motions while reacting to external uncertainties in the environment. Note

Fig. 10: Overlaid snapshots of the _Adaptive Curve_ policy. (A) Uneven terrain. (B) External disturbance. (C) Stairs. (D) External load.

Fig. 9: (A1-A4) Trajectory following performance of policy _Adaptive Curve_. (A1) Square Wave. (A2) Cosine Wave. (A3) Eight Curve. (A4) Four-leaved Clover Curve. (B1-B4) The velocity tracking performance while following trajectory A1-A4.
that the policies are trained on flat ground and encountered no perturbations during the training phase.

## V Discussion and Future work

While our work demonstrated robust locomotion in simulation, it has not yet been implemented on a real robot. For future work, we plan to transfer the policy to a real robot system. Additionally, high-speed locomotion on real robot systems is challenging. Even though our policy is capable of achieving velocities up to 4m/s in simulation while satisfying motor constraints, having the joint torques and velocities constantly reaching the motor limits will inevitably cause a lot of stress and also overheat the motors. Mechanical stress and motor overheating need to be considered during sim-to-real transfer.

This paper has only focused on achieving rhythmic motor task of locomotion for quadruped robots. For future work, we will investigate the feasibility of using our proposed MLP-CPG network to achieve multi-tasking of both rhythmic motor tasks such as locomotion and non-rhythmic motor tasks such as fall recovery.

## VI Conclusion

In this paper, we proposed a bio-inspired network architecture called MLP-CPG that utilizes the intrinsic rhythmicity of CPGs to generate rhythmic motor patterns for locomotion without relying on external phase input. We proposed a reformulation approach that transforms CPGs into stateless networks by exposing the internal states. With our approach, the parameters of CPGs can be optimized jointly with the MLP network using DRL.

Our MLP-CPG has the advantage of interpretability. Locomotion characteristics such as step frequency, step length, gait pattern of the robot can be inferred by analyzing the values of the frequency, amplitude, and phase of the CPGs.

Our MLP-CPG network demonstrates emergent behaviors of step frequency and step length modulation. We observed that the learned policies increase the step frequency and step length accordingly as the locomotion velocity increases, demonstrating that the policy can actively alter locomotion patterns. The learned policy is not only capable of successfully performing the task of target following, but also capable of robustly traversing over uneven terrain and stairs, carrying external load, and resisting external perturbations.
