# Learning Perceptive Bipedal Locomotion over Irregular Terrain

Bart van Marum, Matthia Sabatelli, Hamidreza Kasaei  Department of Artificial
Intelligence
University of Groningen
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper we propose a novel bipedal locomotion controller that uses noisy
exteroception to traverse a wide variety of terrains. Building on the cutting-
edge advancements in attention based belief encoding for quadrupedal
locomotion, our work extends these methods to the bipedal domain, resulting in
a robust and reliable internal belief of the terrain ahead despite noisy
sensor inputs. Additionally, we present a reward function that allows the
controller to successfully traverse irregular terrain. We compare our method
with a proprioceptive baseline and show that our method is able to traverse a
wide variety of terrains and greatly outperforms the state-of-the-art in terms
of robustness, speed and efficiency.

##  I Introduction

Humanoid robots hold immense potential as a general-purpose platform for
various applications due to their compatibility with human-designed
environments. This compatibility enables humanoid robots to seamlessly work
alongside humans, reducing the need for expensive modifications to existing
infrastructure. Despite the benefits, creating a fully functional and general-
purpose humanoid robot still poses several technical challenges, including
locomotion over irregular and previously unseen terrain. To address this
challenge, the present work focuses on developing a robust and reliable
bipedal locomotion controller.

Conventionally bipedal locomotion controllers are designed as complicated
state machines and explicit dynamical models [1, 2]. However, these models
lack in robustness, do not generalize to new scenarios or terrains without
explicit modelling, and are laborious and complicated to develop and maintain.
Moreover, adding exteroceptive capabilities to such methods is not
straightforward.

In recent years, there has been a shift towards the use of Reinforcement
Learning (RL) [3] based controllers for simulated [4, 5], as well as real-
world bipedal robots [6, 7, 8, 9, 10]. These methods model the control policy
as a neural network and train them to maximize some reward signal. This
approach has proven to be robust, even in the face of motor malfunctions [7].

Many RL based approaches rely on the use of reference trajectories and
imitation rewards to train a policy to produce a gait[6], limiting the policy
to learn predetermined behaviour. However, recent work suggests that it is
possible to learn a wide variety of different bipedal gaits in a single neural
network by using periodic reward functions [8].

Figure 1:  In this work we develop a bipedal locomotion control policy based
on both exteroception and proprioception that is able to traverse a wide
variety of terrains. The first row shows Cassie walking over random terrain in
the physics simulator. The second row shows noisy exteroceptive samples that
are input to the policy at the same timesteps. The bottom shows the policy
architecture during inference.

Despite the great progress in recent years for neural network based bipedal
locomotion controllers, most approaches are compatible with flat terrain only.
Some successful attempts have been made to learn blind bipedal locomotion
controllers for more challenging terrains [10], however, these policies resort
to more conservative foot trajectories with higher steps and are unable to
avoid dangerous areas. Such blind strategies do not generalize well to a wide
variety of unseen and irregular terrains and lack in efficiency, and therefore
are not feasible for a fully capable humanoid robot.

In order for a legged locomotion controller to traverse any random previously
unseen terrain robustly, it needs information about the world ahead. A
controller that is based on proprioception only is limited to reactive and
precautionary behaviour. Only a controller that has information about the
world ahead can actively plan steps. Exteroceptive inputs are necessary,
however using both proprioception and exteroception presents a challenge.
Exteroceptive sensors such as cameras, Lidar, or radar often produce spurious
readings in cases such as reflection (water puddle), transparency (glass),
deformation (snow), or fake obstacles (tall grass). This may lead locomotion
policies based on such inputs to unnecessarily avoid certain areas or fail
outright, and raises the question of how to handle disagreeing proprioceptive
data.

The field of quadrupedal locomotion control has shown great progress in recent
years in learning controllers for navigating challenging terrain [11, 12].
Most notably, [13] shows that using a recurrent belief encoder with an
attention mechanism, a neural network policy is able to learn when to trust
and when not to trust the exteroceptive data. This allows the locomotion
controller to utilize exteroceptive data when it is most useful, and fall back
to proprioceptive data when it is not. Our work extends these methods to the
bipedal domain, resulting in a robust and reliable internal belief of the
terrain ahead despite noisy sensor inputs.

#### Contribution

In this work we apply a recurrent attention based belief encoder to a bipedal
locomotion policy to develop a robust controller capable of traversing
irregular terrain based on noisy exteroception. We present a reward function
leveraging prior work that allows the policy to learn traversing rough
terrain. We perform a wide range of simulation based experiments to show that
our controller is able to navigate a variety of terrains while outperforming
state-of-the-art proprioceptive controllers in terms of robustness,
efficiency, and speed. Figure 1 shows the architecture of our controller.

The remainder of the paper is organized as follows: The next section describes
the methods used to train our controller. Section III describes the training
process and the experiments performed to evaluate the performance of our
controller, including the results. Finally, Section IV concludes the paper.

##  II Methods

###  II-A Learning Setup

The main goal is to develop a robust bipedal locomotion controller that is
able to navigate irregular terrain while following a command. In order to do
so we use privileged learning [14] to distill a policy that is able to work
with potentially noisy and spurious exteroceptive observations. Previous work
has shown that directly learning the desired behavior over rough terrain with
RL does not converge within reasonable time budgets [11]. First a teacher
policy with access to perfect, noise free observations is trained in
simulation through reinforcement learning to traverse a wide range of
different terrains. We then train a student policy to imitate the behaviour of
the teacher policy, but without privileged information and noisy inputs. We
use Proximal Policy Optimization (PPO) [15] to train our policies, as PPO has
shown to yield good results in bipedal locomotion control [7, 8, 6].

###  II-B State and Action Representation

We define three observations $\mathbf{o}^{p}_{t},o^{e}_{t},o^{n}_{t}$. Here
$\mathbf{o}^{p}_{t}\in\mathbb{R}^{44}$ is the proprioceptive input, consisting
of motor positions, motor speeds, joint positions, joints speeds, pelvis
orientation, pelvis angular velocity, user commanded velocity $v_{c}$ and
clock inputs $i_{t}$. The user commanded velocity $v_{c}$ is defined as the
pair $(\mathbf{v}_{cmd},\omega_{cmd})$ where
$\mathbf{v}_{cmd}\in\mathbb{R}^{2}$ represents the commanded velocity in the
$x$ and $y$ directions and $\omega_{cmd}$ represents the commanded angular
velocity around the $z$ axis. The clock input $i_{t}$ is defined as the pair
$\sin(2\pi(\phi))$ and $\sin(2\pi(\phi+0.5))$, where $\phi$ is defined as
$\phi=(t\mod T)/T$, with $t$ denoting the current timestep, and $T$ a user
defined gait period in terms of timesteps. Although it has been noted in past
research [9] that clock inputs are necessary, we found in preliminary
experiments that a policy without clock inputs learns a gait with no
meaningful effect on the reward. However, we have not yet performed a thorough
investigation on this matter.

Additionally, $o^{e}_{t}=(\mathbf{e}^{l}_{t},\mathbf{e}^{r}_{t})$ is the pair
of noiseless exteroceptive observations for the left and right feet. To obtain
the observations the terrain height is sampled with a sampling pattern
centered at the location of the respective foot. The sampling pattern consists
of 318 points spaced circularly. Figure 2 shows the height sampling taking
place in the simulation environment.

Finally, $o^{n}_{t}=(\mathbf{n}^{l}_{t},\mathbf{n}^{r}_{t})$ is the pair of
noisy exteroceptive observations. Sampling is similar to $\mathbf{o}^{e}_{t}$
however a noise is applied to the sampling pattern coordinates and sampled
values. Further details on noise generation are discussed in section II-E.

The action $\mathbf{a}_{t}\in\mathbb{R}^{10}$ represents the PD targets for
the actuators in the robot model. Previous research has shown that PD targets
are an effective action parametrization for learning locomotion [16]. The
robot PD controller runs at 2 kHz, whereas actions are sampled from the policy
at a rate of 40 Hz.

|
---|---
(a) | (b)
Figure 2: (a) A close up view of the exteroceptive simulator. The red dots
represent the height sample for the right foot, and the green dots for the
left foot. Samples are taken with the sampling pattern centered around the
$xy$ position of each foot and rotated to match pelvis orientation. (b) Shows
a detailed plot of the pattern used to sample heights from the terrain.

###  II-C Policy Architecture

The teacher and student policy architectures are illustrated in Figure 3. In
this section, we will provide a detailed description of these architectures.

Figure 3: The top figure shows the teacher policy architecture which is
trained with PPO. The bottom figure shows the student policy architecture
which is trained to both imitate the action output of the teacher policy, and
to denoise the noisy exteroceptive input.

(a)

(b)

(c)

(d)

(e)

Figure 4: The five different terrain modes used for training: (a) hills, (b)
edges, (c) quantized hills, (d) squares, (e) stairs.

####  II-C1 Teacher Policy

The teacher policy $\pi^{t}$ consists of an exteroceptive encoder $f_{e}$ and
an LSTM [17]. The encoder $f_{e}$ consists of 3 fully connected layers of size
{256, 160, 96} and the LSTM has two layers of 256 nodes. The teacher policy
receives the observation $o^{t}_{t}=(\mathbf{o}_{t}^{p},\mathbf{o}_{t}^{e})$.
The exteroceptive encoder $f_{e}$ receives both exteroceptive observations
$\mathbf{e}_{t}^{l},\mathbf{e}_{t}^{r}$ that are in $o_{t}^{e}$ and encodes
them separately into the latent vectors $\mathbf{l}_{t}^{el}$ and
$\mathbf{l}_{t}^{er}$ which are concatenated into the latent vector
$\mathbf{l}_{t}^{e}\in\mathbb{R}^{200}$. The LSTM receives the concatenation
of $\mathbf{o}_{t}^{p}$ and $\mathbf{l}_{t}^{e}$ and outputs an action
$\mathbf{a}_{t}$.

####  II-C2 Student Policy

The student policy $\pi^{s}$ takes in the noisy observation
$o^{s}_{t}=(\mathbf{o}_{t}^{p},\mathbf{o}_{t}^{n})$ and has a partially
similar architecture as the teacher policy, using the same encoder $f_{e}$ and
LSTM. An added component is a recurrent belief encoder, which receives the
exteroceptive latent vector $\mathbf{l}^{e}_{t}$ and the proprioceptive
observation $\mathbf{o}_{t}^{p}$ and outputs a belief vector
$\mathbf{b}_{t}\in\mathbb{R}^{192}$. The belief vector is then concatenated
with the proprioceptive observation $\mathbf{o}_{t}^{p}$ and fed into the
LSTM, which in turn outputs an action $\mathbf{a}_{t}$.

The main aspect of the student policy is the recurrent belief encoder, an
approach introduced by [13], which is intended to take the proprioception and
noisy exteroception and develop an internal representation of what the terrain
looks like. In order to train this internal representation a belief decoder is
added to the policy, which takes as input the hidden state of the recurrent
belief encoder. The belief decoder outputs a reconstruction of the
exteroceptive inputs, which is trained to minimize the difference with the
noise free exteroceptive observation $o^{e}_{t}$. This method encourages the
internal hidden state of the belief encoder to represent a representation of
the outside world that is as accurate as possible, despite noisy inputs.
Additionally, the belief encoding system is fitted with an attention
mechanism, such that the policy is able to learn when exteroceptive data is
not useful, and rely on proprioception instead to construct the belief. For
more detail about the belief encoder and decoder we refer the reader to [13].

The student policy is trained with both an action imitation loss, and an
observation reconstruction loss to encourage the internal belief
representation of the outside world to be as accurate as possible.

###  II-D Terrain Generation

We use a linear curriculum to ramp terrain generation intensity. The ramp
starts after the policy has learned to walk on flat ground. All generated
terrains are modelled as a height map in meters and multiplied with the
curriculum factor $c_{t}\in[0,1]$.

We define five different terrain modes, as shown in Figure 4. The first is
hills, which is modelled as a sum of a low frequency and a higher frequency
Perlin noise [18]. The generated values are normalized to a range $[0,0.8]$.
The second terrain mode is edges, which consists of a Perlin noise that has
been quantized to two levels ${\\{0,h\sim\mathcal{U}(0.15,0.25)\\}}$. The
third mode is squares which consists of a grid of squares with sides
$d\in[0.4,0.6]$ of random height $h\in[0,0.4]$. The fourth mode is quantized
hills which is Perlin noise that has been quantized to discrete levels with a
random step size $h\in[0.12,0.18]$. The fifth and final mode of terrain
generation is stairs, consisting of alternating ascending and descending
staircases. To generate a staircase we randomly select a run $d\in[0.3,0.4]$
and rise $r\in[0.1,0.22]$ for 10 equal stairs.

###  II-E Randomization

Although we only focus on simulation based experiments in this work, previous
work has shown that policies trained in simulation are able to bridge the sim-
to-real gap given proper domain randomization [9, 19]. Therefore we randomize
joint damping, body part masses and friction coefficients at the start of each
episode. We use the same parametrization as presented in [10].

Furthermore, we randomize the velocity command to expose the policy to a range
of different velocity commands during training. At the start of each episode
and at one random timestep during each episode a new velocity command is
sampled. The probability distribution for the velocity commands is shown in
Table I.

TABLE I: Velocity Commands Randomization $v_{x}$ | $v_{y}$ | $\omega_{z}$ | Probability
---|---|---|---
$0$ | $0$ | $0$ | $0.15$
$\pm 1$ | $0$ | $0$ | $0.42$
$0$ | $\pm 1$ | $0$ | $0.07$
$0$ | $0$ | $\pm 1$ | $0.025$
$\sim\mathcal{U}(-1,1)$ | $\sim\mathcal{U}(-1,1)$ | $\sim\mathcal{U}(-1,1)$ | 0.1

To obtain the noisy exteroceptive student observation $o^{n}_{t}$ we apply a
noise to the noise free exteroceptive teacher observation $o^{e}_{t}$. We
sample noises for the sampling coordinates $x,y$ and the sampled height values
$z$ at the episode, foot and timestep level. Additionally, we select random
points on the height sampling pattern and apply a large noise to simulate
outliers. For the parametrization of these noises we leverage three modes
{nominal, offset, noisy} defined in prior work by [13] who designed the nosing
method to mimic noise in real exteroceptive sensors.

###  II-F Reward

For the teacher policy to learn to follow arbitrary commands over arbitrary
terrain we use a number of reward terms that are divided into three main
categories: (1) gait, (2) command following and (3) smoothness. The full
reward function we use is defined as:

$r=(0.25r_{frc}+0.25r_{vel}+0.2)\cdot
c_{r}+(r_{air}+0.1r_{one})\cdot(1-c_{r})\\\
+0.2r_{v,xy}+0.2r_{\omega,z}+0.05r_{lov}+0.05r_{fo}\\\
+0.05r_{pm}+0.05r_{po}+0.025r_{t}+0.025r_{a}$

of which the components are explained in the next subsections.

####  II-F1 Gait rewards

We use a combination of two reward methods to learn a gait. The first is a
clock based reward function that oscilates between swing and stance modes in a
gait period, as introduced in [8]. In the swing phase the reward function will
penalize foot forces, while in the stance phase foot velocity is penalized. By
offsetting the clock functions for the left and right leg a gait can be
learned. Figure 5 shows the gait clocks.

Figure 5:  The gait clocks $k_{frc}(\phi)$ and $k_{vel}(\phi)$ for both feet.
The stance phases of both feet are offset by $\phi/2$, but overlap slightly,
producing a walking gait.

The foot force reward component $r_{frc}$ is defined as:

$r_{frc}=\tanh(\pi F_{l}k_{frc,l})+\tanh(\pi F_{r}k_{frc,r})$

where $F_{r}$ and $F_{l}$ represent the normalized norm of the foot forces on
the respective foot, and $k_{frc,l}$ and $k_{frc,r}$ are the respective foot
force gait clocks. The foot velocity reward component $r_{vel}$ is defined as:

$r_{vel}=\tanh(\pi v_{l}k_{vel,l})+\tanh(\pi v_{r}k_{vel,r})$

where $v_{r}$ and $v_{l}$ represent the normalized norm of the foot velocities
of the respective foot, and $k_{vel,l}$ and $k_{vel,r}$ are the respective
foot velocity gait clocks.

We find that this periodic reward function produces good quality gaits on flat
terrain, however performance on rough terrain is less satisfactory. We
hypothesize that this is due to the fixed cadence embedded in the gait clocks,
restricting the policy to a fixed gait period, and that a more flexible reward
function is better suited for rough terrain.

The second is a more flexible reward function that simply rewards foot airtime
[12] for both feet. The foot airtime reward component $r_{air}$ is defined as:

$r_{air}=\sum_{f=0}^{2}(\mathbf{t}_{air,f}-0.5)*\vmathbb{1}_{first\
contact,f}$

where $\mathbf{t}_{air}\in\mathbb{R}^{2}$ denotes the cumulative airtime
during the swing phase of both feet, and $\vmathbb{1}_{first\
contact}\in\mathbb{R}^{2}$ holds binary values indicating whether the swing
phase is ended by contact. To prevent the policy from learning to simply jump,
a component $r_{one}$ is added to reward standing on one foot:

$r_{one}=\vmathbb{1}_{single\ contact}$

where $\vmathbb{1}_{single\ contact}$ is a binary value indicating whether the
robot is standing on one foot. We find that this reward function leads to more
stable gaits on rough terrain, but convergence is slower. Therefore we start
the training process with the clock based reward function until a satisfactory
gait has been learned on flat terrain. We then switch to the airtime based
reward function by setting the reward curriculum factor $c_{r}$ from 1 to 0.

####  II-F2 Command following rewards

We employ similar command following rewards as [11, 13] with the goal of
maximizing velocity in a given direction. The velocity reward component
$r_{v,xy}$ is defined as:

$r_{v,xy}=\begin{cases}\exp(-2.5\cdot\left|\left|\mathbf{v}_{xy}\right|\right|^{2})&\left|\left|\mathbf{v}_{cmd}\right|\right|=0\\\
1&\mathbf{v}_{cmd}\cdot\mathbf{v}_{xy}\geq 1\\\
\exp(-2\cdot(\mathbf{v}_{cmd}\cdot\mathbf{v}_{xy}-1)^{2})&else\end{cases}$

where $\mathbf{v}_{xy}\in\mathbb{R}^{2}$ represents the linear velocity in the
$xy$ plane. The angular velocity reward component $r_{\omega,z}$ is defined
as:

$r_{\omega,z}=\begin{cases}\exp(-5\cdot\omega_{z}^{2})&\mathbf{\omega}_{cmd}=0\\\
1&\omega_{cmd}\cdot\omega_{z}\geq 1\\\
\exp(-2\cdot(\omega_{cmd}\cdot\omega_{z}-1)^{2})&else\end{cases}$

where $\omega_{z}$ represents the pelvis angular velocity. The linear
orthogonal velocity offset reward component $r_{lov}$ is defined as:

$r_{lov}=\exp(-5\cdot\left|\left|\mathbf{v}_{xy}-\mathbf{v}_{cmd}\cdot\mathbf{v}_{xy}\right|\right|)$

where $\mathbf{v}_{xy}\in\mathbb{R}^{2}$ again represents pelvis velocity in
the $xy$ plane. It is intended to penalize linear velocities orthogonal to the
commanded velocity.

####  II-F3 Smoothness rewards

The foot orientation reward component $r_{fo}$ is defined as:

$r_{fo}=\exp(-1.5\cdot(\mathbf{\hat{z}}\cdot\boldsymbol{\psi}_{lf}+\mathbf{\hat{z}}\cdot\boldsymbol{\psi}_{rf}))\cdot(1-c_{t})+c_{t}$

where $\mathbf{\hat{z}}\in\mathbb{R}^{3}$ represents the unit vector in the
$z$ direction and
$\boldsymbol{\psi}_{lf},\boldsymbol{\psi}_{rf}\in\mathbb{R}^{3}$ represent the
foot orientation vectors pointing along the length of both feet. This reward
component encourages the policy to keep the feet flat on the ground, but in
any planar direction. Additionally, the reward is gradually shifted to a
constant reward component as the curriculum is ramped up to allow the policy
to adapt to terrains where a non flat position might be more beneficial. The
pelvis motion reward component $r_{pm}$ is defined as:

$r_{pm}=\exp(-(v_{z}^{2}+\omega_{y}^{2}+\omega_{x}^{2}))$

and is intended to penalize pelvis motions in directions not part of the
command. The pelvis orientation reward component $r_{po}$ is defined as:

$r_{po}=\exp(-3\cdot(\left|\psi_{x}\right|+\left|\psi_{y}\right|))$

where $\psi_{x}$ and $\psi_{y}$ represent pelvis orientation and is intended
to encourage the policy to keep the pelvis level. The torque reward component
$r_{t}$ is defined as:

$r_{t}=\exp(-0.02\cdot\overline{\left|\boldsymbol{\tau}\right|})$

where $\boldsymbol{\tau}$ represents the torque vector exerted by the
actuators, with the aim to reduce energy consumption. The action reward
component $r_{a}$ is defined as:

$r_{a}=\exp(-5\cdot\overline{\left|\mathbf{a}_{t}-\mathbf{a}_{t-1}\right|})$

where $\mathbf{a}_{t}$ represents the action vector at time $t$ and is
intended to penalize large changes in the action vector in order to improve
smoothness and stability.

##  III Experimental Results

###  III-A Simulation

Training and experimentation are performed in simulation on the Cassie robot
[20]. We use the Mujoco [21] physics simulator with the cassie-mujoco-sim
environment [22].

###  III-B Training

For teacher policy training the recurrent PPO algorithm from StableBaselines3
(SB3) [23] is used. In order to achieve a 7 times speedup in terms of
timesteps per second we modified SB3 to use batches of whole sequences. The
hyperparameters for PPO are listed in Table IIa. Teacher policy training takes
around 36 hours for $60\times 10^{6}$ timesteps on a single 12-core V100 node.

The student policy is trained on a dataset of $10^{6}$ timesteps sampled from
the teacher policy, with the hyperparameters listed in Table IIb. The student
policy is implemented in PyTorch [24], and training takes around 1 hour on a
high end laptop.

TABLE II: Policy Training Hyperparameters

Parameter | Value
---|---
max episode length | 300
rollout buffer size | 50000
learning rate | $10^{-4}$
batch size | 8 seq.
epochs | 5
clip range | 0.2
GAE Lambda | 0.95
gamma | 0.99
(a) Teacher policy

Parameter | Value
---|---
max episode length | 300
learning rate | $0.001$
batch size | 12 seq.
epochs | 100
optimizer | Adam
(b) Student policy

In order to compare and quantify the performance of our exteroceptive student
policy we train a baseline policy $\pi^{b}$. This baseline policy observes
proprioception $\mathbf{o}_{t}^{p}$ only and has the same architecture as the
teacher policy but without the exteroceptive encoder, similar to the current
state-of-the-art [10, 8]. The baseline policy is trained on the same terrain
and curriculum as the exteroceptive student policy.

###  III-C Experiments

We conduct a number of simulation based experiments to evaluate the
performance of our exteroceptive policy against the proprioceptive baseline
policy. In all experiments our policy has access to nominal noise
exteroception, and is commanded to walk forward, unless otherwise mentioned.
For all results 100 episodes were attempted and the results averaged. We
include a supplementary video providing a visual representation of the
experiments. 111https://youtu.be/B3Qr-7ZZHZQ

####  III-C1 Maximum speed over various terrains

Figure 6: Policy performance metrics for the five different terrain modes as
well as flat terrain. Error bars denote the standard deviation of the mean.
Our exteroceptive policy achieves higher success rates and speeds as it can
take more decisive actions, thanks to its ability to gather information about
the environment in advance.

Figure 7: Success rate metrics for stepping over a single step (top), a trench
(middle) and walking downstairs (bottom). In all cases the exteroceptive
policy is able to traverse more challenging terrains than the proprioceptive
baseline policy.

We record mean speed, average actuator torque and the success rate for all
terrains in the training curriculum. A success is defined as the episode
completing at 300 timesteps without the robot falling over. The results are
shown in Figure 6. Our policy outperforms the baseline policy on all terrains
in terms of success rate, with near perfect scores on all terrains but stairs.
Although the proprioceptive baseline is able to achieve a near 100% success
rate on hills and flat terrain, its performance is worse on quantized hills,
edges and squares. The largest outperformance is found in the stairs terrain,
and upon further investigation we find that most failures occur during the
descent of the staircase. This underperformance of the baseline policy is
likely due to the large vertical change in the stairs, quantized hills, edges
and squares terrains, which the robot is unable to anticipate due to the lack
of exteroceptive information. To investigate this further we conduct a more
detailed analysis of the staircase terrain in the next experiments.

Our exteroceptive policy is able to achieve higher speeds than the baseline
policy on all terrains but stairs. This is due to the fact that the
exteroceptive policy is able to take more decisive actions, as it can gather
information about the environment in advance. The baseline policy is forced to
resort to more conservative gaits, at slower speeds and lifting feet higher.
The slow speed on stairs for our exteroceptive policy is caused by a
combination of caution and the policy veering slightly off course during stair
descent. This behaviour emerged from the training process and we hypothesize
that, although undesirable, it is beneficial for the policy success, as it
effectively lengthens the run of the stairs, making it easier to find a
suitable foothold.

Applied torque is lower for our policy than the baseline policy on all
terrains, indicating that less energy is consumed despite walking at higher
speeds. Our exteroceptive policy is therefore more energy efficient. However,
we did not investigate whether the difference is large enough to counteract
the added computational cost of the exteroceptive encoder.

####  III-C2 Success rate over a step

We command both policies to go forward over flat terrain with a 1 meter wide
step of varying heights at 1 meter from the starting position and record the
success rate. The results are shown in Figure 7. Our policy outperforms the
baseline policy, reliably traversing over steps up to 30 cm in height, while
the baseline policy starts to fail at 20 cm. This experiment clearly shows the
advantage of exteroceptive information in the case of a step, as the policy
can anticipate the step and take a more reliable action.

####  III-C3 Success rate over a trench

We command both policies to go forward over a 50 cm deep trench of varying
widths to gauge the effects of exteroception on the policy’s ability to avoid
dangerous areas. The results are shown in Figure 7. Surprisingly the
proprioceptive baseline is able to outperform our exteroceptive policy. We
find that the exteroceptive policy does not avoid the trench, but tries to
step inside. We hypothesize this is caused by the policy not encountering
terrains with dangerous areas during training.

####  III-C4 Success rate for stair descent

As can be seen from the success rate results in Figure 6, the stair terrain is
most difficult for both policies. Specifically, most failures occur when the
robot is descending stairs. We believe this is due to Cassies morphology since
the steep backwards incline of the legs make them interfere with the stairs.
Effectively causing the useable portion of a stair run to be shorter and
requiring more precision in foot placement. To investigate whether
exteroception is beneficial in this environment we command both policies to go
down staircases of 10 equal steps and vary the step heights while recording
the success rates. All staircases use the same run of 35 cm. The results are
shown in Figure 7. Our exteroceptive policy is able to walk down stairs with
step heights up to 16 cm with near 100% success rate, while the baseline
policy starts failing at 12 cm. This clearly shows the advantage of
exteroception in this environment. Figure 8 shows both policies traversing
down stairs of 12 cm step height.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

Figure 8: The top two rows shows our exteroceptive policy successfully walking
down a staircase of ten 12 cm steps, with the second row showing the noisy
exteroceptive inputs at the same timesteps. The bottom row shows the
proprioceptive baseline policy attempting to walk down the same staircase,
with less success.

Figure 9:  The policy is commanded to walk in all directions over the squares
terrain, and the commanded and actual linear and angular velocities are
recorded. All plots represent the mean velocity of 100 independent trials. The
shaded area denotes the standard deviation of the mean.

####  III-C5 Command Following

To gauge the capability of the policy to locomote over irregular terrain
according to a user specified command we command the policy to walk in all
directions over the squares terrain and record the speed. We choose the
squares terrain for this experiment for its high density of changes in terrain
height. Figure 9 shows the commanded linear and angular velocities and the
policies responses to them. The policies are able to follow the commands in
the $x$ direction over rough terrain accurately, albeit with a delay. Similar
results apply for speed achieved in the $y$ direction, however the maximum
speed is lower than commanded. We believe this is due to the low range of
motion available in the roll actuators of the robot hips, requiring smaller
steps and limiting velocity. We found that results are similar on flat
terrain, confirming that terrain is not the limiting factor. Lastly, both
policies are able to accurately follow the commanded angular velocity over the
squares terrain. These results clearly show that the policies have learned to
locomote over irregular terrain according to a user specified command, with
our exteroceptive policy slightly outperforming in terms of speed.

####  III-C6 Dealing with spurious exteroceptive inputs

An important aspect of the belief encoder system is the ability to interpret
noisy exteroceptive inputs along with proprioception to form an accurate
belief of the environment. In order to demonstrate this capability we show a
view of the exteroceptive reconstruction produced by the belief decoder in
Figure 10. Our policy is able to denoise large, sometimes alternating offsets
in the height map. Additionally, the belief encoder is able to eliminate
outliers, while keeping an accurate representation of the terrain.

Figure 10: A side view of the sampled height map and reconstructed height map
for the left foot during stair traversal. Cyan is the sampled noisy height
map, yellow is the reconstruction produced by the belief decoder. The policy
is able to denoise large offsets in the height map.

##  IV Discussion

In this work we presented a method to learn a bipedal locomotion policy that
can utilize exteroceptive observations to successfully traverse irregular
terrain, while following a command. We have shown that such a policy observing
exteroception greatly outperforms a purely proprioception based locomotion
policy when traversing irregular terrains. An exteroceptive policy is able to
achieve this outperformance on terrain while at the same time increasing
speed, stability and energy efficiency. Critically, we have shown that our
policy has learned to achieve such behavior while relying on noisy
exteroceptive observations, showcasing the robustness of the control policy.

Limitations of this work include the fact that our experiments have only been
conducted in simulation. Although most methods used in this work have been
proven to work on real robots in the past, future work should include testing
on a real robot to confirm results. Another limitation we observe is the low
speed of iteration in reward and curriculum design caused by the multiple day
training time of the policy. Future work could explore the use of faster
training methods to more effectively optimize the rewards and curriculum, such
as presented in [12].

## Acknowledgment

This work was done as part of the Master’s thesis of the first author. We
thank the authors of the cassie-mujoco-sim environment [22] for making their
code publicly available. Finally, we thank the Center for Information
Technology of the University of Groningen for their support and for providing
access to the Peregrine high performance computing cluster.

## References

  * [1] J. Reher, W.-L. Ma, and A. D. Ames, “Dynamic walking with compliance on a cassie bipedal robot,” in _2019 18th European Control Conference (ECC)_ , 2019, pp. 2589–2595.
  * [2] Y. Gong, R. Hartley, X. Da, A. Hereid, O. Harib, J.-K. Huang, and J. Grizzle, “Feedback control of a cassie bipedal robot: Walking, standing, and riding a segway,” in _2019 American Control Conference (ACC)_ , 2019, pp. 4559–4566.
  * [3] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_ , 2nd ed. The MIT Press, 2018. [Online]. Available: http://incompleteideas.net/book/the-book-2nd.html
  * [4] X. B. Peng, “Developing locomotion skills with deep reinforcement learning,” Ph.D. dissertation, University of British Columbia, 2017. [Online]. Available: https://open.library.ubc.ca/collections/ubctheses/24/items/1.0345638
  * [5] W. Yu, G. Turk, and C. K. Liu, “Learning symmetric and low-energy locomotion,” _ACM Trans. Graph._ , vol. 37, no. 4, jul 2018. [Online]. Available: https://doi.org/10.1145/3197517.3201397
  * [6] Z. Xie, G. Berseth, P. Clary, J. W. Hurst, and M. van de Panne, “Feedback control for cassie with deep reinforcement learning,” _CoRR_ , vol. abs/1803.05580, 2018. [Online]. Available: http://arxiv.org/abs/1803.05580
  * [7] Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath, “Reinforcement learning for robust parameterized locomotion control of bipedal robots,” _CoRR_ , vol. abs/2103.14295, 2021. [Online]. Available: https://arxiv.org/abs/2103.14295
  * [8] J. Siekmann, Y. Godse, A. Fern, and J. Hurst, “Sim-to-real learning of all common bipedal gaits via periodic reward composition,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 7309–7315.
  * [9] J. Siekmann, S. Valluri, J. Dao, L. Bermillo, H. Duan, A. Fern, and J. W. Hurst, “Learning memory-based control for human-scale bipedal locomotion,” _CoRR_ , vol. abs/2006.02402, 2020. [Online]. Available: https://arxiv.org/abs/2006.02402
  * [10] J. Siekmann, K. Green, J. Warila, A. Fern, and J. Hurst, “Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning,” in _Proceedings of Robotics: Science and Systems_ , Virtual, July 2021.
  * [11] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” _Science Robotics_ , vol. 5, no. 47, oct 2020. [Online]. Available: https://doi.org/10.1126%2Fscirobotics.abc5986
  * [12] N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in _5th Annual Conference on Robot Learning_ , 2021. [Online]. Available: https://openreview.net/forum?id=wK2fDDJ5VcF
  * [13] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” _Science Robotics_ , vol. 7, no. 62, jan 2022. [Online]. Available: https://doi.org/10.1126%2Fscirobotics.abk2822
  * [14] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl, “Learning by cheating,” _CoRR_ , vol. abs/1912.12294, 2019. [Online]. Available: http://arxiv.org/abs/1912.12294
  * [15] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1707.06347
  * [16] X. B. Peng and M. van de Panne, “Learning locomotion skills using deeprl: Does the choice of action space matter?” in _Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation_ , ser. SCA ’17. New York, NY, USA: Association for Computing Machinery, 2017. [Online]. Available: https://doi.org/10.1145/3099564.3099567
  * [17] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [18] A. Lagae, S. Lefebvre, R. Cook, T. DeRose, G. Drettakis, D. Ebert, J. Lewis, K. Perlin, and M. Zwicker, “A survey of procedural noise functions,” _Computer Graphics Forum_ , vol. 29, no. 8, pp. 2579–2600, 2010. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-8659.2010.01827.x
  * [19] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2017, pp. 23–30.
  * [20] Agility Robotics, accessed: 2022-12-07. [Online]. Available: https://www.agilityrobotics.com/
  * [21] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2012, pp. 5026–5033.
  * [22] Agility Robotics, “cassie-mujoco-sim,” _GitHub repository_ , 2018. [Online]. Available: https://github.com/osudrl/cassie-mujoco-sim
  * [23] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” _Journal of Machine Learning Research_ , vol. 22, no. 268, pp. 1–8, 2021\. [Online]. Available: http://jmlr.org/papers/v22/20-1364.html
  * [24] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” _CoRR_ , vol. abs/1912.01703, 2019. [Online]. Available: http://arxiv.org/abs/1912.01703


# On the Electrostatic Forces acting on Point Charges
in the Present of a Dielectric Slab

Long T. Nguyen  High School for Gifted Students, Hanoi University of Science,
Vietnam National University, 182 Luong The Vinh Str., Thanh Xuan, Hanoi
100000, Vietnam.  Tuan K. Do<EMAIL_ADDRESS>Department of Mathematics,
Princeton University, Princeton, NJ 08544, USA  Trung V. Phan
<EMAIL_ADDRESS>Department of Physics, Princeton University, Princeton,
NJ 08544, USA.

###### Abstract

How will the electrostatic interaction between two point charges change if we
introduce a dielectric slab in the middle of them? While the physical setting
of this problem is relatively simple, it is easy to be wronged and the
solution is surprisingly complicated. Here we will show the correct answer by
using the method of images, which should be approachable for undergraduate
students and advance high school students. We also obtain analytical and
algebraic results in some special cases.

##  I Problem

Determine the electrostatic force acting on two point charges $q_{1}$ and
$q_{2}$ placed in vacuum in the present of an infinitely large dielectric slab
of thickness $h$ and relative dielectric constant $\epsilon$ inserted in
between the charges. Given that the distance between the charges and the slab
are $d_{1}$ and $d_{2}$, and consider the simple case where the line
connecting the charges is perpendicular to the slab as shown in Fig. 1.

Figure 1: An infinitely large dielectric slab is inserted in between two point
charges.

In many undergraduate introductory and advance high school physics courses,
the students are taught that the electrostatic force $f$ between charges
inside a medium of relative dielectric constant $\epsilon$ is given by the
Coulomb’s equation:

$f=\frac{1}{4\pi\epsilon_{0}\epsilon}\frac{q_{1}q_{2}}{d^{2}}\ ,$ (1)

where $\epsilon_{0}$ is the dielectric constant of vacuum, $q_{1}$ and $q_{2}$
are two point charges of interests and $d$ is the distance between them. These
electrostatic forces acting on the two charges are opposite in direction but
equal in magnitude, and this value is the same as if the charges are distance
$\sqrt{\epsilon}d$ apart in vacuum. Due to that, when dealing with the setting
in Fig. 1, many students in correctly assume that the present of the
dielectric slab is no different than having the region of vacuum space it is
filling expands by the factor $\sqrt{\epsilon}$, thus arrive at:

$f=\frac{1}{4\pi\epsilon_{0}}\frac{q_{1}q_{2}}{\left(d_{1}+\sqrt{\epsilon}h+d_{2}\right)^{2}}\
.$ (2)

In this notes, we will show that not only this value is wrong, but also these
forces are non-reciprocal: the magnitudes of the forces acting on charge
$q_{1}$ and $q_{2}$ are in general not the same, $|f_{1}|\neq|f_{2}|$. The
image charges configuration and the calculation is quite non-trivial, and to
the best of our knowledge it has not been carried out and analyzed in much
details.

##  II Solution

The key insight dealing with this problem is to realize that the dielectric
slab doesn’t effectively expand vacuum space as viewed by electrostatic
interaction, but the freely-moving charges on the slab redistributed to screen
out electrical field so that the boundary conditions for the displacement
field are satisfied.

The standard dielectric image method is for the setting where the space is
filled half-by-half with mediums of different relative dielectric constant
$\epsilon_{1}$ and $\epsilon_{2}$, separated by an infinite-planar interface
(see Fig. 2a). Place a point charge $q$ inside region $\epsilon_{1}$, the
final distribution of free-moving charges in space will give electric field in
region $\epsilon_{1}$ equivalent to that of two charge: the original charge
$q$ and an image charge $q^{\prime}$ which
$q^{\prime}/q=(\epsilon_{1}-\epsilon_{2})/(\epsilon_{1}+\epsilon_{2})$ at the
mirror-reflected position of charge $q$ through the interface (see Fig. 2b),
while in in region $\epsilon_{2}$ it is equivalent to that of another image
charge $q^{\prime\prime}$ which
$q^{\prime\prime}/q=2\epsilon_{1}/(\epsilon_{1}+\epsilon_{2})$ at the same
position with $q$ (see Fig. 2c). Hammond (1960); Jackson (1999)

Figure 2: (a) The space is divided into two regions having different
dielectric constants by an infinite-planar interface. A point charge $q$ is
introduced inside on of these regions. (b) The electrical field seen inside
the same region as the charge $q$ is equivalent to the electrical field
produced by two point charges: $q$ and image charge $q^{\prime}$. (c) The
electrical field seen inside the different region as the charge is equivalent
to the electrical field produced by one point charge: image charge
$q^{\prime\prime}$

We consider the electrical field created by a system placed in vacuum consists
of a point charge $q_{1}$ at distance $d_{1}$ from an infinitely large
dielectric slab of thickness $h$ and relative dielectric constant $\epsilon$.
The slab divides space into three regions: region $1$ where the charge $q_{1}$
is in, region $2$ where the slab is filling and region $3$ has the rest of the
space. Using the method of images to deal consecutively with the two infinite-
planar interfaces, we obtain an infinite series of image charges for any
region. The electrical field seen inside the region $1$ is equivalent to the
electrical field produced by an infinite series of point charges: $q_{1}$, an
image charge $q_{1}^{\prime}$ placed outside region $1$ at distance $2d_{1}$
and image charges $q_{1}^{(1,k)}$ where $k=0,1,2,...$ placed outside region
$1$ at distance $2(d_{1}+h)+2kh$ (see Fig. 3b):

$q_{1}^{\prime}=-\frac{\epsilon-1}{\epsilon+1}q_{1}\ ,\
q_{1}^{(1,k)}=\epsilon\left(\frac{2}{\epsilon+1}\right)^{2}\left(\frac{\epsilon-1}{\epsilon+1}\right)^{2k+1}q_{1}\
.$ (3)

The electrical field seen inside the region $0$ is equivalent to the
electrical field produced by an infinite series of point charges: image
charges $q_{1}^{(0,k)}$ where $k=0,2,4,...$ placed inside region $1$ at
distance $kh$ and image charges $q_{1}^{(0,k)}$ where $k=1,3,5,...$ placed
outside region $1$ at distance $2(d_{1}+h)+(k-1)h$ (see Fig. 3c):

$q_{1}^{(0,k)}=\left(\frac{2}{\epsilon+1}\right)\left(\frac{\epsilon-1}{\epsilon+1}\right)^{k}q_{1}\
.$ (4)

The electrical field seen inside the region $2$ is equivalent to the
electrical field produced by an infinite series of point charges: image
charges $q_{1}^{(0,k)}$ where $k=0,1,2,...$ placed inside region $1$ at
distance $2kh$ (see Fig. 3d):

$q_{1}^{(2,k)}=\epsilon\left(\frac{2}{\epsilon+1}\right)^{2}\left(\frac{\epsilon-1}{\epsilon+1}\right)^{2k}q_{1}\
.$ (5)

Figure 3: (a) The space is divided into three regions by the infinitely large
dielectric slab. A point charge $q_{1}$ is introduced outside of the slab. (b)
The electrical field seen inside the same region as the charge $q$ is
equivalent to the electrical field produced by an infinite series of point
charges: $q_{1}$ and image charges $q_{1}^{\prime}$, $q_{1}^{(1,0)}$,
$q_{1}^{(1,1)}$, $q_{1}^{(1,2)}$, … (c) The electrical field seen inside the
slab is equivalent to the electrical field produced by an infinite series of
point charges: image charges $q_{1}^{(0,0)}$, $q_{1}^{(0,1)}$,
$q_{1}^{(0,2)}$, … (d) The electrical field seen inside the vacuum region
without charge $q_{1}$ is equivalent to the electrical field produced by an
infinite series of point charges: image charges $q_{1}^{(2,0)}$,
$q_{1}^{(2,1)}$, $q_{1}^{(2,2)}$, …

###  II.1 The general case

Now we have enough ingredient to solve the original problem by considering the
superposition of all the charges and image charges as we introduced charge
$q_{2}$ at region $2$ distance $d_{2}$ away from the dielectric slab (see Fig.
4a). For example, the electrical as seen in region $1$ is equivalent to the
electrical field produced by not only charge $q_{1}$ and image charges
$q_{1}^{\prime}$, $q_{1}^{(1,k)}$ where $k=0,1,2,...$ but also $q_{2}^{(1,k)}$
placed inside region $2$ at distance $(d_{1}+h+d_{2})+2kh$ away from charge
$q_{1}$ (see Fig. 4b):

$q_{2}^{(1,k)}=\epsilon\left(\frac{2}{\epsilon+1}\right)^{2}\left(\frac{\epsilon-1}{\epsilon+1}\right)^{2k}q_{2}\
.$ (6)

Thus the electrostatic force $f_{1}$ acting on charge $q_{1}$ can be
calculated as an infinite summation series:

$\begin{split}f_{1}=\frac{q_{1}}{4\pi\epsilon_{0}}\Bigg{(}\frac{q_{1}^{\prime}}{(2d_{1})^{2}}&+\sum^{\infty}_{k=0}\frac{q_{1}^{(1,k)}}{\big{(}2(d_{1}+h)+2kh\big{)}^{2}}\\\
+&\sum^{\infty}_{k=0}\frac{q_{2}^{(1,k)}}{\big{(}(d_{1}+h+d_{2})+2kh\big{)}^{2}}\Bigg{)}\
.\end{split}$ (7)

Define the ratios $\alpha_{1}=d_{1}/h$, $\alpha_{2}=d_{2}/h$ and define the
summation $F_{a}(z)$:

$S_{a}(z)=\sum^{\infty}_{k=0}\frac{z^{2k}}{(a+2k)^{2}}\ ,$ (8)

we can rewrite the force equation as:

$\begin{split}&f_{1}=\frac{q_{1}}{4\pi\epsilon_{0}h^{2}}\Bigg{(}-q_{1}\left(\frac{\epsilon-1}{\epsilon+1}\right)\frac{1}{(2\alpha_{1})^{2}}\\\
&\ \ \ \
+q_{1}\epsilon\left(\frac{2}{\epsilon+1}\right)^{2}\left(\frac{\epsilon-1}{\epsilon+1}\right)S_{2(\alpha_{1}+1)}\left(\frac{\epsilon-1}{\epsilon+1}\right)\\\
&\ \ \ \
+q_{2}\epsilon\left(\frac{2}{\epsilon+1}\right)^{2}S_{\alpha_{1}+1+\alpha_{2}}\left(\frac{\epsilon-1}{\epsilon+1}\right)\
.\end{split}$ (9)

Similarly, we get $f_{2}$ acting on charge $q_{2}$, which magnitude can be
shown to be different from that of $f_{1}$ in general. This means the
interaction between these two charges mediated by the dielectric slab is non-
reciprocal.

The result here can also by found by solving the Poisson equation.

Figure 4: (a) Introduce charge $q_{2}$ on the opposite side of charge $q_{1}$
with respect to the slab. (b) Superposing the image charges from charge
$q_{1}$ and $q_{2}$, as seen from region $1$.

###  II.2 Some special cases

There are special cases where Eq. (9) can be represented by special functions
and even have a nice short algebraic form.

The dilogarithm function Cartier  _et al._ (2007)
$\text{Li}_{2}(z)=\sum^{\infty}_{k=1}z^{k}/k^{2}$ can be used to rewrite the
infinite sum in Eq. (8) with finite terms. For examples, when $a\in
2\mathbb{Z}^{+}$:

$S_{a}(z)=\frac{\text{Li}_{2}(z^{2})-\sum^{(a-2)/2}_{k=1}\frac{z^{2k}}{k^{2}}}{4z^{a}}\
,$ (10)

and when $a\in 2\mathbb{Z}^{+}+1$:

$S_{a}(z)=\frac{\text{Li}_{2}(z)-\text{Li}_{2}(z^{2})/4-\sum^{(a-3)/2}_{k=0}\frac{z^{2k+1}}{(2k+1)^{2}}}{z^{a}}\
.$ (11)

When $d_{1}=h=d_{2}$, we get the ratios $\alpha_{1}=\alpha_{2}=1$ and the
electrostatic force in Eq. (9) becomes:

$\begin{split}&f_{1}=\frac{q_{1}^{2}}{4\pi\epsilon_{0}h^{2}}\left(-\frac{\zeta}{4}+\frac{(1-\zeta^{2})\left(\text{Li}_{2}(\zeta^{2})-\zeta^{2}\right)}{4\zeta^{3}}\right)\\\
&+\frac{q_{1}q_{2}}{4\pi\epsilon_{0}h^{2}}\left(\frac{(1-\zeta^{2})\left(4\text{Li}_{2}(\zeta)-\text{Li}_{2}(\zeta^{2})-4\zeta\right)}{4\zeta^{3}}\right)\
,\end{split}$ (12)

where $\zeta=(\epsilon-1)/(\epsilon+1)$. Consider further simplification with
$q_{1}=q_{2}$, the interaction is now reciprocal $|f_{1}|=|f_{2}|$ and we also
arrive at:

$f_{1}=\frac{q_{1}^{2}}{4\pi\epsilon_{0}h^{2}}\left(1-\frac{1}{4\zeta}-\frac{1}{\zeta^{2}}-\frac{\text{Li}_{2}(\zeta)}{\zeta}+\frac{\text{Li}_{2}(\zeta)}{\zeta^{3}}\right)\
.$ (13)

For the relative dielectric constant $\epsilon=3$, we get $\zeta=1/2$. Since
$\text{Li}_{2}(1/2)=\pi^{2}/12-\ln^{2}2/2$, Loxton (1984) we can arrive at the
following algebraic form:

$\begin{split}f_{1}&=\frac{q_{1}^{2}}{4\pi\epsilon_{0}h^{2}}\left(\frac{\pi^{2}-7}{2}-3\ln^{2}2\right)\\\
&\approx\frac{q_{1}^{2}}{4\pi\epsilon_{0}h^{2}}\times\left(-6.6\times
10^{-3}\right)\ .\end{split}$ (14)

The “$-$” sign indicates that the electrostatic forces are pulling the charges
toward the dielectric slab. Compare with Eq. (2) which comes from the naive
assumption of expanding space:

$\begin{split}f_{1}&=\frac{q_{1}^{2}}{4\pi\epsilon_{0}(2+\sqrt{3})^{2}h^{2}}\\\
&\approx\frac{q_{1}^{2}}{4\pi\epsilon_{0}h^{2}}\times\left(+7.2\times
10^{-2}\right)\ ,\end{split}$ (15)

this wrong answer has incorrect sign and the interaction strength is off by an
order of magnitude.

There are other interesting limits that can be read-off easily from Eq. (13).
For the relative dielectric constant satisfies $|\epsilon-1|\ll 1$, we have
$\zeta\approx(\epsilon-1)/2$, $\text{Li}_{2}(\zeta)\approx(\epsilon-1)/2$ and
get the approximation:

$f_{1}\approx\frac{q_{1}^{2}}{4\pi\epsilon_{0}h^{2}}\left(\frac{1}{9}-\frac{3(\epsilon-1)}{32}\right)\
,$ (16)

which is consistent with the slab vanishing as $\epsilon\rightarrow 1$. For
the relative dielectric constant satisfies $\epsilon\gg 1$, we have
$\zeta\approx 1$, $\text{Li}_{2}(\zeta)\approx\pi^{2}/6$ and get the
approximation:

$f_{1}\approx\frac{q_{1}^{2}}{4\pi\epsilon_{0}h^{2}}\left(-\frac{1}{4}\right)=-\frac{q_{1}^{2}}{4\pi\epsilon_{0}(2d_{1})^{2}}\
,$ (17)

which is consistent with the slab being a conductor (the interaction is
between $q_{1}$ and its image charges $q_{1}^{\prime}\approx-q_{1}$ and the
electrical field from $q_{2}$ is perfectly shielded out).

##  III Acknowledgement

We thank Tung X. Tran, Duy V. Nguyen and the xPhO journal club for their
support to share this finding to a wider audience.

## References

  * Hammond (1960) P. Hammond, Proceedings of the IEE-Part C: Monographs 107, 306 (1960).
  * Jackson (1999) J. D. Jackson, Classical electrodynamics (1999).
  * Cartier  _et al._ (2007) P. E. Cartier, B. Julia, P. Moussa, and P. Vanhove,  _Frontiers in number theory, physics, and geometry II: on conformal field theories, discrete groups and renormalization_ , Vol. 2 (Springer Science & Business Media, 2007).
  * Loxton (1984) J. Loxton, Acta Arithmetica 43, 155 (1984).


[affiliation=1]SarthakJain* [affiliation=2]OrchidChetia Phukan*
[affiliation=2]ArunBalaji Buduru [affiliation=2,3]RajeshSharma

# The Reasonable Effectiveness of Speaker Embeddings for Violence Detection

###### Abstract

In this paper, we focus on audio violence detection (AVD). AVD is necessary
for several reasons, especially in the context of maintaining safety,
preventing harm, and ensuring security in various environments. This calls for
accurate AVD systems. Like many related applications in audio processing, the
most common approach for improving the performance, would be by leveraging
self-supervised (SSL) pre-trained models (PTMs). However, as these SSL models
are very large models with million of parameters and this can hinder real-
world deployment especially in compute-constraint environment. To resolve
this, we propose the usage of speaker recognition models which are much
smaller compared to the SSL models. Experimentation with speaker recognition
model embeddings with SVM & Random Forest as classifiers, we show that speaker
recognition model embeddings perform the best in comparison to state-of-the-
art (SOTA) SSL models and achieve SOTA results.

###### keywords:

Violence Detection, Speaker Embeddings, x-vector

##  1 Introduction

Violence not only inflicts immediate harm but also leaves lasting scars on the
fabric of society. The ripple effects extend beyond individual victims,
impacting families, neighborhoods, and entire communities. As such, ensuring
public safety and security becomes paramount, requiring proactive measures to
address violence and foster environments where all members can thrive without
fear of harm or injustice. In response to the negative effects associated with
violence, previous research has made substantial leap towards violence
detection through use of various modalities. In this work, we focus on audio
violence detection (AVD). AVD has advantages in conditions such as in areas
with low visibility or at night for detecting violence, in comparison to
visual or audio-visual violence where access to a visual monitoring device
will be required. Researchers have explored AVD with methods ranging from
classical machine learning to deep learning [1, 2]. However, despite all these
advancements AVD still lacks in accuracy and considering the importance of
AVD, systems with higher accuracy are need of the hour.

Following related speech and audio processing applications, the most common
and beneficial way to boost up a system performance would be using embeddings
from pre-trained models (PTMs) especially self-supervised (SSL) PTMs [3].
However, these state-of-the-art (SOTA) SSL PTMs are very large with millions
of parameters that may hinder real-world deployment of AVD systems especiallly
in scenarios with less compute power. In response, in this work, we propose to
use embeddings extracted from speaker recognition PTMs, models pre-trained
primarily for speaker recognition for AVD which is much smaller in size than
the SSL PTMs. With comprehensive comparative analysis, we show that x-vector
(speaker recognition) embeddings attains the topmost performance. This topmost
behavior of x-vector embeddings can be attributed to its effectiveness in
capturing intensity, intonation, etc more effectively than SSL PTMs for AVD.

Figure 1: Audio Violence Detection Application; Application architecture
pipeline is presented in Figure 1 (a) with the flow of information when the
input audio is provided to the final inference received by the user through
the user interface; Figure 1(c) shows the confusion matrix of the best model
RF with x-vector embeddings

##  2 Application

In this section, we discuss various parts of the proposed AVD application.
Firstly, we discuss the various PTMs whose embeddings are used in our study
and the classifiers. Followed by the database considered and the results of
the experiments. Lastly, we discuss the workings of the application and how a
user can interact with the application.

Pre-Trained Models: We use SOTA SSL PTMs WavLM [4] and Unispeech-SAT [5]
because of its performance in various applications in SUPERB. For speaker
recognition PTMs, we choose x-vector [6] and ECAPA [7] due to their SOTA
performance in speaker recognition. The number of parameters of the PTMs are
94.7M, 94.68M, 7M, 20M for wavLM, Unispeech-SAT, x-vector, ECAPA respectively.
We extract embeddings of 768 from the last hidden state of SSL PTMs through
average pooling and for x-vector, we extract embeddings of dimension of 512.

Classifiers: We experiment with Random Forest (RF) and Support Vector Machine
(SVM) classifier with default parameters. We train the models with 5-fold
where four folds will be used for training and fifth fold for testing.

Database and Pre-Processing: We use AVD
database111https://www.kaggle.com/datasets/fangfangz/audio-based-violence-
detection-dataset used by Zhu et al. [8]. We divided each of the violence and
non violence audio files into sub audio files of 2.5 sec following [8] and
results in dataset with non-violence audios of 7241 and violent audios of
1374. We resample the audios to 16kHz for passing as input to the PTMs.

Table 1: Performance Scores; Scores are average of 5 folds; Scores are presented in %; F1 is macro-average F1-Score; MFCCs are used as baseline PTM | Accuracy | F1
---|---|---
Random Forest
x-vector | 99.14 | 98.40
ECAPA | 94.01 | 87.42
MFCC | 72.46 | 49.01
Unispeech-SAT | 94.07 | 87.99
WaveLM | 91.05 | 80.61
Support Vector Machine
x-vector | 99.13 | 98.38
ECAPA | 98.02 | 98.02
MFCC | 71.93 | 48.71
Unispeech-SAT | 97.73 | 95.87
WaveLM | 95.53 | 91.40

Experimental Results: The evaluation results of the various models are
presented in Table 1. Models trained with speaker recognition models
(x-vector, ECAPA) embeddings performed the best. This can be attributed to
their supremacy in capturing speech characteristics such as intensity,
intonation, etc more effectively. We use the best model Random Forest trained
with x-vector embeddings as backend model in the application.

User Interface: Working and Backend Details: The user interface of the
application is presented in Figure 1. In mode 1 (Figure 1 (a)), user can use
Start Recording and Stop Recording button to record real-world audio for AVD
and in mode 2 (Figure 1 (b)) user can upload a audio with Choose File button
for detecting violence in the uploaded audio. With Predict button, we can get
the predictions out of the application. We have used React.Js and Flask for
creating the Front-End and the Back-End for model inference respectively. The
average inference time for a 1 min audio file is around 1 sec, thus making it
deployable to real-world use.

##  3 Conclusion

In this demonstration, we've created an application that harnesses speaker
recognition model embeddings to enhance AVD. The application streamlines the
process of detecting violent activities across surveillance feeds without
access to visual-monitoring device . Overall, the application offers law
enforcement agencies a powerful tool for more efficient and effective
monitoring, ultimately enhancing public safety.

## References

  * [1] A. M. Yildiz, P. D. Barua, S. Dogan, M. Baygin, T. Tuncer, C. P. Ooi, H. Fujita, and U. R. Acharya, ``A novel tree pattern-based violence detection model using audio signals,'' _Expert systems with applications_ , vol. 224, p. 120031, 2023.
  * [2] A. Bakhshi, J. García-Gómez, R. Gil-Pita, and S. Chalup, ``Violence detection in real-life audio signals using lightweight deep neural networks,'' _Procedia Computer Science_ , vol. 222, pp. 244–251, 2023.
  * [3] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, ``SUPERB: Speech Processing Universal PERformance Benchmark,'' in _Proc. Interspeech 2021_ , 2021, pp. 1194–1198.
  * [4] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, ``Wavlm: Large-scale self-supervised pre-training for full stack speech processing,'' _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1505–1518, 2022.
  * [5] S. Chen, Y. Wu, C. Wang, Z. Chen, Z. Chen, S. Liu, J. Wu, Y. Qian, F. Wei, J. Li, and X. Yu, ``Unispeech-sat: Universal speech representation learning with speaker aware pre-training,'' in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 6152–6156.
  * [6] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, ``X-vectors: Robust dnn embeddings for speaker recognition,'' in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 5329–5333.
  * [7] B. Desplanques, J. Thienpondt, and K. Demuynck, ``ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,'' in _Proc. Interspeech 2020_ , 2020, pp. 3830–3834.
  * [8] F. Zhu-Zhou, D. Tejera-Berengué, R. Gil-Pita, M. Utrilla-Manso, and M. Rosa-Zurera, ``Computationally constrained audio-based violence detection through transfer learning and data augmentation techniques,'' _Applied Acoustics_ , vol. 213, p. 109638, 2023.