# Autonomous Blimp Control via H∞ Robust Deep Residual Reinforcement Learning

Yang Zuo2,∗, Yu Tang Liu1,2,∗, Aamir Ahmad1,2 1Max Planck Institute for
Intelligent Systems, 72076 Tübingen, Germany. 2Institute of Flight Mechanics
and Controls, University of Stuttgart, 70569 Stuttgart, Germany.
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>stuttgart.de ∗Yang Zuo and Yu-Tang Liu contribute equally to this work as the
first author.

###### Abstract

Due to their superior energy efficiency, blimps may replace quadcopters for
long-duration aerial tasks. However, designing a controller for blimps to
handle complex dynamics, modeling errors, and disturbances remains an unsolved
challenge. One recent work combines reinforcement learning (RL) and a PID
controller to address this challenge and demonstrates its effectiveness in
real-world experiments. In the current work, we build on that using an
$H_{\infty}$ robust controller to expand the stability margin and improve the
RL agent’s performance. Empirical analysis of different mixing methods reveals
that the resulting H∞-RL controller outperforms the prior PID-RL combination
and can handle more complex tasks involving intensive thrust vectoring. We
provide our code as open-source at https://github.com/robot-perception-
group/robust_deep_residual_blimp.

##  I Introduction

Unmanned aerial vehicles (UAVs) like multirotors and fixed wings are
increasingly being used for visual tracking tasks such as aerial
cinematography, wildlife monitoring[1], and precision farming[2]. However,
while multirotors have limitations such as short battery life and small
payload, fixed-wings must constantly move to stay airborne. We propose using
autonomous blimps, which are more energy-efficient and have a higher payload
for long-duration, small-region hovering tasks.

Blimp control, however, presents challenges in the context of modeling
uncertainties and wind disturbances. Prior work used a deep residual
reinforcement learning (DRRL) framework[4, 3] to address this with a model-
free proportional-integral-derivative (PID) base controller and an RL agent
[5]. During training, the RL agent’s action can be considered as an extra
disturbance to the base controller, so the robustness of the base controller
defines the permitted exploratory actions.

In the current work, we replace the PID base controller with a robust model-
based $H_{\infty}$ controller to expand the stability margin. The $H_{\infty}$
robust design framework generates a controller that makes decisions based on
the worst-case scenario, which offers the most significant safety bound at the
cost of control performance. This gives the RL agent a larger exploration
bandwidth and more potential performance growth. The model-based approach also
allows deriving the worst-case bound that considers the total amount of model
uncertainty and disturbance from both the environment and the RL agent.

We show in the simulated environment that the DRRL agent, consisting of the
$H_{\infty}$ robust control and a proximal policy optimization (PPO) agent[6],
outperforms the previous PID-PPO combination in performance and robustness and
can even handle more challenging tasks. We also improve the DRRL framework by
a variable mixing factor such that the controller can grant the RL agent a
variable amount of control authority. We designed the base controller’s thrust
vectoring to enhance the final performance further, allowing the RL agent to
access a more significant state and action space for better exploration.

Figure 1: Top: the simulated blimp with the proposed $H_{\infty}$-PPO
controller in the challenging coil trajectory. Bottom: descend trajectory of
our $H_{\infty}$-PPO versus prior PID-PPO [5] controller. Our controller is
more robust against disturbance and improves altitude control by utilizing
thrust vectoring, while PID-PPO controller can only rely on the elevators for
altitude control. As a result, it can deviate nearly 15 meters from the
desired path.

##  II Related Work

Research on reliable robotic platforms for aerial tracking tasks has led to
the exploration of blimp and vision-based control, with most using PID-based
control [10, 8, 7, 11, 9]. However, PID controllers are often a suboptimal
solution for non-linear control problems like a blimp. Alternative solutions
from model-based control frameworks, such as optimal control [12, 13],
adaptive control [14], or robust control [15] have been sought, but these have
not yielded reliable controllers for real-world experiments due to model
uncertainty and output disturbance. In recent years, RL with Gaussian Process-
based models has been used for low-dimensional tasks [17, 18, 16]. In
contrast, Deep RL (DRL) with large capacity models achieved a 3D path-
following task in simulation [19] and in the first real-world experiment with
the data-driven approach [5]. This work has extended the prior DRRL agent [5]
by replacing PID with a robust $H_{\infty}$ controller to improve safety and
performance growth.

##  III Methodology

In this section, we first introduce the simulators and formulate the task in
the reinforcement learning framework (Sec. III-B). Different from [5], we
introduce the $H_{\infty}$ controller as our base control (Sec. III-C).
Lastly, we introduce our robust $H_{\infty}$-based deep residual reinforcement
learning controller (Sec. III-E) as shown in Fig. 2, where mixer block
represents the following equation,

$a_{mixed}=(1-q)a+qu$ (1)

$a_{t}\sim\pi(\cdot|s_{t})$$u_{t}=H_{\infty}(s_{t})$Mixer
(1)Environment$a_{t}$$a_{mixed,t}$$\;s_{t+1},r_{t}$$s_{t+1}$$u_{t},q_{t}$$u_{t},q_{t}$
Figure 2: Our robust deep residual reinforcement learning framework. Every
time step, the mixer gathers the action command from the policy $a_{t}$ and
the controller $u_{t}$ and then mixes them based on the mixing factor $q_{t}$
evaluated by the controller.

The second difference from the previous work [5], which applies a fixed number
of $q$, is that we sample it randomly from a distribution. The variable $q$
allows the controller to decide how much authority can be granted to the RL
agent, depending on the situation. For example, when the wind disturbance is
prominent, the controller can increase $q$ for more intervention and safety.

In the experiments section, we demonstrate that reducing the amount of
intervention from the base control $q$ improves the final performance
(Sec.IV-C). Therefore, our goal is to design a robust controller to guarantee
control stability during both learning and testing phase while a minimum
amount of intervention is required.

###  III-A Markov Decision Process (MDP)

We first formulate RL problems as an MDP, and it can be represented as a
tuple, $\mathcal{M}=(\mathcal{S},\mathcal{A},R,\mathcal{P},\gamma,\rho)$,
where $s_{t}\in\mathcal{S}$ and $a_{t}\in\mathcal{A}$ are the state and action
space respectively. At any time step $t\in\mathbb{R}$, the RL agent samples an
action from its control policy based on the observed environmental state
$a_{t}\sim\pi(\cdot|s_{t})$. Then the environment returns the next state and a
reward base on the underlying transition dynamics
$s_{t+1}\sim\mathcal{P}(\cdot|s_{t},a_{t})$ and a reward function
$r_{t}=R(s_{t},a_{t})$, which defines the desired behavior and can be viewed
as a task description. Given the discount factor $\gamma\in[0,1)$ and initial
state distribution $s_{0}\sim\rho(\cdot)$, the goal of the RL agent is to find
a control policy such that the total amount of discounted reward can be
maximized, i.e.

$\displaystyle\pi^{*}=\operatorname*{arg\,max}_{\pi}\mathbb{E}_{\rho}[\sum^{\infty}_{t=0}\gamma^{t}r(s_{t},a_{t})|a_{t}{\small\sim}\pi(\cdot|s_{t}),s_{t+1}{\small\sim}\mathcal{P}(\cdot|s_{t})]$
(2)

###  III-B Task Formulation

We train and test our mixed $H_{\infty}$-RL controller (Fig. 2) to perform
navigation tasks in the simulated environments. The agent’s goal is to control
the vehicle to a desired position. We first introduce two environments: a
simplified toy environment, TurtleSim, and the blimp simulator[11].

####  III-B1 Turtle Control Task

Due to the similarity to the blimp control problem, we introduce it for the
ablation study. As shown in Fig. 3(a), the agent observes the state in every
time step and controls the robot turtle to a stationary target position. Both
robot and target positions spawn randomly in every new episode. We formulate
the problem by an MDP with the following state and action space,

  * •

state space: $s_{t}=(s_{\theta},s_{l},u_{v},u_{\omega},q)_{t}\in[0,1]$,

  * •

action space: $a_{t}=(a_{v},a_{\omega})_{t}\in[-1,1]$,

where all states are in the range $[0,1]$, and the scaled state
$(s_{\theta},s_{l})$ are the relative yaw angle $\theta$ and relative distance
$l$, augmented with the mixing factor $q\in[0,1]$, and the base control
$({u_{v}},{u_{\omega}})$ corresponds to thrust and yaw velocity command and
share the same command channel with the agent’s actions $(a_{v},a_{\omega})$.
Then the navigation task can be formulated by the reward function,

$\displaystyle r_{t}=\begin{bmatrix}w_{success}&w_{track}\\\
\end{bmatrix}\begin{bmatrix}r_{success,t}\\\ r_{track,t}\\\ \end{bmatrix}$ (3)
$\displaystyle r_{success,t}=1\;if\;|l_{t}|\leq\epsilon\;else\;0,$ (4)
$\displaystyle r_{track,t}=-|l_{t}|,$ (5)

where by default $w_{success}=500$, $w_{track}=0.1$, and $\epsilon=0.1$. The
environment resets itself when the success reward is obtained.

(a) TurtleSim with parallel data collection. The green turtle is the robot,
and the target position is represented by a turtle in another color. The white
curve displays the position odometry.

(b) The blimp simulator is implemented in ROS/Gazebo framework. It provides
high-fidelity fluid dynamics, and supports software-in-the-loop simulation
(SITL) [11].

Figure 3: Simulation Environments

####  III-B2 Blimp Control Task

Similar to the turtle control task, the goal of the RL agent is to navigate
the robotic blimp (Fig. 3(b)) to a virtual position target. The state and
action space are specified as follows,

  * •

state space:
$s_{t}=(s_{z},s_{l},s_{\theta},u_{\zeta},u_{\eta},u_{\epsilon},u_{\delta},q)_{t}$

  * •

action space: $a_{t}=(a_{\zeta},a_{\eta},a_{\epsilon},a_{\delta})_{t}$

where all states are scaled in the range $[0,1]$, and the scaled state
$(s_{z},s_{l},s_{\theta})$ are the relative altitude $z$, relative distance
$l$, relative yaw angle $\theta$, augmented with the mixing factor $q\in[0,1]$
and the base control command $(u_{\zeta},u_{\eta},u_{\epsilon},u_{\delta})$
corresponding to the control of rudder deflection, elevator deflection, the
servo thrust angle, and the thrust magnitude. The actions
$(a_{\zeta},a_{\eta},a_{\epsilon},a_{\delta})$ corresponds to the same command
channels. Note that the action dynamics are coupled; for example, one can
ascend by an elevator when moving forward or directly thrusting upward through
the thrust vector.

In this context, there are two major differences from the prior work[5]. First
is the usage of the reverse thrust. Descending a blimp is challenging since
the blimp’s heading velocity is usually slow, and, consequently, the altitude
descent velocity from the elevator is also slow. This can cause significant
altitude tracking errors. And therefore, even though reverse thrusting is
generally less efficient, it helps the blimp descend much faster when lacking
the heading velocity.

Another difference is that we trigger the next target waypoint only when the
total distance to the target is less than a threshold of 5 meters instead of
the planar distance. This requires much higher efficiency over the altitude
control and poses a more significant challenge for control allocation as there
are diverse ways to achieve it, e.g., elevator or thrust vector. We
demonstrate in the experiment that the RL agents fail to find any viable
control policy without efficiently using thrust vectoring.

The following reward function formulates the navigation task,

$\displaystyle r_{t}=\begin{bmatrix}w_{success}&w_{track}&w_{penalty}\\\
\end{bmatrix}\begin{bmatrix}r_{success,t}\\\ r_{track,t}\\\ r_{penalty,t}\\\
\end{bmatrix}$ (6) $\displaystyle
r_{success,t}=1\;if\;|l_{t}|\leq\epsilon\;else\;0,$ (7) $\displaystyle
r_{track,t}=-w_{z}|z_{t}|-w_{l}|l_{t}|-w_{\theta}|\theta|,$ (8) $\displaystyle
r_{penalty,t}=\Delta(a,u),$ (9)

where the default value of the task weight is
$(w_{success},w_{track},w_{penalty})=(500,1,10)$, the tracking reward weight
$(w_{z},w_{l},w_{\theta})=(2,5,2)$ and $\epsilon=5$[m]. The term $\Delta(a,u)$
penalizes when the action deviates too much from the base control to encourage
the synergy between the agent and the controller. In practice, we found out
that without this ad-hoc penalty, RL agents fail to find any viable control
policy. At each time step, we initialize $\Delta(a,u)=0$, and then accumulate
it if any of the conditions are triggered,

  * •

$\Delta(a,u)\mathrel{+}=-0.5$, if $a_{\zeta}u_{\zeta}<0$ and
$|a_{\zeta}-u_{\zeta}|>0.4$.

  * •

$\Delta(a,u)\mathrel{+}=-0.5$, if $a_{\eta}u_{\eta}<0$ and
$|a_{\eta}-u_{\eta}|>0.4$.

  * •

$\Delta(a,u)\mathrel{+}=-0.5$, if $a_{\epsilon}a_{\delta}>0$ .

  * •

$\Delta(a,u)\mathrel{+}=1$, if $a_{\epsilon}u_{\epsilon}>0$.

  * •

$\Delta(a,u)\mathrel{+}=-0.5$, if $u_{\eta}=-1$, $u_{\epsilon}=0.5$ and
$a_{\epsilon}>0.7$.

To avoid misuse of the reverse thrust, the third condition penalizes when the
agent commands the thrust vector to tilt backward $a_{\epsilon}>0$ and the
thrust $a_{\delta}$ to be positive and vice versa. Similarly, the last
condition penalizes when the controller commands the thrust vector to tilt
backward at its maximum $u_{\epsilon}=0.5$, but the RL agent tilts the thrust
vector even more, which leads to inefficient reverse thrusting. Lastly, all
other conditions penalize when the action commands between the agent and
controller are too different.

###  III-C $H_{\infty}$ Robust Control

$K$$G$$u$$Sensor$$W_{S}$$W_{KS}$$W_{T}$$w$$e$$du$$dy$$y$$-$$y_{m}$$z_{1}$$z_{2}$$z_{3}$
Figure 4: $H_{\infty}$ control framework. The goal is to design controller $K$
such that the robot G can be stabilized and input and output disturbance ($du$
and $dy$) can be rejected.

This framework can be illustrated by the feedback control loop in Fig. 4,
where K is the controller, G is our robot, and the state is directly observed
from the sensor without any estimator. Given the tracking signal $w$ and
feedback observation $y_{m}$, The goal is to design the controller such that
the tracking error $e=w-y_{m}$ can be reduced over time subject to input and
output disturbance $du$, $dy$. Note that in the hybrid control Fig. 2, the
agent action can be viewed as part of $du$. The weighting filters have the
following forms,

$\displaystyle
W_{T}(s)=\frac{1}{T_{min}}\cdot\frac{s+\omega_{zt}}{s+\omega_{nt}},$ (10a)
$\displaystyle
W_{KS}(s)=\frac{1}{KS_{min}}\cdot\frac{s+\omega_{zks}}{s+\omega_{nks}},$ (10b)
$\displaystyle
W_{S}(s)=\frac{1}{S_{max}}\cdot\frac{s+\omega_{zs}}{s+\omega_{ns}},$ (10c)

where $\omega$ is the cut-off frequency of each filter. The weight filter
parameters in our experiments are presented in Table. II. Then the controller
K can be solved by satisfying the following constraint,

$\left\lVert\begin{matrix}W_{S}\cdot S\\\ W_{KS}\cdot KS\\\ W_{T}\cdot T\\\
\end{matrix}\right\rVert_{\infty}\leq 1$ (11)

In practice, we design the weight filters manually and then solve the
controller K in MATLAB [20]. Let $DU$ and $U$ be the Laplace transform of the
unknown input disturbance $du$ and control command $u$, then system response
can be formulated as follows,

$G\cdot(I+\Delta\cdot W_{T})\cdot U=G\cdot(U+DU),$ (12)

by treating the input disturbance as part of model uncertainty, where $I$ is
the identity matrix and $\Delta=\Delta(s)$ is the uncertainty matrix with
$\left\lVert\Delta\right\rVert_{\infty}\leq 1$. After factoring out $G$, we
can derive the following relation by matrix sub-multiplicative,

$\displaystyle\left\lVert DU\right\rVert_{2}$
$\displaystyle=\left\lVert\Delta\cdot W_{T}\cdot U\right\rVert_{2}$ (13)
$\displaystyle\leq\left\lVert\Delta\right\rVert_{2}\cdot\left\lVert
W_{T}\right\rVert_{2}\cdot\left\lVert U\right\rVert_{2}$ (14)
$\displaystyle\leq\left\lVert W_{T}\right\rVert_{2}\cdot\left\lVert
U\right\rVert_{2}$ (15)

Since matrix $W_{T}$ is diagonal and only consists of identical values, we can
consider the $i$-th row of (15):

$\left\lVert DU_{i}\right\rVert_{2}\leq\left\lVert
W_{T,i}\right\rVert_{2}\cdot\left\lVert U_{i}\right\rVert_{2}.$ (16)

Note that (16) is sufficient but not necessary for (15), which means that (16)
is a more strict condition to determine the upper bound of $\left\lVert
DU\right\rVert_{2}$. And by Parseual’s theorem, we have two identities for
(16):

$\displaystyle\left\lVert DU_{i}(j\omega)\right\rVert_{2}=\left\lVert
du_{i}(t)\right\rVert_{2}$ (17a) $\displaystyle\left\lVert
U_{i}(j\omega)\right\rVert_{2}=\left\lVert u_{i}(t)\right\rVert_{2}$ (17b)

Finally, we can derive a conservative upper bound for the plant input
disturbance from (16) and (17), i.e., $\left\lVert
du_{i}\right\rVert_{2}\leq\left\lVert W_{T,i}\right\rVert_{2}\left\lVert
u_{i}\right\rVert_{2}$, such that the $H_{\infty}$-controller stabilizes the
plant G. We derive the theoretical upper bound for $\left\lVert
du_{i}\right\rVert_{2}$ in both simulators as shown in the Table. I, assuming
that the controller commands a step input $u_{i}=1$ when $t\geq 0$.

Simulator | $\left\lVert W_{T,i}\right\rVert_{2}(j\omega)$ | $\left\lVert u_{i}\right\rVert_{2}$ | Maximum $\left\lVert du_{i}\right\rVert_{2}$ |  $\omega$ $[rad/s]$
---|---|---|---|---
TurtuleSim | 1.84 | $\sqrt{t}$ | $1.84\sqrt{t}$ | 100
Blimp | 33.34 | $\sqrt{t}$ | $33.34\sqrt{t}$ | 10
TABLE I: Maximum allowed input disturbance. Depending on the sampling
frequency $\omega$, we increase $\left\lVert W_{T}\right\rVert_{2}$ of the
blimp simulator for more robustness and less of TurtleSim for more
performance. Symbol $t$ denotes the time duration the controller sending step
input $u_{i}=1$.

Now consider the mixed command in (1). As long as the following relation is
satisfied, then the process will remain stable.

$\displaystyle\left\lVert(1-q(t))\cdot(u_{i}(t)-a_{i}(t))\right\rVert_{2}\leq\left\lVert
W_{T,i}(j\omega)\right\rVert_{2}\cdot\left\lVert u_{i}(t)\right\rVert_{2}$
(18)

Intuitively, the RHS is the upper bound of the input disturbance the base
control $u$ can reject. If we choose the weighting parameter $q$ as a constant
through time, we can further reduce (18) to :

$\displaystyle q\geq 1-\frac{\left\lVert
W_{T,i}(j\omega)\right\rVert_{2}\cdot\left\lVert
u_{i}(t)\right\rVert_{2}}{\left\lVert(u_{i}(t)-a_{i}(t))\right\rVert_{2}}$
(19)

Then according to Table. I and the constraint (19) and assuming average case
when agent actions are uniform random and have an average zero, i.e.,
$\mathbb{E}\left[a\right]=0$, we can select any distribution for the mixing
factor $q$ as long as the mean of the distribution is positive, i.e.,
$\mathbb{E}\left[q\right]\geq 0$. Or assuming worst-case scenario when we have
an adversarial agent, i.e., $\mathbb{E}\left[a\right]=-u$, then when
$\mathbb{E}\left[q\right]\geq 0.08$ for TurtleSim or
$\mathbb{E}\left[q\right]\geq 0$ for the blimp simulator, the process will
remain stable.

However, because our plant model is likely imperfect and considering other
disturbance and noise that is not modeled, the allowed maximum input
disturbance will be less than the estimation. In practice, our conservative
choice of $\mathbb{E}\left[q\right]=0.5$ for the TurtleSim and
$\mathbb{E}\left[q\right]=0.3$ for the blimp simulator seem to work well.

Parameters | turtle | blimp, yaw | blimp, as | blimp, ds
---|---|---|---|---
$T_{max}$ | $\sqrt{5}$ | $\sqrt{2.2}$ | $\sqrt{2}$ | $\sqrt{2}$
$T_{min}$ | $0.01$ | $0.01$ | $0.01$ | $0.01$
$KS_{max}$ | $10$ | $0.8$ | $0.5$ | $0.5$
$KS_{min}$ | $0.01$ | $0.01$ | $0.01$ | $0.01$
$S_{max}$ | $2\sqrt{2}$ | $\sqrt{2}$ | $\sqrt{2}$ | $\sqrt{2}$
$S_{min}$ | $0.001$ | $0.01$ | $0.01$ | $0.01$
$\omega_{ns}$ | $0.00625\sqrt{2}$ | $0.001\sqrt{2}$ | $0.001\sqrt{2}$ | $0.001\sqrt{2}$
$\omega_{zt}$ | $25$ | $0.2$ | $0.2$ | $0.2$
$\omega_{nt}$ | $2500\sqrt{5}$ | $20\sqrt{2.2}$ | $20\sqrt{2}$ | $20\sqrt{2}$
$\omega_{zks}$ | $200$ | $0.8$ | $0.5$ | $0.5$
$\omega_{nks}$ | $2\cdot 10^{5}$ | $64$ | $25$ | $25$
$\omega_{zs}$ | $25$ | $0.2$ | $0.2$ | $0.2$
TABLE II: $H_{\infty}$ control weighting filter parameters. Note that column
”turtle” denotes the $H_{\infty}$-controller in TurtleSim, and ”blimp,
yaw/as/ds” denote the yaw, ascend, and descend motion, respectively, in the
blimp simulator.

###  III-D Robust Hybrid $H_{\infty}$-RL in TurtleSim

Given the controller command $u=[u_{v},u_{\omega}]^{T}$, reference
$w=[0,0]^{T}$, the state vector $x=[l,\theta]^{T}$, and plant output
$y=[l,\theta]^{T}$, the kinematics of the turtle can be represented by the
following state space model,

$\displaystyle A_{turtle}=\begin{bmatrix}0&0\\\ 0&0\\\
\end{bmatrix},B_{turtle}=\begin{bmatrix}-1\\\ 1\\\ \end{bmatrix},$ (20)
$\displaystyle C_{turtle}=\begin{bmatrix}1&0\\\ 0&1\\\
\end{bmatrix},D_{turtle}=\begin{bmatrix}0\\\ 0\\\ \end{bmatrix}.$ (21)

Recall that $l$ denotes the relative distance between the turtle and its
target, and $\theta$ represents the heading angle difference to the target.
The kinematic model of the turtle, i.e., the plant $G$, including the
disturbances $du$ and $dy$, sensor measurement, and the weighting filters $W$,
yields the augmented plant $P$ (Fig. 5). The augmented plant $P$ is stabilized
by a controller $K$, which is obtained by applying the $H_{\infty}$ design
method given the constraint (11) and Table. II.

$P$$K$$w$$z$$e$$u$ Figure 5: $H_{\infty}$ controller compact form

Now consider the hybrid scenario, and the control command has the following
form,

$a_{mixed}=\begin{bmatrix}(1-q)a_{v}+qu_{v}&(1-q)a_{\omega}+qu_{\omega}\\\
\end{bmatrix}^{T}$ (22)

where $a\sim\pi(\cdot|s)$. The mixing factor is sampled randomly from the
uniform random distribution at each time step, i.e., $q\sim\mathcal{U}(0,1)$,
which satisfies the constraint in (19). Note that it is important for the
agent to observe the entire interval of $q\in[0,1]$ during training so that
the agent will be able to generalize to any arbitrary $q$ in the testing
phase.

###  III-E Robust Hybrid $H_{\infty}$-RL in the Blimp Simulator

Since one compact MIMO (multiple-input and multiple-output)-controller using
the $H_{\infty}$-method is hard to derive, we split the entire blimp dynamic
into two linear sub-systems: the yaw motion and the others.

The yaw dynamic is modeled as $(A,B,C,D)_{yaw}=(0,-20,1,0)$, with the state
$x_{yaw}=\theta$, $y_{yaw}=\theta$, $u_{yaw}=u_{\zeta}$ and
$w_{yaw}=\theta_{ref}$, where $\theta$ is the heading angle of the blimp w.r.t
the world frame. Furthermore, we model the time delay by the second-order Padé
approximation with a dead time $T=0.65$,

$e^{-Ts}\approx\frac{2-Ts}{2+Ts}$ (23)

The rest of the dynamics is required for the velocity and altitude control.
Since the thrusting angle introduces non-linearity, we linearize at two trim
points and design two $H_{\infty}$-controllers for the ascending and
descending motions, respectively. In both modes, we have the same states,
plant outputs, and control commands, i.e., $x_{as/ds}=[-l,z]^{T}$,
$y_{as/ds}=[l,z]^{T}$, $u_{as/ds}=[u_{\eta},u_{\epsilon},u_{\delta}]^{T}$, and
$w_{as/ds}=[0,0]^{T}$.

Recall that $l$ denotes the relative distance, and $z$ denotes the relative
altitude. Then we have the ascending dynamics,

$\displaystyle A_{ascend}=\begin{bmatrix}0&0\\\ 0&0\\\
\end{bmatrix},B_{ascend}=\begin{bmatrix}0&-5&9.8\\\ -0.4&-0.77&0\\\
\end{bmatrix}$ (24) $\displaystyle C_{ascend}=\begin{bmatrix}-1&0\\\ 0&1\\\
\end{bmatrix},D_{ascend}=\begin{bmatrix}0&0&0\\\ 0&0&0\\\ \end{bmatrix}$ (25)

and descending dynamics,

$\displaystyle A_{descend}=\begin{bmatrix}0&0\\\ 0&0\\\
\end{bmatrix},B_{descend}=\begin{bmatrix}0&5&-9.8\\\ -0.4&0.77&0\\\
\end{bmatrix},$ (26) $\displaystyle C_{descend}=\begin{bmatrix}-1&0\\\ 0&1\\\
\end{bmatrix},D_{descend}=\begin{bmatrix}0&0&0\\\ 0&0&0\\\ \end{bmatrix}.$
(27)

As we are applying two linear controllers to two highly non-linear dynamics,
we further restrict the controller commands by heuristics to assure that the
blimp works near the linearization points, i.e., $u_{\epsilon}\in[-1,-0.5]$
and $u_{\delta}\in[0.4,0.6]$ in ascending mode while $u_{\epsilon}\in[0.5,1]$
and $u_{\delta}\in[-0.6,-0.4]$ in descending mode.

Now, we obtain in total three $H_{\infty}$ controllers by applying the
$H_{\infty}$ design method via (11) and the weighting filter (Table. II) for
their linearized dynamics. The altitude controller switches the mode at zero
relative altitudes, i.e., $z=0$, while the yaw controller remains independent
of the altitude control.

Finally, our hybrid DRRL agent has the action in the format,

$a_{mixed}=(1-q)\begin{bmatrix}a_{\zeta}\\\ a_{\eta}\\\ a_{\epsilon}\\\
a_{\delta}\\\ \end{bmatrix}+q\begin{bmatrix}u_{\zeta}\\\ u_{\eta}\\\
u_{\epsilon}\\\ u_{\delta}\\\ \end{bmatrix}$ (28)

During training, we sample $q$ in some distributions, while in the testing
phase, the controller can provide $q$ based on the constraint (19) or apply a
constant $q$ as we did in this work. We have experimented with different $q$
distributions and empirically found that the mixing factor with any
distribution works well if it covers the full range $q\in[0,1]$. More details
are displayed in the experiment section (Sec. IV-D).

###  III-F Training the Robust DRRL Agent

Our networks (Table. III) follow the actor-critic architecture, which requires
two function approximators, e.g., deep neural networks, for the value
estimation, $V_{\theta}(s,a)$ and the policy distribution
$\pi_{\phi}(\cdot|s)$. A PPO agent (proximal policy optimization, [6]), with
hyper-parameters in Table. IV, is employed to optimize our networks’
parameters.

| Value Network | Policy Network
---|---|---
Simulator | $o$ | $L$ | $F_{1}$ | $F_{2}$ | $v$ | $o$ | $L$ | $F_{1}$ | $F_{2}$ | $\mu$
TurtuleSim | 5 | 64 | 64 | 64 | 1 | 5 | 24 | 24 | 24 | 2
Blimp | 8 | 196 | 196 | 196 | 1 | 8 | 64 | 64 | 64 | 4
TABLE III: Network architecture. Notation $L$ denotes the LSTM layer, F is the
fully connected layer, and the numbers indicate the layer’s size. Following
the suggestion of a recent work [21], we choose tanh as our activation
function and initialize the last layer with small weights (e.g., 0.01) to
improve the exploration.

The hybrid agent collects data by interacting with the $N_{env}$ parallelized
environment with randomized mixing factor $q$. A waypoint is sampled randomly
in every episode, and it will be triggered when the robot is within a certain
distance, e.g., 1[m] for the TurtleSim and 5[m] for the blimp simulator. The
environment resets when the waypoint is triggered. We inject noise into the
observations to increase the agent’s robustness and randomize wind disturbance
and buoyancy in every episode.

The transition data are stored in the buffer with size $N_{epoch}$. When the
buffer is full, the agent will start learning by querying $L/N_{batch}$ mini-
batches and updating $N_{update}$ times for each of them or when the KL
threshold $D_{KL}$ is reached. Note that the base control can be considered
part of the environment in our hybrid control scenario. The agent’s goal is to
optimize the total amount of reward considering the base control’s decision.

Parameters | TurtleSim | Blimp
---|---|---
Time Steps per Environment $N_{step}$ | 50000 | 86400
Parallelization $N_{env}$ | 8 | 7
Episode Length $L$ | 2000 | 2400
Loop Rate [$Hz$] | 100 | 10
Epoch Length $N_{epoch}$ | 1000 | 1920
Mini-batch Size $N_{batch}$ | 100 | 128
Update per Epoch $N_{update}$ | 20 | 20
Initial Policy Learning Rate $\alpha_{0}$ | $5\cdot 10^{-5}$ | $5\cdot 10^{-5}$
Initial Value Learning Rate $\beta_{0}$ | $1\cdot 10^{-4}$ | $1\cdot 10^{-4}$
KL Threshold $D_{KL}$ | $\infty$ | 0.03
Discount Factor $\gamma$ | 0.999 | 0.99
GAE Smoothing $\lambda$ | 0.95 | 0.9
Gradient Optimizer | $Adam$ | $Adam$
TABLE IV: PPO hyper-parameters. The learning rate is scheduled by multiplying
a constant of less than one every episode until it drops to a minimum of 1e5.

##  IV Experiments and Results

The experiment aims to understand whether increasing the robustness of the
base control can enable more performance growth and generate a robust and
performant controller.

###  IV-A Experiment Setup

We perform all experiments on a single computer (AMD Ryzen Threadripper 3960X,
24x 3.8GHz, NVIDIA GeForce RTX2080 Ti, 11GB). The PPO agent and base
controllers, i.e., $H_{\infty}$ and PIDs, are implemented based on Pytorch[22]
to facilitate vectorized computing. Both TurtleSim and the blimp simulator are
implemented based on the ROS and the latter is integrated with the Gazebo SITL
simulation.

We compare our robust $H_{\infty}$-RL controller to the previous PID-RL
baseline in the TurtleSim (Sec. III-B1) and blimp simulator (Sec. III-B2).

  1. 1.

$H_{\infty}$-PPO agent: our proposed approach

  2. 2.

PID-PPO agent: previous approach [5]. We re-implemented with a randomized
mixing factor $q$ and a servo control so that it has the chance to challenge
our more difficult waypoint following task.

Note when the mixing factor is $q=1$ we recover the base control, and $q=0$
corresponds to the pure PPO agent. In the training phase, $q$ is sampled
randomly from different distributions while fixed in the testing phase.

###  IV-B Performance Metric

Because the total rewards can be pretty misleading as it only reflects an
agent’s performance tracking one specific waypoint, and each agent’s training
reward can differ. We introduce a metric, $b_{score}$, to compare the relative
performance between the controllers to replace the reward for our path-
following tasks.

$b_{score}(\pi_{i})=100\cdot\left(1-\frac{T_{\pi_{i}}}{\sum_{\pi_{j}\in\pi}T_{\pi_{j}}}-\frac{E_{\pi_{i}}}{\sum_{\pi_{j}\in\pi}E_{\pi_{j}}}\right),$
(29)

where $T_{\pi_{i}}$ is the average amount of time for each controller to
complete the task and $E_{\pi_{i}}$ computes the average control effort for
each controller. The more time or energy the controller consumes to complete
the task, the worse the score relative to other controllers.

###  IV-C TurtleSim

We sample a random position target in every new episode to test the proposed
hybrid agent. The environment resets when the robot reaches the target
position. The experiment terminates when the agent successfully controls the
robot to the target position 100 times. The goal of each agent is to trigger
the terminal condition with a minimum amount of time and energy in every
episode. Table. V is the result of our experiment in TurtleSim, where the
energy penalty for each control policy is defined as
$E_{\pi}=0.25\cdot|\bar{a}_{mix,v}|+0.75\cdot|\bar{a}_{mix,\omega}|$, bar
denotes the average over the whole trajectory. Each experiment is conducted
with ten random seeds.

Note that because TurtleSim did not have any dynamics, we applied the
disturbance $du$ during training to output action to simulate wind, which is
formulated as time dependant noise,

$\displaystyle a_{mixed,t}$ $\displaystyle\leftarrow a_{mixed,t}+\delta_{t}$
(30) $\displaystyle\delta_{t}$ $\displaystyle=\delta_{t-1}+[n_{v}\quad
n_{\omega}]^{\top}$ (31)

where $\delta_{t}$ is initialized as zero and bounded in $[-1,1]$, and both
noises are sampled from standard Gaussian
$n_{v},n_{\omega}\sim\mathcal{N}(0,1)$. This noise is amplified five times in
the testing phase, increasing the bound to $[-5,5]$.

Controller | $q$ | $\bar{|a_{v}|}$ | $\bar{|a_{\omega}|}$ | $T$ | $E$ | $b_{score}$
---|---|---|---|---|---|---
PPO | 0 | 5.58 | 4.72 | 9916 | 4.93 | 70.63
$H_{\infty}$-PPO | 0.5 | 5.62 | 5.56 | 11709 | 5.58 | 66.05
PID-PPO | 0.5 | 5.88 | 4.96 | 10055 | 5.19 | 69.66
$H_{\infty}$-PPO | 1 | 7.18 | 6.85 | 11765 | 6.93 | 61.91
PID-PPO | 1 | 7.32 | 5.69 | 12281 | 6.10 | 63.65
TABLE V: Testing in TurtleSim. Each row displays the average value of 10
experiments with different seeds.

Table. V indicates that the PPO agent alone has the best performance and
energy efficiency, followed by the PID and then $H_{\infty}$ controller.
Therefore, with less controller intervention $q$, the hybrid agent will
achieve better performance. Unsurprisingly, the PID controller performs better
than the $H_{\infty}$ controller, which trades both the performance and energy
efficiency for more robustness against noise and disturbance. Base controller
with more robustness allows training with lower mixing factor $q$, but since
TurtleSim is relatively simple and allows both base controllers to train and
test with arbitrary $q$, the advantage of $H_{\infty}$ is not well reflected
in this experiment.

controller | q | $a_{\zeta}$ | $a_{\eta}$ | $a_{\epsilon}$ | $a_{\delta}$ | $T$ | $L$ | $E$ | $b_{score}$ | fail | wind | $T$ | $L$ | $E$ | $b_{score}$ | fail | buoyancy | $T$ | $L$ | $E$ | $b_{score}$ | fail
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PID-PPO | 0 | 0.83 | 0.90 | 1 | 1 | 3414 | 43.90 | 0.8705 | 84.38 | 7 | 0 | 2996 | 37.74 | 0.7252 | 86.50 | 4 | 0.93 | 2420 | 37.89 | 0.4988 | 88.95 | 8
PID-PPO | 0.5 | 0.67 | 0.79 | 0.99 | 0.79 | 4294 | 66.09 | 0.6943 | 81.24 | 5 | 0.5 | 4255 | 70.88 | 0.6048 | 81.38 | 5 | 1 | 2633 | 55.99 | 0.6284 | 86.64 | 3
PID-PPO | 1 | 0.47 | 0.55 | 0.94 | 0.57 | 2615 | 38.42 | 0.5048 | 88.36 | 4 | 1 | 2464 | 33.55 | 0.4927 | 89.15 | 8 | 1.07 | 5416 | 43.98 | 0.7562 | 39.73 | 6
$H_{\infty}$-PPO | 0 | 0.52 | 0.88 | 0.57 | 1 | 2646 | 35.97 | 0.8641 | 87.00 | 0 | 0 | 2464 | 35.12 | 0.6813 | 88.28 | 0 | 0.93 | 3442 | 35.69 | 0.6875 | 85.60 | 0
$H_{\infty}$-PPO | 0.5 | 0.44 | 0.63 | 0.61 | 0.76 | 2484. | 36.10 | 0.6665 | 88.23 | 0 | 0.5 | 2685 | 38.07 | 0.6830 | 87.49 | 0 | 1 | 2510 | 36.88 | 0.6720 | 88.08 | 0
$H_{\infty}$-PPO | 1 | 0.47 | 0.43 | 0.71 | 0.55 | 3456 | 37.10 | 0.5075 | 86.19 | 0 | 1 | 3438 | 35.98 | 0.6739 | 85.65 | 0 | 1.07 | 2633 | 36.60 | 0.6788 | 87.74 | 0
TABLE VI: Robustness test. Each experiment is conducted nine times with the
coil trajectory. Fail trials are excluded from computing the $b_{score}$ and
marked as fail. The energy penalty is defined as
$E_{\pi_{i}}=0.15\cdot\bar{|u_{\zeta}|}+0.05\cdot\bar{|u_{\eta}|}+0.1\cdot(1-\bar{|u_{\epsilon}|})+0.7\cdot\bar{|u_{\delta}|}$,
which penalizes majorly on thrusting. The unit for wind is [m/s] and for
buoyancy is [%]. The maximum speed of the simulated blimp is 2 [m/s].

###  IV-D Blimp Simulator

The training method of the hybrid agent is introduced in Sec. III-F for both
PID-PPO baseline and our $H_{\infty}$-PPO agent. We conduct an ablation study
on the effect of the different sampling distributions for the mixing factor
$q$ during training. The testing phase is conducted in the coil trajectory,
represented by a sequence of waypoints with three different wind disturbance
and buoyancy levels. Each evaluation finishes when the agent completes the
designated trajectory five times. The waypoints are triggered when the robot
is within 10 meters, and the following waypoints will become active.

The coil trajectory consists of 15 waypoints with a 50 meters radius. Each
waypoint is placed 45 degrees counter-clockwise from the previous one with 3
meters increase in altitude. The coil trajectory poses a great challenge. Due
to the shorter planar distance, the controllers must constantly slow down the
blimp to prevent overshooting the waypoints, which can incur significant
altitude loss.

Since deviating far from the track compromises the blimp’s safety, the
position tracking error L is introduced to the $b_{score}$ for the blimp
navigation task, i.e.,

$\displaystyle b$ ${}_{score}(\pi_{i})$
$\displaystyle=100\cdot\left(1-\frac{3T_{\pi_{i}}}{\sum_{\pi_{j}\in\pi}T_{\pi_{j}}}-\frac{E_{\pi_{i}}}{\sum_{\pi_{j}\in\pi}E_{\pi_{j}}}-\frac{L_{\pi_{i}}}{\sum_{\pi_{j}\in\pi}L_{\pi_{j}}}\right),$
(32)

where $T_{\pi_{i}}$ and $E_{\pi_{i}}$ are the time and energy penalty, and the
$L_{\pi_{i}}$ is the average distance loss computed by the norm of the
relative position.

The robustness test for the agents is displayed in Table VI. The
$H_{\infty}$-PPO outperforms the PID-PPO combination with a significantly
higher success rate regardless of the wind or buoyancy condition. Even when
the controller has zero intervention $q=0$, the PPO agent trained with
$H_{\infty}$ base control performs much better. And the $H_{\infty}$-PPO
performs the best when $q=0.5$ without failing to any condition. This shows
that our robust residual RL framework can generate a robust, high-performance
controller. The explanations can be found in the table as well.

First, base control robustness is vital to the final performance of the PPO
agent. When $q=1$, although $H_{\infty}$ performs worse than PID, its
robustness against disturbance secures its success rate. During training, PID
can become unstable due to input disturbance from the RL agent, further
jeopardizing the PPO training stability while $H_{\infty}$ is less affected.
As a result, the PPO trained with PID performs even worse than PID. The wind
and buoyancy test reflect the robustness of the controller. The PID-PPO
failure rate increases significantly outside the nominal condition in which
PID is tuned. The effect of wind can be visualized in Fig.6. The wind barely
influences $H_{\infty}$-PPO controller while PID-PPO controller is blown away
and loses yaw control when descending. Second, the PID controller has
relatively poor altitude control even after installing thrust vectoring. Since
thrust vectoring introduces high nonlinearity to the system, the PID
controller does not benefit much from it. The poor performance in altitude
control is reflected in the buoyancy test.

Figure 6: Snippet of the coil trajectories (green curves) and waypoints (red
dots). The first and second columns correspond to the $H_{\infty}$-PPO and
PID-PPO controllers. Rows correspond to the wind velocities 0, 0.5, and 1
[m/s]. The buoyancy is in nominal condition while the mixing factor $q=0.5$.

We conducted an ablation study about the effect of wind disturbance and $q$
distribution during training. Table. VIII displays the effect of incorporating
wind disturbance during training. Regardless of the mixing factor $q$, the
$b_{score}$ always decreases when training with the wind. With $q=0$, the
performance drops significantly, implying that the wind negatively impacts the
agent more than the controller. As a result, we suggest training without any
disturbance to encourage aggressive behavior since, under the supervision of
the robust controller, the agent no longer needs to behave conservatively.

The effect of mixing factor $q$ during training is displayed in Table.VIII. We
found the training success rate increases when $q$ becomes larger since the
base control is critical in improving the training stability. The distribution
with a higher average $q$ is preferred during training as it increases the
training success rate. The distribution type is not essential as long as it
covers the entire range $q\in[0,1]$. Lastly, as mentioned, a lower average $q$
is desired for improving the control performance in the testing phase.

wind | $q$ | $T$ | $L$ | $E$ | $b_{score}$
---|---|---|---|---|---
False | $0.5$ | 7378 | 20.81 | 0.36 | 88.54
True | $0.5$ | 7727 | 20.66 | 0.44 | 87.58
False | $0.2$ | 6750 | 19.85 | 0.40 | 88.60
True | $0.2$ | 7354 | 21.69 | 0.51 | 86.88
False | $0$ | 6637 | 19.40 | 0.42 | 88.61
True | $0$ | 8874 | 22.80 | 0.58 | 85.23
TABLE VII: Training with random wind disturbance.

$q$ | $T$ | $L$ | $E$ | $b_{score}$ | fail
---|---|---|---|---|---
$0.2$ | 12280 | 26.32 | 0.35 | 85.26 | 2
$\mathcal{U}(0,0.4)$ | 29180 | 20.65 | 0.5 | 77.49 | 1
$Beta(2,8)$ | 6921 | 20.02 | 0.39 | 88.58 | 0
$0.5$ | 7523 | 19.69 | 0.45 | 87.84 | 0
$\mathcal{U}(0,1)$ | 6727 | 19.62 | 0.43 | 88.42 | 0
$Beta(5,5)$ | 8981 | 19.66 | 0.47 | 86.96 | 0
TABLE VIII: Training with different $q$ distributions.

Ablation study with $H_{\infty}$-PPO controller tested in coil trajectory.
Every experiment is conducted three times.

##  V Conclusions

In this work, we have introduced a $H_{\infty}$-PPO hybrid agent for the blimp
control task. We first improve the altitude control efficiency by
incorporating the thrust vectoring into the base control and enabling the
usage of reverse thrusting. Then, we applied the variable mixing factor, which
allows the controller to balance robustness and performance based on the
situation. A theoretical lower bound for the mixing factor is derived to
guarantee stability. Lastly, we test in the blimp simulator that our robust
hybrid agent can outperform the prior PID-PPO combination and demonstrate
greater robustness against wind disturbance and buoyancy changes.

## References

  * [1] Luis Gonzalez et al.  “Unmanned Aerial Vehicles (UAVs) and Artificial Intelligence Revolutionizing Wildlife Monitoring and Conservation”  In _Sensors_ 16, 2016, pp. 97  DOI: 10.3390/s16010097
  * [2] Vishakh Duggal et al.  “Plantation monitoring and yield estimation using autonomous quadcopter for precision agriculture”  In _2016 IEEE international conference on robotics and automation (ICRA)_ , 2016, pp. 5121–5127  IEEE
  * [3] Tobias Johannink et al.  “Residual Reinforcement Learning for Robot Control”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 6023–6029  DOI: 10.1109/ICRA.2019.8794127
  * [4] Tom Silver, Kelsey Allen, Josh Tenenbaum and Leslie Kaelbling  “Residual policy learning”  In _arXiv preprint arXiv:1812.06298_ , 2018
  * [5] Yu Tang Liu, Eric Price, Michael J. Black and Aamir Ahmad  “Deep Residual Reinforcement Learning based Autonomous Blimp Control”, 2022  URL: https://arxiv.org/pdf/2203.05360
  * [6] John Schulman et al.  “Proximal policy optimization algorithms”  In _arXiv preprint arXiv:1707.06347_ , 2017
  * [7] Jinjun Rao, Zhenbang Gong, Jun Luo and Shaorong Xie  “A flight control and navigation system of a small size unmanned airship”  In _IEEE International Conference Mechatronics and Automation, 2005_ 3, 2005, pp. 1491–1496 Vol. 3  DOI: 10.1109/ICMA.2005.1626776
  * [8] Hong Zhang and James P Ostrowski  “Visual servoing with dynamics: Control of an unmanned blimp”  In _Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No. 99CH36288C)_ 1, 1999, pp. 618–623  IEEE
  * [9] Toshihiko Takaya et al.  “PID landing orbit motion controller for an indoor blimp robot”  In _Artificial Life and Robotics_ 10, 2006, pp. 177–184
  * [10] Eric Price, Michael J Black and Aamir Ahmad  “Perception-driven Formation Control of Airships”  In _arXiv preprint arXiv:2209.13040_ , 2022
  * [11] Eric Price, Yu Tang Liu, Michael J Black and Aamir Ahmad  “Simulation and control of deformable autonomous airships in turbulent wind”  In _Intelligent Autonomous Systems 16: Proceedings of the 16th International Conference IAS-16_ , 2022, pp. 608–626  Springer
  * [12] Hiroaki Fukushima et al.  “Model predictive control of an autonomous blimp with input and output constraints”  In _2006 IEEE Conference on Computer Aided Control System Design, 2006 IEEE International Conference on Control Applications, 2006 IEEE International Symposium on Intelligent Control_ , 2006, pp. 2184–2189  IEEE
  * [13] J.R. Azinheira et al.  “Visual servo control for the hovering of all outdoor robotic airship”  In _Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292)_ 3, 2002, pp. 2787–2792 vol.3  DOI: 10.1109/ROBOT.2002.1013654
  * [14] Shi Qian Liu, Yuan Jun Sang and James F Whidborne  “Adaptive sliding-mode-backstepping trajectory tracking control of underactuated airships”  In _Aerospace Science and Technology_ 97 Elsevier, 2020, pp. 105610
  * [15] Lin Cheng, Zongyu Zuo, Jiawei Song and Xiao Liang  “Robust three-dimensional path-following control for an under-actuated stratospheric airship”  In _Advances in Space Research_ 63.1 Elsevier, 2019, pp. 526–538
  * [16] Jonathan Ko, Daniel J Klein, Dieter Fox and Dirk Haehnel  “Gaussian processes and reinforcement learning for identification and control of an autonomous blimp”  In _Proceedings 2007 ieee international conference on robotics and automation_ , 2007, pp. 742–747  IEEE
  * [17] Axel Rottmann and Wolfram Burgard  “Adaptive autonomous control using online value iteration with gaussian processes”  In _2009 IEEE International Conference on Robotics and Automation_ , 2009, pp. 2106–2111  IEEE
  * [18] Axel Rottmann, Christian Plagemann, Peter Hilgers and Wolfram Burgard  “Autonomous blimp control using model-free reinforcement learning in a continuous state and action space”  In _2007 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2007, pp. 1895–1900  DOI: 10.1109/IROS.2007.4399531
  * [19] Chunyu Nie, Zewei Zheng and Ming Zhu  “Three-dimensional path-following control of a robotic airship with reinforcement learning”  In _International Journal of Aerospace Engineering_ 2019 Hindawi Limited, 2019, pp. 1–12
  * [20] MATLAB  “version 7.10.0 (R2022a)”  Natick, Massachusetts: The MathWorks Inc., 2022
  * [21] Marcin Andrychowicz et al.  “What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study”, 2020
  * [22] Adam Paszke et al.  “Pytorch: An imperative style, high-performance deep learning library”  In _Advances in neural information processing systems_ 32, 2019


# Re-annotation of cough events
in the AMI corpus

Paul Leamy, Ted Burke, Damon Berry, David Dorran

Biomedical Research Group
Technological University Dublin

(17-18 June 2019)

##  1 Introduction

Cough sounds act as an important indicator of an individual’s physical health
[1], often used by medical professionals in diagnosing a patient’s ailments
[2]. In recent years progress has been made in the area of automatically
detecting cough events and, in certain cases, automatically identifying the
ailment associated with a particular cough sound. Ethical and sensitivity
issues associated with audio recordings of coughs make it more difficult for
this data to be made publicly available. Without the public availability of a
reliable database of cough sounds, developments in the area of audio event
detection are likely to be hampered [3]. The purpose of this paper is to
spread awareness of a database [4] containing a large amount of naturally
occurring cough sounds that can be used for the implementation, evaluation,
and comparison of new machine learning algorithms that allow for audio event
detection associated with cough sounds. Using a purpose built GUI designed in
MATLAB, a re-annotated version of the Augmented Multi-party Interaction (AMI)
corpus’ cough location annotations was produced with 1369 individual cough
events. All cough annotations and the re-annotation tool are available for
download and public use [5].

##  2 The AMI corpus

The AMI corpus data is publicly available under the Creative Commons
Attribution 4.0 license agreement, and contains audio recordings of indoor
meetings taken in an office environment.

Figure 1: Recordings were captured from omni-directional microphone arrays
(near and far mounted) and headset and lapel microphones - Dashed insert:
Closeup of omni-directional microphone arrays.

The audio recordings are accompanied by annotations of a large number of audio
events throughout the entire corpus. Audio recordings were captured from
multiple microphone locations, with an example of one meeting room shown in a
Figure 1.

###  2.1 Re-annotation procedure

Following an initial analysis of the annotations, it was found that a number
of issues existed relating to the original cough event annotations in the AMI
corpus including. These include incorrect start and end times, and successive
coughs annotated as single coughs, examples of which are illustrated in Figure
2.

(a)  (b)

(c)  (d)

Figure 2: Examples of inaccurate annotations. Dashed black lines in the
figures are the original annotations.

The re-annotation criteria for the cough events is illustrated in Figure 3.

Figure 3: New annotation criteria, red “x” marks the onset/start of the cough,
and green “o” marks the end of the cough event.

The task of importing the audio, and the recording of the new time-stamps was
carried out using a purpose built GUI, as shown in Figure 4.

Figure 4: GUI created to carry out the re-annotation procedure of cough events
in the AMI corpus.

##  3 Results

Following the re-annotation procedure, a new set of time-stamps corresponding
to the start and end times of single cough events was produced. The total
number of single cough events before and after the re-annotation procedure is
displayed in Table 1.

Table 1: Number of cough event annotations Number of cough annotations
---
Original annotations | Re-annotated list
1116 | 1369

##  4 Conclusions

Justification for the re-annotation of the cough locations in the AMI corpus
came following the discovery of a number of issues relating to the original
annotations. In the area of cough sound analysis and detection the presence of
a reliably annotated database will be a useful addition particularly in the
area of machine learning. This database will be useful for progressing the
development and testing of new audio event detection algorithms concerned with
cough sounds. The re-annotated version of the cough location annotations have
been made publicly available along with the GUI used in the re-annotation
procedure [5]. A consistent approach to labelling of start and end times of
the individual cough events was used, and as a result the potential for the
AMI corpus to become a more reliable and reusable source of annotated cough
recordings has increased.

###  4.1 References

## References

  * [1] M. El Ayadi, M. S. Kamel, and F. Karray, “Survey on speech emotion recognition: Features, classification schemes, and databases,” _Pattern Recognition_ , vol. 44, no. 3, pp. 572–587, 2011.
  * [2] S. M. Schappert and C. W. Burt, “Ambulatory care visits to physician offices, hospital outpatient departments, and emergency departments: United states, 2001-02.” _Vital and Health Statistics. Series 13, Data from the National Health Survey_ , no. 159, pp. 1–66, 2006.
  * [3] J. Schröder, J. Anemiiller, and S. Goetze, “Classification of human cough signals using spectro-temporal gabor filterbank features,” in _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , March 2016, pp. 6455–6459.
  * [4] J. Carletta, “Unleashing the killer corpus: experiences in creating the multi-everything ami meeting corpus,” _Language Resources and Evaluation_ , vol. 41, no. 2, pp. 181–190, 2007.
  * [5] P. Leamy, “paulleamy/ami-cough-annotations,” Mar 2019. [Online]. Available: https://github.com/paulleamy/AMI-Cough-Annotations.git