# Model-assisted Reinforcement Learning of a Quadrotor

Arshad Javeed  Dept. of Automatic Control, Lund University

###### Abstract

In recent times, reinforcement learning has produced baffling results when it
comes to performing control tasks with highly non-linear systems. The
impressive results always outweigh the potential vulnerabilities or
uncertainties associated with the agents when deployed in the real-world.
While the performance is remarkable compared to the classical control
algorithms, the reinforcement learning-based methods suffer from two flaws,
robustness and interpretability, which are vital for contemporary real-world
applications. The paper attempts to alleviate such problems with reinforcement
learning and proposes the concept of “model-assisted” reinforcement learning
to induce a notion of conservativeness in the agents. The control task
considered for the experiment involves navigating a CrazyFlie quadrotor. The
paper also describes a way of reformulating the task to have the flexibility
of tuning the level of conservativeness via multi-objective reinforcement
learning. The results include a comparison of the vanilla reinforcement
learning approaches and the proposed approach. The metrics are evaluated by
systematically injecting disturbances to classify the inherent robustness and
conservativeness of the agents. More concrete arguments are made by computing
and comparing the backward reachability tubes of the RL policies by solving
the Hamilton-Jacobi-Bellman partial differential equation (HJ PDE).

##  1 Introduction

The classical control approach relies on the ability to accurately model the
system and leverage a proven and robust control algorithm. The challenge here
is coming up with a decently accurate mathematical description of the system,
and the problem can be exacerbated in the case of a non-linear or a high-
dimensional system. In this regard, the reinforcement learning approach, with
the machine-learning methodology of learning from the data has yielded
unmatched results in diverse application domains while requiring a minimal
description of the system [Kober and Peters, 2014, Razzaghi et al., 2022, Bai
et al., 2023, Arulkumaran et al., 2017].

Reinforcement learning is closely related to optimal control [Sutton and
Barto, 2018]. In the context of control, the agent tries to learn an optimal
control policy by interacting with the environment by receiving rewards for
its actions. The fundamental idea is to maximize the expected reward received
from the environment. The learning can be model-based or model-free. The
contemporary model-based and model-free algorithms are enriched by deep
learning and neural networks acting as non-linear function approximators. The
“model” here refers to the model of the environment. In an MDP setting, the
model-free approach does not leverage the dynamics of the world in any way,
instead tries to approximate the reward function and the cost function by
repeated interaction. There are several algorithms in the model-free category.
[Mnih et al., 2016, Schulman et al., 2017b] directly try to optimize the
policy function parameterized by a deep neural network. [Haarnoja et al.,
2018, Silver et al., 2014, Fujimoto et al., 2018] are actor-critic variants
that employ neural networks for the actor and the critic, where the critic
approximates the true cost function using the recursive Bellman equation.
Model-based reinforcement learning, on the other hand, uses the dynamics
(predefined or learns simultaneously) to define the state transition
probabilities to seek better actions and reduce repeated trial and error in
the environment, and the state transition probabilities can be utilized by the
algorithm in several ways. MBVE [Feinberg et al., 2018] uses model-based
learning for better estimation of the value function. MBMF [Nagabandi et al.,
2017] first learns the model or the environment dynamics using a shallow
neural network and then uses a Model Predictive Control (MPC). Expert
Iteration [Anthony et al., 2017] uses an expert improvement step to define the
imitation learning targets and then resolves to imitation learning for faster
convergence.

With reinforcement algorithms employed in diverse applications, robustness and
interpretability are imperative. When employed in real-world applications, any
reckless behavior can have ramifications for the agent itself or the
surroundings. There have been several attempts to incorporate safety within
reinforcement learning. [Srinivasan et al., 2020] employ a safety critic that
evaluates the safety of the given state-action pair looking at the unsafe
experiences of the agent and then transforms the task to safety-constrained
MDP. Since the unsafe conditions are determined by the data, this may induce a
bias. The constraints themselves are less interpretable due to the fact that a
black box model or neural network tries to learn from them. SAVED [Thananjeyan
et al., 2020] is a deep MPC that learns the model by training neural networks
observing suboptimal demonstrations. The learned model is then used to impose
constraints on MPC optimization. [Brunke et al., 2021] is a comprehensive
review of safety-constrained reinforcement learning. MPC approach is one of
the only non-linear control strategies that can guarantee the constraint
requirements, the only downside is that it is computationally expensive.

Broaching the topic of robustness, “robustness” can be defined as the ability
of the model to perform consistently (or even better than expected) in diverse
situations. Robustness does not necessarily imply that the agent performs
better in unknown environments. On the contrary, we want the agent to honor
certain physical constraints whilst compromising the reward objective, but
ultimately accomplishing the task. In the context of reinforcement learning,
this could be quite tricky, as often conflicting objectives are represented by
the reward signal. [Glossop et al., 2022] have evaluated the robustness of
reinforcement learning algorithms by systematic disturbance injection. A
startling discovery made by them is that subjecting the agent to disturbances
during the training phase does not necessarily improve the robustness, i.e.
the inherent robustness of agents trained with and without disturbances is
about the same. [Mayne et al., 2005] have solved the classical model
predictive control approach by incorporating bounded disturbances during
online optimization. [Morimoto and Doya, 2005] propose an adversarial setting
(robust reinforcement learning), where an adversarial agent modeling the
environment tries to impede the performance, while the RL agent performing the
task tries to overcome it. However, [Glossop et al., 2022] report that the
robust reinforcement learning methods do not considerably outperform the
baseline model.

In contrast, the proposed work aims to use a partially specified model or
learn a partial model and then use the model constraints as one of the reward
objectives with the intention of reinforcement learning agent learning to act
in a way that would honor the model constraints. Often in reinforcement
learning, there are conflicting objectives and there is always an added
advantage of taking the risk. The primary motivation of the design is the
ability to individually tune the constraints during run-time by employing
multi-objective reinforcement learning (depending on the user preferences).
Similar to [Glossop et al., 2022], the evaluation is done by systematically
injecting disturbances during the test phase and looking at the agent’s
capability to adhere to the defined constraints. The proposed work
corroborates the implicit robustness of reinforcement learning agents, i.e.
training with disturbance does not necessarily improve the reward. But at the
same time, discusses interesting properties of the proposed methodology. All
the experiments are performed on a highly non-linear real-world system, a
CrazyFlie quadrotor. More details of the task are presented in the subsequent
sections.

##  2 Background

Reinforcement learning is a type of unsupervised machine learning, where the
agent learns an optimal policy for the given task by interacting with the
environment. The “feedback signal” or the reward signal is the key to
reinforcement learning. During the training process, the agent’s goal is to
simply learn a strategy that would maximize the cumulative reward.

At any given time $t$, the agent could be in an arbitrary state
$s_{t}\in\mathcal{S}$ and can choose to perform an action
$a_{t}\in\mathcal{A}$. As a consequence of the action, the agent receives a
reward $R(s_{t},a_{r})=r_{t}\in\mathbb{R}$. So the cumulative reward for
acting according to the policy $\pi$ starting from state $s_{t}$ is
$\Sigma_{t^{\prime}=t}^{\infty}\gamma^{t^{\prime}-t}R(s_{t^{\prime}},\pi(s_{t^{\prime}}))$,
where $\gamma\in[0,1)$ is called the discount factor. Thus, having defined a
reward function $R$, the goal is to learn an optimal policy $\pi^{*}$ that
maps the states to actions $\pi^{*}:\mathcal{S}\to\mathcal{A}$ that yields the
best discounted reward.

If the above task is done iteratively without assuming any knowledge of the
system or the dynamics of the environment, it is called “model-free”
reinforcement learning. On the other hand, if the information about the
dynamics or the state transitions is leveraged or learned during the process,
it is “model-based learning” RL. The transition probabilities here are the
conditional probabilities $P(s_{t+1}|s_{t},a_{t})$, which resemble the
transition probabilities typically encountered in a Markov decision problem
(MDP).

In the above formulation, the reward $r_{t}$ was assumed to be a scalar.
Practical applications (as well as in the scope of the paper) deal with the
tradeoff between multiple constraints, and there are often conflicting
objectives or rewards. Traditionally, a utility function is used to scalarize
the reward. A utility function scalarizes the reward as
$u:\mathbb{R}^{d}\to\mathbb{R}$ and the discounted reward becomes
$\Sigma_{t^{\prime}=t}^{\infty}\gamma^{t^{\prime}-t}u(r_{t^{\prime}})$, and
the value function then corresponds to Expected Scalarized Return (ESR). A
common utility function is simply the weighted average,
$u(r_{t})=\Sigma_{i}w_{i}r_{ti}$. However, this would muddle the reward
objectives and make them less interpretable. An alternative approach alluding
to multi-objective reinforcement learning is the Scalarized Expected Return
(SER) approach, where the utility function is applied after the inner
expectation, i.e.
$u\left(\Sigma_{t^{\prime}=t}^{\infty}\gamma^{t^{\prime}-t}r_{t^{\prime}}\right)$.
In multi-objective reinforcement learning, the goal is now to learn an array
of policies each of which prioritizes the reward objectives differently.
[Hayes et al., 2022] provide a detailed overview of the strategies involved in
multi-objective learning.

The usage of deep neural networks as function approximators to approximate the
policies $\pi:\mathbb{R}^{n_{S}}\to\mathbb{R}^{n_{A}}$ and value functions
$Q:\mathbb{R}^{n_{S}}\times\mathbb{R}^{n_{A}}\to\mathbb{R}^{n_{R}}$ makes it
feasible to model non-linear dynamical systems and learn effective control
strategies [Arulkumaran et al., 2017, Ladosz et al., 2022, Lee and Qin, 2019].

##  3 Formulating the RL Task

The non-linear dynamical system considered here is a CrazyFlie quadrotor. The
task for the reinforcement learning agent is to learn a policy to navigate the
quadrotor to the specified destination starting from an initial state $s_{0}$
(figure 1). However, the environment poses a challenge to the agent, there are
always disturbances (forces emulating wind) acting along the $X,Y,Z$
directions randomly. The observation space or the state space consists of the
system kinematics (equation 1), involving the 3D position information,
orientation (roll, pitch, and yaw), and linear and angular velocities along
the $X,Y,Z$ axes. The state space $S$ will later be extended with additional
states later on.

$\mathcal{S}=[x,y,z,r,p,y,v_{x},v_{y},v_{z},w_{x},w_{y},w_{z}]$ (1)

Now, there are several ways of formulating the action space for the navigation
task. We can have the reinforcement learning agent have complete granular
control of the quadrotor by letting it control the low-level rotor RPMs or use
a hierarchical control approach. The hierarchical approach involves the
reinforcement learning agent defining high-level objectives for the quadrotor
and the quadrotor employing a low-level controller such as a PID controller to
execute the actions. Employing a trusted low-level controller has a twofold
advantage. Firstly, it means that the RL agent does not have to learn
everything from scratch, for instance, simple maneuvers like moving up, down,
diagonally, etc. can be performed by the low-level controller confidently,
while the high-level controller is only concerned about taking the appropriate
course of actions. This significantly alleviates the algorithmic convergence.
Secondly, the overall model is more interpretable, due to the fact that the
high-level actions can be interpreted easily. [Panerati et al., 2021] is a
simulation bed supporting both approaches. However, the latter approach is run
in an open-loop fashion, i.e. given a target position, the PID controller is
used to compute the corresponding rotor RPMs and the implementation involves
applying the RPMs for a fixed number of physics steps, but this is flawed in
the way that it does not necessarily guarantee that the quadrotor executes the
actions successfully, i.e. it may fall short or be far off the target. In
contrast, the proposed approach executes a closed-loop PID control action to
ensure that the quadrotor successfully executes the high-level action to
reasonable accuracy. This makes the overall action execution even more
interpretable and significantly impacts the algorithmic convergence.
Additionally, the action also includes a target velocity for the agent, to
help compensate for the effect of wind by attempting to move faster.

Figure 1: Navigation Task

Thus, given a state $s_{t}=[x,y,z,r,p,y,v_{x},v_{y},v_{z},\allowbreak
w_{x},w_{y},w_{z}]$ the action space for the reinforcement learning agent
consists of predicting the relative coordinates and the target velocity, i.e.
$\mathcal{A}=[\Delta x,\Delta y,\Delta z,v_{target}]$. To avoid taking large
strides leading to potentially dangerous situations, the relative coordinates
have an upper limit (equation 2). This essentially means that given a state
$s_{t}$, the agent can move anywhere within a cube of 0.05 m, traveling a
distance of $\sqrt{|\Delta x|^{2}+|\Delta y|^{2}+|\Delta z|^{2}}$.
Additionally, the target velocity $v_{target}$ is a scalar quantity with which
the agent desires to move to the next location. The individual velocity
components along $X,Y,Z$ directions are computed using equation 3, the
absolute value ensures that the target velocities stay positive and the
velocities are clipped by a safe upper limit $v_{max}$.

$|\Delta x|\leq 0.05,|\Delta y|\leq 0.05,|\Delta z|\leq 0.05$ (2)

$v=\begin{bmatrix}v_{x}\\\ v_{y}\\\ v_{z}\\\
\end{bmatrix}=clip\left(v_{target}*\begin{bmatrix}|\Delta x|\\\ |\Delta y|\\\
|\Delta z|\\\ \end{bmatrix},v_{max}\right)$ (3)

Thus, given the current state vector
$s_{t}=[x,y,z,r,p,y,v_{x},v_{y},v_{z},w_{x},w_{y},w_{z}]$ and an action
$a_{t}=[\Delta x,\Delta y,\Delta z]$, the ideal next state to be in would be
$s_{t+1}=[x+\Delta x,y+\Delta y,z+\Delta
z,r^{\prime},p^{\prime},y^{\prime},v_{x}^{\prime},v_{y}^{\prime},v_{z}^{\prime},w_{x}^{\prime},w_{y}^{\prime},w_{z}^{\prime}]$.
Defining the action space this way serendipitously enables us to define a
partial state transition model (equation 4), where the covariance matrix of
the normal distributions is assumed to be the deviations from the ideal state
due to the wind disturbances acting on the system. In an ideal scenario, the
PID control would ensure that the quadrotor reaches the expected mean of the
normal distribution.

$\begin{bmatrix}X_{t+1}\\\ Y_{t+1}\\\ Z_{t+1}\\\
\end{bmatrix}\sim\mathcal{N}\left(\begin{bmatrix}x_{t}+\Delta x_{t}\\\
y_{t}+\Delta y_{t}\\\ z_{t}+\Delta z_{t}\\\ \end{bmatrix},\Sigma_{t+1}\right)$
(4)

Equation 4 now serves as a partial state transition probability model
$P(s_{t+1}|s_{t},a_{t})$, which can be thought of as additional set
constraints (equations 6 \- 8) for the RL agent, i.e. the ability to
successfully execute the actions by reaching the ideal expected next state. We
can now augment our original system state-space with 3 new quantities
$x_{e},y_{e},z_{e}$ serving as a notion of “action errors” along $X,Y,Z$
directions (equation 5).

$\mathcal{S}_{e}=[\mathcal{S};\ x_{e},\ y_{e},\ z_{e}]$ (5)

$x_{t+1}=x_{t}+\Delta x$ (6)

$y_{t+1}=y_{t}+\Delta y$ (7)

$z_{t+1}=z_{t}+\Delta z$ (8)

As the task involves reaching a predefined destination
$s_{d}=[x_{d},y_{d},z_{d}]$, one of the obvious reward objectives (given a
state action pair $(s_{t},\pi(s_{t}))$) could be the negative of squared
distance from the destination. Equation 9 concerns the navigation part of the
task. Thus the objective is to maximize the reward, with zero being the
highest reward.

$R_{nav}(s_{t},\pi(s_{t}))=-\left((x_{t+1}-x_{d})^{2}+(y_{t+1}-y_{d})^{2}+(z_{t+1}-z_{d})^{2}\right)$
(9)

$R_{e}(s_{t},\pi(s_{t}))=-\left(x_{e}^{2}+y_{e}^{2}+z_{e}^{2}\right)$ (10)

The other requirement is to reach the goal by executing an ideal or expected
behavior. We can now enforce the constraints (equations 6 \- 8), by
introducing a new reward objective to drive the action errors to zero.
Equation 10 defines the action error reward given the extended state (equation
5). Now, it is imperative to rightly define these action errors. A
straightforward way would be to simply have these errors as the difference
between the next observed state and the expected next state from the model
description (equation 4), i.e. set $x_{e}=x_{t+1}-(x_{t}+\Delta
x),y_{e}=y_{t+1}-(y_{t}+\Delta y),z_{e}=z_{t+1}-(z_{t}+\Delta z)$. However,
the RL agent can easily exploit the second reward objective (equation 9) by
taking a small step to reset the reward to 0 and then followed by a longer
inconsistent stride. To circumvent this, we can introduce a cumulative error
term in the equation (equation 3), where $\alpha<1$ is a discount factor and
$x_{e}^{\prime}$ is the previous error (from the previous timestep). The
discount factor was observed to be crucial, which would otherwise induce too
many oscillations in the control analogous to pure integral control.
Similarly, we define the error states $y_{e},z_{e}$.

$\displaystyle
x_{e}=\Sigma_{t^{\prime}=0}^{t}\alpha^{t-t^{\prime}}(x_{t+1}-(x_{t}+\Delta
x_{t}))$ $\displaystyle=\alpha(x_{t+1}-(x_{t}+\Delta x_{t}))+\alpha
x_{e}^{\prime}$ (11)

The reward function from equation 10 resembles a typical LQR cost function,
however, there are stark differences when looked at closely. Consider the
partial state of the system encapsulating the position of the quadrotor in
space, $p_{t}=[x_{t},\ y_{t},\ z_{t}]^{T}$. Assuming the dynamics to be of the
form $p_{t+1}=\Phi p_{t}+\Gamma u_{t}$, a typical LQR cost would look like
$\Sigma_{t}p_{t}^{T}Q_{1}p_{t}+u_{t}^{T}Q_{2}u_{t}+p_{t}Q_{12}u_{t}$. In the
case of a stochastic LQR problem, the dynamics would often evolve as
$p_{t+1}=\Phi p_{t}+\Gamma u_{t}+Gw_{t}$, where $w_{t}$ is usually assumed to
be a Gaussian noise with zero mean, which would yield the same expected cost
function as in the deterministic LQR case. However, the point to note here is
that the standard LQR cost penalizes the magnitude of the control signal
$u_{t}$, while the RL objective function proposed in equation 10 does not,
instead, the penalty would be proportional to the deviation from the expected
next state $x_{t}+u_{t}$ defined by the dynamics. The controller is free to
choose an arbitrarily large control signal as long as it can execute it
successfully, this contrasts a standard LQR control approach. This will be
evident from experiments carried out in the subsequent sections. The same
argument can also be extended to other error states $y_{e},z_{e}$.

Given that $\mathbb{E}[p_{t+1}]=\Phi p_{t}+\Gamma u_{t}$ (since
$\mathbb{E}[w_{t}]=0$), one might wonder if it would be redundant to optimize
the agent to counteract the disturbance in the first place. In this context
let $e_{t}=[x_{e},\ y_{e},\ z_{e}]$ be the deviation of the agent from its
ideal state after executing the action $u_{t}$ from state $p_{t}$ as a
consequence of $w_{t}$ (external wind, actuator noise, noise sensor data,
etc). The net effect of $e_{t}$ can thus be measured as the Euclidean distance
from its ideal next state $p_{t}+u_{t}$ (equation 12.

$e_{t}=\sqrt{x_{e}^{2}+y_{e}^{2}+z_{e}^{2}}$ (12)

To analyze the characteristics of the disturbances acting on the system, a
random walk was simulated by sampling the high-level actions uniformly as per
equation 2 serving as relative coordinates for the quadrotor to move to and
the low-level PID controller executing the actions while subjecting the
quadrotor to random step disturbances (figure 3). The resulting error
distributions are shown in figure 2(b). The individual distributions of
$x_{e},y_{e},z_{e}$ are normally distributed with zero means (figure 2(b)).
The means were estimated as $\mathbb{E}[x_{e}]=\frac{1}{N}\
\Sigma_{i}x_{e_{i}}$ and the covariances as
$\mathbb{C}[x_{e},y_{e}]=\frac{1}{N}\
\Sigma_{i}x_{e_{i}}y_{e_{j}}-\mathbb{E}[x_{e}]\mathbb{E}[y_{e}]$ (similarly
$\mathbb{C}[x_{e},z_{e}],\mathbb{C}[y_{e},z_{e}]$), table 1. It can be
observed that although the means are close to zero, the covariances seem quite
high to ignore. The error $e_{t}$ can now be represented by a multivariate
normal distribution using the estimated means and covariances (table 1) and
the expected value of equation 12 can be calculated by sampling the
probability density function and approximating the expectation by averaging
under the law of large numbers (equation 3). The resulting estimate was found
to be $0.08168$ (meters), i.e. the quadrotor is was found to be off by a
magnitude of 8 cm from its ideal next state. This can also be verified by
inspecting figure 2(c), which shows the magnitude of the error (the difference
between the magnitudes of predicted action and the executed maneuver), and the
mean of the distribution exactly matches the estimated deviation of 8 cm. It
can also be observed that the error can even grow as large as a staggering 13
cm per step, which is well above the specified action limits (equation 2).
Thus, the conclusion is that although the error distributions can be modeled
as a zero mean noise acting on the system, it would be unwise to not
compensate for them, at least in this particular task.

Figure 2: Random Walk Experiment

(a) Trajectory

(b) Error Distribution

(c) Error Magnitude ($|\text{action}|_{2}-|\text{executed}|_{2}$, in meters,
the lower the better)

Table 1: Error Statistics | Covariances | Means
---|---|---
| $x_{e}$ | $y_{e}$ | $z_{e}$ |
$x_{e}$ | 0.07218 | 0.02731 | 0.01915 | $-0.001759$
$y_{e}$ | 0.02731 | 0.03799 | 0.01650 | $-0.002713$
$z_{e}$ | 0.01915 | 0.01650 | 0.03614 | $0.0055015$

$\displaystyle\mathbb{E}\left[e_{t}\right]$ $\displaystyle=\int_{e_{t}}e_{t}\
P(e_{t})\ de_{t}$
$\displaystyle=\int_{x_{e}}\int_{y_{e}}\int_{z_{e}}\sqrt{x_{e}^{2}+y_{e}^{2}+z_{e}^{2}}\
P(x_{e},y_{e},z_{e})\ dx_{e}\ dy_{e}\ dz_{e}$
$\displaystyle\approx\lim_{N\to\infty}\frac{1}{N}\sum_{i}\sqrt{x_{e}^{2}+y_{e}^{2}+z_{e}^{2}}\
\ ,\ \left[\begin{array}[]{l}x_{e}\\\ y_{e}\\\
z_{e}\end{array}\right]\sim\mathcal{N}(\mu_{e},\Sigma_{e})$ (16)

##  4 Single-objective Reinforcement Learning

The goal here is to learn a strategy to navigate the quadrotor to a fixed
point in space. The quadrotor starts off at an initial state $s_{0}$ and
learns to navigate to the point $[0,0,1]$ during the training phase. To test
the theory and claims made in the previous sections, a rigorous comparison is
made on the following approaches:  (i) Baseline model: This is a vanilla
reinforcement learning model formulated for the navigation task. It uses the
kinematics state space (equation 1). The reward function only consists of the
navigation reward (equation 9) and the agent is not subjected to any form of
disturbances during the training phase.  (ii) Baseline model with
disturbances: The formulation is very similar to the baseline model (i),
except that the model experiences disturbances during the training phase.
(iii) Model with error correction (proposed approach for robustness): This is
a model with disturbances (ii) but uses the extended state space (equation 3)
and also receives a feedback regarding the state errors which was introduced
earlier(equation 10). Since the model sees two distinct reward objectives, the
expected scalarized return (ESR) approach is employed by defining a utility
function to scalarize the rewards prior to the expectation (equation 17).
(iv) An LSTM model: The problem setting is exactly similar to (iii) except
that the reinforcement learning agent has an LSTM architecture. Having the
LSTM architecture dissents from the rest of the options adhering to the MDP
assumptions, the motivation for including the LSTM architecture in the
comparison was to explore the possibility of the RL agent learning to behave
robustly given a sequence of previous states, i.e. whether a pure deep
learning architecture can render the proposed method obsolete.  To draw a fair
comparison, all of the above approaches use the exact same underlying
reinforcement learning algorithm, Proximal Policy Gradient (PPO) [Schulman et
al., 2017b]. PPO is a stochastic policy gradient algorithm that improves on
the prior policy gradient implementations like Trusted Region Policy
Optimization [Schulman et al., 2017a]. The algorithmic implementations are
made available by stable-baselines3 [Raffin et al., 2021].

$U\left(\begin{bmatrix}R_{nav}\\\
R_{err}\end{bmatrix}\right)=w^{T}\begin{bmatrix}R_{nav}\\\
R_{err}\end{bmatrix}=\left[1\ \ 0.5\right]\begin{bmatrix}R_{nav}\\\
R_{err}\end{bmatrix}$ (17)

The disturbances during the training phase are feeble step/pulse disturbances
acting along one or all of the $X,Y,Z$ directions emulating a wind force.
Figure 3 shows the disturbances applied during the training phase, the
magnitude and/or the direction of the disturbances change every 20 simulation
steps, with the intention of simulating real-world disturbances that are not
completely random and for the RL agent to make ad-hoc adjustments to adapt to
the disturbances. During the evaluation phase, the magnitude and the direction
of the step disturbance remain fixed throughout and the performance is
evaluated against disturbances of varying magnitudes with much stronger
intensities. The initial state is set to $[0,\ 0,\ 0]$ at the start of every
training episode. Although the RL agent would benefit from random
initialization, the intention is to keep the comparison fair for all the
approaches by eliminating the randomness in the environment. However, the
initial state is offset during the evaluation phase to look at the agent’s
behavior to work with and against the disturbances acting on it. Table 3 lists
the training hyperparameters.

Figure 3: Training Disturbances

$X,Y,Z$ disturbances applied during training. The disturbances are flipped
every 20 simulation steps.

Table 2: Training Hyperparameters Hyperparameter | Value
---|---
Episode Length (in secs) | 2
Initial XYZ state | $[0,\ 0,\ 0]$
Disturbance $|X_{max}|$ | 0.05
Disturbance $|Z_{max}|$ | 0.05
Disturbance $|XYZ_{max}|$ | 0.025
Disturbance Flip Frequency | 20 simulation steps
Number of Timesteps | 1,000,000
Neural Net Policy Architecture (hidden layers) | $[64,\ 64]$

Multiple evaluation metrics are formulated to effectively assess and compare
the pros and cons of the different approaches described above. We obviously
have the negative of squared reward and a few other metrics to evaluate the
robustness. The distance traveled $\Sigma_{t}|p_{t}-p_{p+1}|_{2}$ is measured
as the physical distance traveled by the agent while executing the trajectory
as opposed to the typical cumulative RL reward. Here $p_{t}$ denotes the state
comprising of the XYZ coordinates of the quadrotor $p_{T}=[x_{t},\ y_{t},\
z_{t}]$. The smoothness of the trajectory is measured as the mean of the
second derivative, i.e. the rate of change of direction of the quadrotor,
$\frac{1}{N}\int|\tau^{\prime\prime}(t)|_{2}dt$. Ideally, we would desire the
system to navigate in a smooth trajectory, as opposed to a zig-zag or jagged
trajectory. As the ascent phase of the quadrotor is crucial prior to it
reaching the destination and start hovering, the average ascent step is
computed during the initial few timesteps of the trajectory. A plot of
intended minus the executed distance ($|u_{t}|_{2}-|p_{t}-p_{t+1}|_{2}$) is
tracked, ideally, we would like this to be zero, i.e. the quadrotor should be
able to execute the actions successfully. Finally, the “converged” metric
indicates whether the quadrotor managed to reach the destination within a 10
cm accuracy.

Figures 4(a) \- 4(c) plot the evaluated reward functions for different
combinations and magnitudes of XYZ disturbances. The evaluations were captured
by setting the disturbances to be constant throughout the episode, unlike the
training phase where the disturbances change directions randomly. The curves
are asymmetric due to the fact that the initial state of the drone is moved to
$[2,\ 0,\ 0]$ for evaluation. So, the drone is assisted by the wind when the
disturbance is along the negative X direction and opposed when positive. The
first thing to notice is that the baseline model, which was not exposed to any
form of disturbance during the training phase outperforms the rest of the pack
by a considerable margin in terms of the cumulative RL reward, albeit the
other models achieved a higher reward in a few extreme cases (figure 4(b)).
This finding corroborates the claims made by [Glossop et al., 2022]. However,
the reward functions do not paint the complete picture. Figures 5(a) \- 5(c)
plot the evaluation trajectories of policies for different cases of
evaluation. Looking closely at the trajectories we find that the baseline
model takes longer strides and manages to reach the in fewer RL steps, while
the rest of the pack takes a considerably more number of RL steps during
ascent. This explains the reason why the baseline model achieves the best RL
reward, as the RL reward is a cumulative discounted sum of the reward, the
more RL steps the policy executes, the more negative reward is accumulated.
While achieving a high RL reward is always the goal in reinforcement learning,
it can also have adverse consequences. The figures 5(d) \- 5(f) spotlight the
underlying problem, the baseline model appears to be a true outlier. As per
the design formulation 2, the maximum possible relative distance the agent was
supposed to execute was $\sqrt{0.05^{2}+0.05^{2}+0.05^{2}}=0.0866\ \text{m or
8.66 cm}$. However, the baseline model (green curve) seems to be encountering
an enormous amount of action error, starting off from a staggering 30 cm and
then gradually decreasing to 0 cm after the ascent period. An error of 30 cm
essentially means that the quadrotor was off by 30 cm from where it planned to
go by executing the particular action. In contrast, the models trained with
disturbances (the rest of the pack) appear to be much more stable, and the
proposed model (“dist-err”) in particular appears to do a better job, implying
that the agent was able to execute the actions with sufficient certainty. This
further supports the assumptions made by simulating the random walk experiment
earlier, that ignoring the effects of disturbances by assuming the normal
nature of the errors (figure 2(b)) would not be the best thing to do, because
although the expected values of errors are zero, the effect on the quadrotor
is actually $\mathbb{E}[\sqrt{x_{e}^{2}+y_{e}^{2}+z_{e}^{2}}]$ which was found
to be much larger and not equal to zero. Table 3 further lists more insightful
metrics captured and spotlights the proposed method. Although the “baseline”
model outperformed in terms of the RL reward, surprisingly it does not hold up
in terms of the actual physical distance traveled by the quadrotor. “dist-err”
manages to reach the destination of $[0,\ 0,\ 1]$ from $[2,\ 0,\ 0]$ by
traveling a significantly shorter distance compared to the baseline. The rest
of the models perform similarly, but “dist-err” appears to have an edge in
most cases. This can also be viewed by inspecting the trajectory plots
(figures 5(a) \- 5(c)), where the trajectory appears to be much more focused
for “dist-err”. The “dist-err” model also appears to be much smoother, this is
evident from the fact that the mean of the magnitude of the second derivate is
much smaller (group 2, table 3). All the models trained with disturbance seem
to perform similarly when it comes to the average ascent step, albeit “dist-
err-u” with a slight edge. However, the “baseline” again appears to be an
outlier, taking significantly longer strides during the ascent phase. It can
also be observed that the “dist-err” model appears to converge to the goal
under all the experiments except the last one with a disturbance of 0.100
acting along a positive Z direction. While the rest of the models fail to
reach the goal under multiple scenarios.

The conclusion we draw from these results is that a higher RL reward does not
necessarily imply the inherent robustness of the models to all aspects of
evaluation. There could be severe consequences or adverse effects surrounding
the process dynamics or other relevant metrics to look at to truly assess the
robustness. Thus, enforcing the dynamics by a reward function (model-assisted)
learning appears to have some positive effect with regards to the certainty of
RL algorithms.

Figure 4: Evaluation Rewards.
Models compared: i. lstm: an LSTM agent. ii. dist-err: the proposed arch for
single-objective, receives feedback for the action errors. iii. dist-err-u:
similar to LQR cost, agent receives a negative L2 reward of the control/action
taken. iv. dist: baseline model subjected to disturbances during training. v.
baseline: the baseline model.

(a) Disturbance along X

(b) Disturbance along Z

(c) Disturbance along XYZ

Figure 5: Evaluation Trajectories and Action Errors During Ascent.
The figures on top plot the trajectories of the agent during the evaluation
phase. The figures at the bottom compare the action errors
$|\text{action}|_{2}-|\text{executed}|_{2}$ (in meters, the lower the better).
Models compared: i. lstm: an LSTM agent. ii. dist-err: the proposed arch for
single-objective, receives feedback for the action errors. iii. dist-err-u:
similar to LQR cost, agent receives a negative L2 reward of the control/action
taken. iv. dist: baseline model subjected to disturbances during training. v.
baseline: the baseline model.

(a) Disturbance: -0.050 along X

(b) Disturbance: +0.050 along Z

(c) Disturbance: +0.075 along XYZ

(d) Disturbance: -0.050 along X

(e) Disturbance: +0.050 along Z

(f) Disturbance: +0.075 along XYZ

Table 3: Single-objective Navigation Metrics.
Models compared: i. lstm: an LSTM agent. ii. dist-err: the proposed arch for
single-objective, receives feedback for the action errors. iii. dist-err-u:
similar to LQR cost, agent receives a negative L2 reward of the control/action
taken. iv. dist: baseline model subjected to disturbances during training. v.
baseline: the baseline model.

Distance Travelled Smoothness Average Ascent Step (L2 Norm of Action)
Converged Direction Mag. Dist. lstm dist-err dist-err-u dist baseline lstm
dist-err dist-err-u dist baseline lstm dist-err dist-err-u dist baseline lstm
dist-err dist-err-u dist baseline x -0.050 6.599416 6.132900 5.714949 6.111227
6.618811 0.009211 0.008646 0.006601 0.007670 0.032359 0.138675 0.099394
0.173148 0.123149 0.383917 Yes Yes Yes Yes Yes x -0.100 6.499861 5.331071
5.918821 5.719601 5.798424 0.007364 0.005405 0.006002 0.005149 0.022950
0.156945 0.114534 0.168128 0.112654 0.411196 Yes Yes Yes Yes Yes x 0.000
7.603805 5.364380 5.280013 6.599345 5.843086 0.011430 0.005480 0.005477
0.011106 0.029162 0.155782 0.129354 0.133528 0.099125 0.380344 Yes Yes Yes Yes
Yes x 0.050 5.256427 4.934614 5.306519 7.602165 6.909383 0.040803 0.003857
0.004457 0.015453 0.036542 0.968272 0.099889 0.164785 0.099618 0.378084 No Yes
Yes Yes Yes x 0.100 8.016002 5.009077 5.364408 7.573557 4.562196 0.232892
0.003696 0.004433 0.015047 0.033325 1.349194 0.125718 0.173467 0.120015
0.599928 No Yes Yes Yes No xyz -0.075 6.836447 5.335730 5.959134 6.811069
6.524232 0.010284 0.005464 0.007842 0.012476 0.034549 0.131541 0.111868
0.127855 0.116927 0.384197 Yes Yes Yes Yes Yes xyz -0.150 7.568094 6.465555
6.239807 6.259547 15.359850 0.011566 0.009653 0.009864 0.008517 0.117175
0.126255 0.126986 0.148956 0.117847 0.372258 Yes Yes Yes Yes Yes xyz 0.000
7.425029 4.861160 4.859477 6.655158 5.883497 0.010969 0.003722 0.004404
0.010254 0.026074 0.152020 0.098812 0.157771 0.129522 0.379013 Yes Yes Yes Yes
Yes xyz 0.075 7.115747 4.965403 5.040673 6.277422 7.427389 0.009977 0.003966
0.004081 0.008231 0.049423 0.167109 0.138969 0.141843 0.115283 0.376754 Yes
Yes Yes Yes Yes xyz 0.150 1.359120 4.844337 4.875827 5.126340 0.883666
0.014639 0.003606 0.002904 0.003983 0.001789 0.271770 0.149187 0.187735
0.133928 0.176308 No Yes No Yes No z -0.050 7.723198 5.668666 6.375411
8.300653 7.623795 0.019203 0.008097 0.012004 0.018782 0.048181 0.121373
0.096880 0.157460 0.098883 0.393317 Yes Yes Yes Yes Yes z -0.100 0.015911
10.773465 11.093853 11.260336 12.575253 0.000002 0.040536 0.043189 0.043243
0.066770 0.002792 0.073959 0.118678 0.079583 0.370222 No Yes No Yes Yes z
0.000 7.511052 4.978000 5.233706 6.299238 6.076239 0.011097 0.004251 0.005609
0.007886 0.027799 0.148883 0.121492 0.136734 0.105076 0.402755 Yes Yes Yes Yes
Yes z 0.050 6.084520 4.561033 4.835022 5.856955 5.789299 0.006865 0.003129
0.002743 0.006014 0.030311 0.164402 0.154412 0.178154 0.143068 0.391134 Yes
Yes Yes Yes Yes z 0.100 1.208510 11.522041 5.586891 4.752776 1.591956 0.007685
0.635960 0.004139 0.002017 0.018765 0.168021 0.143042 0.199129 0.154843
0.287403 No No No No No

##  5 Reachability Analysis of Single-objective Reinforcement Learning

This section focuses on further analysis of the single-objective reinforcement
learning solutions described in the previous sections. Reinforcement learning-
based control can be thought of as an optimal control strategy, which means
that the RL agent predicts optimal action given a specific state, and
ultimately over time, the agent will meet the reward constraint. This also
means that the classical control verification methods can no longer be applied
for stability and robustness. However, Hamilton-Jacobi (HJ) reachability
analysis is a prominent method when it comes to formal verification of optimal
control strategies. [Bansal et al., 2017] provide an extensive overview of
both conventional and contemporary ways of performing reachability analysis.
Typically, reachability analysis involves computing a “reachability set”,
given a goal state, a reachability set is a set of all possible states from
which the system can reach the given goal state exactly at time $\tau$. A
backward reachability tube is defined as the set of all possible states from
which the system can reach the goal state within a duration of time $\tau$. If
the goal state is a desirable destination, we would like the control policy to
have a large set, if on the other hand, the destination is unsafe, we desire
our policy to have a minimal set or preferably a null set that can avert the
unsafe states. There are several ways of computing these sets for optimal
control policies [Chen et al., 2017, Kaynama and Oishi, 2013, Niarchos and
Lygeros, 2006, Rubies-Royo and Tomlin, 2017], that make computation of
reachability sets feasible for non-linear systems.

Computing the backward reachability set (BRS) or backward reachability tube
(BRT) via the level-set method involves solving the Hamilton-Jacobi-Bellman
partial differential equation. Given an initial state $x$ and a time period
$\tau$, the Bellman equation can be written as equation 18, where $C$ is the
cost function $C:\mathbb{R}^{d}\times\mathbb{R}^{a}\to\mathbb{R}$ and $D$ is
the terminal cost function $D:\mathbb{R}^{d}\to\mathbb{R}$. The HJ partial
differential equation can be written by Taylor expanding the Bellman equation
as in equation 19, where $\nabla V=\frac{\partial V}{\partial x}$ denotes the
spatial derivative, i.e. the gradient of the cost function with respect to the
state variables, and $F$ denotes the system dynamics, i.e. the next state
given the current state $x$ and action $u$ (derived from the policy $\pi$). If
we have an adversarial input or disturbance $d$ acting on the system, we would
be interested in minimizing the value function with respect to $u$ while
maximizing the cost with respect to the adversarial disturbance $d$ (equation
20), meaning we would like the control input to minimize the value function
making it feasible for the system to reach the goal starting from an initial
state, while the adversarial attempts to make it harder for the agent to reach
its goal.

$V(x,t)=\min_{u}\int_{t=0}^{\tau}C(x_{t},u_{t})dt+D(x_{\tau})$ (18)

$V(x+dx,t+dt)=\frac{dV(x,t)}{dt}+\min_{u\sim\pi}\nabla V\cdot F(x,u)+C(x,u)$
(19)

$V(x+dx,t+dt)=\frac{dV(x,t)}{dt}+\min_{u\sim\pi}\max_{d}\nabla V\cdot
F(x,u,d)+C(x,u)$ (20)

In our case, the policy $\pi$ is given by a reinforcement learning agent which
is already optimized with respect to the reward function (equations 9 and 10),
so we are only interested in studying the effect of $d$ by looking at the
tolerance of the policy $\pi$. To approximate the BST for our RL policy, we
can formulate an experiment where $\pi$ emulates an optimal control policy
$\pi^{*}$ and $d$ can then emulate the notion of action errors from the
earlier experiments carried out.

Going back to the quadrotor navigation example, where the objective is to
navigate and hover at the destination $x,y,z=(0,0,1)$. Consider the partial
states of the system relevant to the navigation task
$p_{t}=[x_{t},y_{t},z_{t}]$ (ignoring the linear and angular velocities which
can be assumed to be constant, equation 1). As before, the control signal or
the action is the relative coordinate for the quadrotor to move to,
$a_{t}=u_{t}=[\Delta x,\Delta y,\Delta z]$. And let $d_{t}$ denote the
deviation or the noise acting on the system. Since the destination coordinate
is $(0,0,1)$ we can formulate the system dynamics for the optimal control
problem by a set of linear equations (equation 5), where the reference signal
$r=[r_{x},r_{y},r_{z}]=[0,0,1]$ and $d_{x},d_{y},d_{z}$ are assumed to be
Gaussian noise. Equation 5 represents the $F(x,u,d)$ term in the Hamilton-
Jacobi-Bellman partial differential equation (equation 20). The term $\nabla
V\cdot F$ can further be decomposed as $\nabla V\cdot F(x,u,d)=\nabla V\cdot
F_{1}(x,u)+\nabla V\cdot F_{2}(x,d)$ as the disturbances act additively on the
system (equation 5). This yields equation 22, where $F_{1}$ is the ideal
dynamics of the system given the state and action, while $F_{2}$ is the
dynamics capturing the disturbance.

$\displaystyle x_{t+1}=r_{x}-x_{t}+\Delta x+d_{x}=0-x_{t}+\Delta x+d_{x}$
$\displaystyle y_{t+1}=r_{y}-y_{t}+\Delta y+d_{y}=0-y_{t}+\Delta y+d_{y}$
$\displaystyle z_{t+1}=r_{z}-z_{t}+\Delta z+d_{z}=1-z_{t}+\Delta z+d_{z}$ (21)

$V(x+dx,t+dt)=\frac{dV(x,t)}{dt}+\min_{u\sim\pi}\nabla V\cdot
F_{1}(x,u)+\max_{d}\nabla V\cdot F_{2}(x,d)+C(x,u)$ (22)

The RL policy $\pi$ was trained to maximize the negative of the distance from
the reference point (equation 9). Thus, the value function $V$ can simply
represent the distance from the reference point $r=[r_{x},r_{y},r_{z}]$. The
solution to the optimal control problem (equation 5) is
$(x_{t},y_{t},z_{t})=(r_{x},r_{y},r_{z})$. As the value function is known, we
can use an optimal bang-bang control policy $\pi^{*}$ to emulate the optimal
RL policy $\pi$ and evaluate the robustness by imposing the deviations $d$
experienced by the different RL agents. This would make it feasible to
approximate the backward reachability tube using the classic level-set method.
The assumption we make here is that the optimal RL policy under no disturbance
is similar to a bang-bang control policy, which is valid because the bang-bang
policy optimizing the HJ PDE follows the optimal path and so does the RL agent
when no disturbances are acting on the system.

Given the state $p_{t}=[x_{t},y_{t},z_{t}]$, the value function is given by
equation 23. The spatial gradients of the value function is the partial
derivatives with respect to the state variables (equation 24). A bang-bang
control policy to minimize the HJ PDE will be to use a step in the opposite
direction of the spatial gradient (equation 25), and the optimal disturbance
inhibiting the agent from reaching its destination or attempting to maximize
the HJ PDE is given by equation 26. Figures 6(a) and 6(b) visualize the
optimal actions, we observe that the bang-bang policy drives the agent to the
destination $(0,0,1)$.

$V(p_{t})=V([x_{t},y_{t},z_{t}])=\sqrt{x_{t}^{2}+y_{t}^{2}+(1-z_{t})^{2}}$
(23)

$\nabla V(p_{t})=\begin{bmatrix}\frac{\partial V}{\partial x}\\\
\frac{\partial V}{\partial y}\\\ \frac{\partial V}{\partial z}\\\
\end{bmatrix}=\frac{1}{\sqrt{x_{t}^{2}+y_{t}^{2}+(1-z_{t})^{2}}}\begin{bmatrix}x_{t}\\\
y_{t}\\\ -(1-z_{t})\end{bmatrix}$ (24)

$u_{i}=\begin{cases}-|\Delta u_{max}|\ \text{if}\frac{\partial V}{\partial
x_{i}}>0\\\ |\Delta u_{max}|\ \text{if}\frac{\partial V}{\partial
x_{i}}<0\end{cases}$ (25)

$d_{i}=\begin{cases}|\Delta d_{max}|\ \text{if}\frac{\partial V}{\partial
x_{i}}>0\\\ -|\Delta d_{max}|\ \text{if}\frac{\partial V}{\partial
x_{i}}<0\end{cases}$ (26)

Figure 6: Bang-bang Policy

(a) Optimal Controls XY plane. Destination $(0,0)$

(b) Optimal Controls XZ plane. Destination $(0,1)$

The next objective is then to model the disturbances $d$ for the linearized
system. This disturbance should emulate what the RL agent experiences during
the evaluation phase. For the particular experiment, we pick a step wind
disturbance (of magnitude 0.075 N along $XYZ$) acting on the quadrotor during
the episode and simply capture the errors during the flight, the error is just
the difference between the intended position of the quadrotor by executing the
action and where it really ended up, i.e. $x_{t+1}-(x_{t}+u_{t})$. The
statistics of these errors are listed in tables 4 and 5, clearly the
covariances of the dist-err model (proposed model) are significantly lower,
this essentially means that the model was able to execute its actions more
concretely compared to the baseline model. We will now use these disturbances
(1 standard deviations apart) as $d_{x},d_{y},d_{z}$ while computing the BRT.
This would reveal the worst-case tolerances of the models and enable us to
compare the robustness.

Table 4: Error Statistics - baseline model | Covariances | Means
---|---|---
| $x_{e}$ | $y_{e}$ | $z_{e}$ |
$x_{e}$ | 0.05963 | 0.02477 | 0 | $-0.01970$
$y_{e}$ | 0.02477 | 0.09416 | 0.01642 | $-0.0271$
$z_{e}$ | 0 | 0.01642 | 0.08471 | $0.01089$
Table 5: Error Statistics - dist-err model | Covariances | Means
---|---|---
| $x_{e}$ | $y_{e}$ | $z_{e}$ |
$x_{e}$ | 0.03474 | 0 | 0 | $-0.02000$
$y_{e}$ | 0 | 0.02833 | 0.00691 | $0.03506$
$z_{e}$ | 0 | 0.00691 | 0.02764 | $0.010255$

The target set is a sphere of radius 10 cm centered at $[0,0,1]$. As we are
interested in evaluating the ability of the agent to reach the target set, the
BRT computed at the end will comprise of all the possible initial states from
which the agent can navigate to the specified target set. The time duration
for computing the BRT is set to $\tau=3\ \text{s}$, meaning we look backward
starting from our target set for all the possible reachable sets within a
duration of 3s. The robustness of the proposed model is clearly evident
(figures 7(a) and 7(b)). The reachability tube of the proposed model is much
larger, implying that the agent can reach the target set from a wider range of
initial states within the span of 3 seconds. Figure 7(c) compares the 2D
slices of the BRTs at $z=1$. Although these tubes are a crude approximation
due to the fact that the optimal RL policy was emulated by an optimal bang-
bang policy, we get a tangible measure of robustness by computing the BRTs.
These results further corroborate the observations from the previous section.
For instance, the “converged” column in table 3 represents the true
simulations of the trained RL agents, and we see that the baseline model fails
to converge in two additional cases where the proposed model (dist-err) has no
problem reaching the target.

Figure 7: Backward Reachability Tubes (BRT).
Models compared: i. baseline: the baseline model. ii. dist-err: the proposed
arch for single-objective, receives feedback for the action errors.

(a) 3D BRT baseline model

(b) 3D BRT dist-err model

(c) Comparison of BRTs, 2D slice ($z=1$)

##  6 Multi-objective Reinforcement Learning

Inducing the notion of conservativeness can have a positive impact on
interpretability, but it may not be always desired for the model to act
conservatively. In certain applications, the agent may be expected to take a
risk and achieve the best RL reward, on the other hand, in the context of
safety-critical applications, we would want the agent to adhere to certain
physical constraints. In this regard, it would be desirable to have the
ability to tune the level of conservativeness required and dynamically change
it during the runtime. So a multi-objective RL problem can be formulated where
the primary objective is task-specific and additional reward objectives can be
introduced to constrain the agent. We would then employ a scalarized expected
reward (SER) approach to learn an array of RL policies each of which
prioritizes the rewards differently. A typical SER reward for multi-objective
RL similar to equation 27, where $w\in W$ can be a finite set of weight
vectors for the agents to weigh the rewards. Thus, we can learn an array of
policies that can differently prioritize the conservativeness $R_{err}$. And
the key difference to the single-objective RL task (equation 17) is that the
utility function would be applied after the expectation.

$U\left(\mathbb{E}\begin{bmatrix}R_{nav}\\\
R_{err}\end{bmatrix}\right)=w^{T}\begin{bmatrix}\mathbb{E}\left[R_{nav}\right]\\\
\mathbb{E}\left[R_{err}\right]\end{bmatrix}$ (27)

##  7 Connection to Model-based Reinforcement Learning

All the results discussed thus far required formulating a reward objective
that would penalize the agent for deviating from the normal or expected
behavior. The combination of hierarchical control and the specific way of
formulating the actions serendipitously made it feasible to compose an error
model without any additional effort. However, in cases where such a
formulation is not as obvious, we can leverage model-based reinforcement
learning methods. Typically, the model-based reinforcement learning methods
learn the dynamics of the system by fitting a non-linear model (often a neural
network) that can predict the next state given the current state, i.e.
$s_{t+1}=\hat{f}(s_{t},a_{t})$ and leverage it later on in several ways. By
building up a dataset $\mathcal{D}$ comprising of observations
$(s_{t},a_{t},s_{t+1})$ under ideal conditions, we can thus learn the “ideal”
dynamics of the system. Once $\hat{f}$ is learned, the rest of the procedure
would be exactly the same, i.e. introducing a reward objective for the system
to behave conservatively as per the ideal dynamics. When the dynamics span a
high-dimension space, it should be discretionary to choose a subset of states
that make sense, as in the task of quadrotor discussed in the paper, although
the actual state-space of the quadrotor spanned 15 dimensions, only 3 of the
states were absolutely vital for interpretability. Thus, we could formulate a
reward objective for such states.

##  8 Conclusion

Robustness and interpretability are some of the key challenges for
contemporary reinforcement learning-based applications, and the paper presents
a plausible approach to alleviate the issues in the context of standard
reinforcement learning. The approach of introducing a secondary reward
objective constraining the dynamics of the agent was found to have a positive
impact after rigorous testing and comparison. Although the constrained agent
the new objective did not necessarily outperform in terms of the RL reward, it
was observed that the agent exhibited some very nice physical properties under
evaluation and was found to have an edge when looking at a relevant set of
physical metrics for the task involved.

It is important to note that the approach discussed does guarantee that the
process dynamics are adhered to always, unlike MPC-based solutions that
guarantee a feasible solution, this is a distinguishing quality of model
predictive control. However, MPC has its own set of pros and cons compared to
reinforcement learning. This work attempts to improve and elevate the problems
surrounding classical RL in optimal control-based applications. While the task
considered was restricted to a navigation problem of a quadrotor, the ideas
presented can be extended to any non-linear control problem, formulating a
hierarchical control task with a partial dynamics model would be ideal, in the
case where the dynamics are known, a non-linear model could be fit on the data
to estimate the crucial states for the problem, which could then serve as a
ground truth for the RL task.

## References

  * [Anthony et al., 2017] Anthony, T., Tian, Z., and Barber, D. (2017).  Thinking fast and slow with deep learning and tree search.
  * [Arulkumaran et al., 2017] Arulkumaran, K., Deisenroth, M. P., Brundage, M., and Bharath, A. A. (2017).  Deep reinforcement learning: A brief survey.  IEEE Signal Processing Magazine, 34(6):26–38.
  * [Bai et al., 2023] Bai, H., Cheng, R., and Jin, Y. (2023).  Evolutionary reinforcement learning: A survey.  Intelligent Computing, 2.
  * [Bansal et al., 2017] Bansal, S., Chen, M., Herbert, S., and Tomlin, C. J. (2017).  Hamilton-jacobi reachability: A brief overview and recent advances.
  * [Brunke et al., 2021] Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A. P. (2021).  Safe learning in robotics: From learning-based control to safe reinforcement learning.
  * [Chen et al., 2017] Chen, M., Herbert, S. L., Vashishtha, M. S., Bansal, S., and Tomlin, C. J. (2017).  Decomposition of reachable sets and tubes for a class of nonlinear systems.
  * [Feinberg et al., 2018] Feinberg, V., Wan, A., Stoica, I., Jordan, M. I., Gonzalez, J. E., and Levine, S. (2018).  Model-based value estimation for efficient model-free reinforcement learning.
  * [Fujimoto et al., 2018] Fujimoto, S., van Hoof, H., and Meger, D. (2018).  Addressing function approximation error in actor-critic methods.  CoRR, abs/1802.09477.
  * [Glossop et al., 2022] Glossop, C. R., Panerati, J., Krishnan, A., Yuan, Z., and Schoellig, A. P. (2022).  Characterising the robustness of reinforcement learning for continuous control using disturbance injection.
  * [Haarnoja et al., 2018] Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018).  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  CoRR, abs/1801.01290.
  * [Hayes et al., 2022] Hayes, C. F., Rădulescu, R., Bargiacchi, E., Källström, J., Macfarlane, M., Reymond, M., Verstraeten, T., Zintgraf, L. M., Dazeley, R., Heintz, F., Howley, E., Irissappane, A. A., Mannion, P., Nowé, A., Ramos, G., Restelli, M., Vamplew, P., and Roijers, D. M. (2022).  A practical guide to multi-objective reinforcement learning and planning.  Autonomous Agents and Multi-Agent Systems, 36(1).
  * [Kaynama and Oishi, 2013] Kaynama, S. and Oishi, M. (2013).  A modified riccati transformation for decentralized computation of the viability kernel under LTI dynamics.  IEEE Transactions on Automatic Control, 58(11):2878–2892.
  * [Kober and Peters, 2014] Kober, J. and Peters, J. (2014).  Reinforcement Learning in Robotics: A Survey, pages 9–67.  Springer International Publishing, Cham.
  * [Ladosz et al., 2022] Ladosz, P., Weng, L., Kim, M., and Oh, H. (2022).  Exploration in deep reinforcement learning: A survey.  Information Fusion, 85:1–22.
  * [Lee and Qin, 2019] Lee, Y. L. and Qin, D. (2019).  A survey on applications of deep reinforcement learning in resource management for 5g heterogeneous networks.  In 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1856–1862.
  * [Mayne et al., 2005] Mayne, D., Seron, M., and Raković, S. (2005).  Robust model predictive control of constrained linear systems with bounded disturbances.  Automatica, 41(2):219–224.
  * [Mnih et al., 2016] Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. (2016).  Asynchronous methods for deep reinforcement learning.
  * [Morimoto and Doya, 2005] Morimoto, J. and Doya, K. (2005).  Robust Reinforcement Learning.  Neural Computation, 17(2):335–359.
  * [Nagabandi et al., 2017] Nagabandi, A., Kahn, G., Fearing, R. S., and Levine, S. (2017).  Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning.
  * [Niarchos and Lygeros, 2006] Niarchos, K. N. and Lygeros, J. (2006).  A neural approximation to continuous time reachability computations.  In Proceedings of the 45th IEEE Conference on Decision and Control, pages 6313–6318.
  * [Panerati et al., 2021] Panerati, J., Zheng, H., Zhou, S., Xu, J., Prorok, A., and Schoellig, A. P. (2021).  Learning to fly—a gym environment with pybullet physics for reinforcement learning of multi-agent quadcopter control.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7512–7519.
  * [Raffin et al., 2021] Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. (2021).  Stable-baselines3: Reliable reinforcement learning implementations.  Journal of Machine Learning Research, 22(268):1–8.
  * [Razzaghi et al., 2022] Razzaghi, P., Tabrizian, A., Guo, W., Chen, S., Taye, A., Thompson, E., Bregeon, A., Baheri, A., and Wei, P. (2022).  A survey on reinforcement learning in aviation applications.
  * [Rubies-Royo and Tomlin, 2017] Rubies-Royo, V. and Tomlin, C. (2017).  Recursive regression with neural networks: Approximating the hji pde solution.
  * [Schulman et al., 2017a] Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. (2017a).  Trust region policy optimization.
  * [Schulman et al., 2017b] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017b).  Proximal policy optimization algorithms.
  * [Silver et al., 2014] Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014).  Deterministic policy gradient algorithms.  In Xing, E. P. and Jebara, T., editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 387–395, Bejing, China. PMLR.
  * [Srinivasan et al., 2020] Srinivasan, K., Eysenbach, B., Ha, S., Tan, J., and Finn, C. (2020).  Learning to be safe: Deep rl with a safety critic.
  * [Sutton and Barto, 2018] Sutton, R. S. and Barto, A. G. (2018).  Reinforcement Learning: An Introduction.  The MIT Press, second edition.
  * [Thananjeyan et al., 2020] Thananjeyan, B., Balakrishna, A., Rosolia, U., Li, F., McAllister, R., Gonzalez, J. E., Levine, S., Borrelli, F., and Goldberg, K. (2020).  Safety augmented value estimation from demonstrations (saved): Safe deep model-based rl for sparse cost robotic tasks.