# Feedback Linearization of Car Dynamics for Racing via Reinforcement Learning

Michael Estrada, Sida Li , Xiangyu Cai  Electrical Engineering and Computer
Science
University of California, Berkeley
Berkeley, US
Email: {mkestrada2, listar2000<EMAIL_ADDRESS>
###### Abstract

Through the method of Learning Feedback Linearization, we seek to learn a
linearizing controller to simplify the process of controlling a car to race
autonomously. A soft actor-critic approach is used to learn a decoupling
matrix and drift vector that effectively correct for errors in a hand designed
linearizng controller. The result is an exactly linearizing controller that
can be used to enable the well developed theory of linear systems to design
path planning and tracking schemes that are easy to implement and
significantly less computationally demanding. To demonstrate the method of
feedback linearization, it is first used to learn a simulated model whose
exact structure is known, but varied from the initial controller, so as to
introduce error. We further seek to apply this method to a system that
introduces even more error in the form of a gym environment specifically
designed for modelling the dynamics of car racing. To do so, we posit an
extension to the method of learning feedback linearization; a neural network
that is trained using supervised learning to convert the output of our
linearizing controller to the required input for the racing environment. Our
progress towards these goals is reported and next-steps in their
accomplishment are discussed.

###### Index Terms:

Autonmous Racing, Learning Feedback Linearization

##  I Introduction

The control of nonlinear systems is traditionally considered challenging
because of the many, and varied classes of nonlinear systems that each display
unique behaviors. typically this needs to be accounted for on a case-by-case
basis. The field of geometric control seeks to address this by taking
advantage of the structure of the system’s dynamics to simplify related tasks
like trajectory tracking and planning [1, 2]. The fundamental issue with this
approach is the necessity for accurate knowledge of the system’s dynamics,
which can involve a great deal of system identification work or otherwise
simplifying the model of the dynamics. This is in contrast to the techniques
of model free reinforcement learning, [3, 4, 5] which can be used to
automatically compute optimal control strategies with no prior knowledge of
the system’s dynamics. Recent work in this direction has shown some promise
but remains difficult in many cases due to poor sample complexity in many
robotics applications [6, 7, 8]. In this paper, we seek to demonstrate the
technique of Learning Feedback Linearization (LFBL), first proposed in [9].
LFBL is designed to exploit the advantages of both geometric control and
model-free reinforcement learning to optimize a geometric controller for a
plant with unknown dynamics and limited structural assumptions about those
dynamics. Specifically, it seeks to learn an optimal implementation of a
technique called Feedback Linearization (FBL). A successful feedback
linearization scheme has the extremely useful property of rendering the input-
output relationship of an otherwise nonlinear plant to be linear. This then
enables the use of path planning and control techniques from the much more
broadly applicable, and easily applied, field of linear systems theory [10,
11, 12]. While this technique has been used extensively in robotics, the
fundamental concern with feedback linearization is the necessity for highly
accurate models of the dynamics governing the robot [13, 14, 15, 1, 2]. The
field of adaptive control has seen extensive work on the construction of
exactly linearizing controllers [16, 17, 18, 19, 20, 21, 22, 23], but this too
has the limitation of needing highly structured representations of how and
where the nonlinearities arise in the system’s dynamics. The method of LFBL,
in constrast, requires relatively limited knowledge about the structure of a
nonlinear system to exactly linearize it.

Figure 1: Diagram depicting the overall flow of the proposed control scheme.

Beyond briefly presenting the method of FBL and extending it to LFBL, this
paper is aimed at applying the method for the task of controlling 4-wheeled
motor vehicles. Specifically, we want to employ this method to exploit the
benefits of linear control in the development of algorithms for racing cars.
The nonlinear and nonholonomic nature of a car’s dynamics, paired with their
prevalence in our daily lives, makes the study of their efficient and
effective control a tantalizing pursuit. A great deal of work has been done to
accommodate the idiosyncrasies of path planning and control for vehicles, and
autonomous racing has become a point of interest for profit, sport, and hobby
alike [24, 25, 26, 27]. There is a mature body of literature on the design of
Model Predictive Control (MPC) algorithms that seeks to create optimal paths
based on an explicitly stated sets of dynamics and system constraints. There
has also been some recent work towards learning various elements of MPC [28,
29, 30], With this background in mind, seek to use LFBL to learn the dynamics
of a car. The resulting linear input-output relationship can be used to vastly
simplify its planning and control. We demonstrate this by the construction of
a simple path planner that uses a Linear Quadratic Regulator (LQR) structure
paired with linear constraints to enforce the dynamics of the vehicle. In
addition, we posit a novel extension to the method of LFBL; a preprocessing
network that is designed to convert the values calculated by an exactly
linearizing controller into an appropriate set of inputs for the system we
wish to control. This conversion then allows for the application of a feedback
linearizing controller to systems that do not share the same inputs as the
linearizing controller.

The remainder of this paper is structured as follows: Section II: Theory, will
briefly discuss the requisite theory for the method of FBL and how we
formulate the problem of LFBL. Section III, Design Considerations, will
address the specifics of the vehicle dynamics needed for this work as well as
a variety of modifications and extensions made to the core methodology of LFBL
to realize the desired controller. Section IV: Experementation, Results, and
Discussion, will briefly address what experiments have been run to debug and
validate the performance of our controller, and Section V: Conclusions and
Future Work, will briefly summarize the goals and accomplishments of this work
with an eye towards improvements that can be made moving forward.

##  II Theory

###  II-A Feedback Linearization

To begin, we consider a system in the control affine form:

$\displaystyle\dot{x}$ $\displaystyle=f(x)+g(x)u$ (1) $\displaystyle y$
$\displaystyle=h(x),$

The method presented here can address the ”square” case of this form where the
number of inputs matches the number of outputs. $x\in{\mathbb{R}^{n}}$ is the
state, $u\in{\mathbb{R}^{q}}$ is the input, and $y\in{\mathbb{R}^{q}}$ is the
output with the mappings $f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$,
$g:\mathbb{R}^{n}\rightarrow\mathbb{R}^{nxq}$, and
$h:\mathbb{R}^{n}\rightarrow\mathbb{R}^{q}$ are assumed smooth.

For the purposes of intuition, we will primarily address the single-input-
single-output (SISO; q=1) case with the knowledge that the Multi-input Multi-
output (MIMO) square case abides the same logic, albeit more mechanically
obtuse. In the SISO case, we begin by taking time derivatives of $y=h(x)$
until the input $u$ appears and invert this relationship so-as to enforce the
desired linear input-output behavior. Consider the first derivative of the
output posed in (1):

$\displaystyle\dot{y}$
$\displaystyle=\frac{dh}{dx}(x)\cdot\Big{(}f(x)+g(x)u\Big{)}$
$\displaystyle=\frac{dh}{dx}(x)\cdot f(x)+\frac{dh}{dx}(x)\cdot g(x)u$

where $L_{f}h(x)=\frac{dh}{dx}(x)\cdot f(x)$ and
$L_{g}h(x)=\frac{dh}{dx}(x)\cdot g(x)$ are called Lie derivatives and serve to
quantify the rate of change of $y=h(x)$ along vector fields defined by $f$,
and $g$ respectively. In the case where $L_{g}h(x)\neq 0$, $u$ appears in the
derivative and we can exactly control the outputs $y=h(x)$ with a controller
of the form:

$\begin{array}[]{lcl}u(x,v)=\frac{1}{L_{g}h(x)}(-L_{f}h(x)+v)\\\ \end{array}$
(2)

where $v\in R^{q}$ is the set of virtual inputs. This transformation between
$u$ and $v$ serves to ”cancel out” the nonlinearities of the original system
and enforce the relationship $\dot{y}=v$. If, however, $L_{g}h(x)\equiv 0$,
then the input can’t directly affect the first derivative as desired and the
control law (LABEL:eq:controller) is not well defined. In this case, we can
continue taking derivatives of $y(x)$ until the input appears. When the input
appears in the $\gamma^{th}$ derivative, the derivative can be expressed as:

$\begin{array}[]{lcl}y^{(\gamma)}=L^{\gamma}_{f}h(x)+L_{g}L_{f}^{\gamma-1}h(x)u\\\
\end{array}$ (3)

where $L^{\gamma}_{f}h(x)$ and $L_{g}L_{f}^{\gamma-1}h(x)$ are higher order
Lie Derivatives. As before, if $L_{g}L_{f}^{\gamma-1}h(x)\neq 0$ then the
control law can be rewritten as:

$\begin{array}[]{lcl}u(x,v)=\frac{1}{L_{g}L_{f}^{\gamma-1}h(x)}(-L_{f}^{\gamma}h(x)+v)\\\
\end{array}$ (4)

As above, this choice of control law enforces the relationship $y^{\gamma}=v$,
where gamma is called the ”relative degree” of the nonlinear system.

This process can be extended readily to square, MIMO systems [$q>1$]. By
successively taking the derivatives of each output variable until at least one
of the inputs appears, we can obtain an input-output relationship of the form:

$\begin{array}[]{lcl}[y_{1}^{\gamma_{1}},...,y_{p}^{\gamma_{q}}]^{T}=b(x)+A(x)u\\\
\end{array}$ (5)

where the square matrix $A(x)\in\mathbb{R}^{q\times q}$ is called the
”decoupling matrix” and $b(x)\in\mathbb{R}^{q}$ is called the ”drift term”. If
$A(x)$ is invertable, then the control law can again be written as:

$\begin{array}[]{lcl}u(x,v)=A^{-1}(x)(-b(x)+v)\end{array}$ (6)

with virtual inputs $v\in\mathbb{R}^{q}$ corresponding to the decoupled linear
system in (6):

$\begin{array}[]{lcl}[y_{1}^{\gamma_{1}},...,y_{q}^{\gamma_{q}}]^{T}=[v_{1},....,v_{q}]^{T}\end{array}$
(7)

where the $j^{th}$ entry of $v$ is denoted $v_{j}$, $y_{j}^{\gamma_{j}}$ is
the $\gamma^{th}$ derivative of the $j^{th}$ output, and
$\gamma=(\gamma_{1},...,\gamma_{q})$ is called the ”relative degree” of the
system. The system (7) is Linear Time invariant (LTI) and can be represented
in the ”normal form”: with

$\begin{array}[]{lcl}\dot{\xi_{r}}=A^{\prime}\xi_{r}+B^{\prime}v_{r}\end{array}$
(8)

With state variables
$\xi_{r}=(y_{1},\dot{y_{1}},...,y_{1}^{\gamma_{1}-1},...,y_{q},...,y_{q}^{\gamma_{q}-1})$
and $A^{\prime}\in\mathbb{R}^{\gamma\times\gamma}$ and
$B^{\prime}\in\mathbb{R}^{\gamma\times q}$. This simplified normal form can be
used to more easily create a desired trajectory for the output of the
nonlinear system using the techniques of linear control theory. A key insight
from this analysis is the manner in which the transformed LTI system is made
to behave. Specifically, the normal form is formulated in such a way that the
$\gamma_{j}^{th}$ derivative of the $j^{th}$ output is the value being
directly controlled while the actual output, $y_{j}$ is related to this value
by a chain of integrators, e.g. we directly control the acceleration of a
vehicle to achieve the desired position values.

###  II-B Learning Feedback Linearization

The primary shortcoming of feedback linearization as a methodology is it’s
inherent dependence on exact knowledge of the dynamics governing the plant,
or, the system that is being controlled. This naturally leads to two sets of
dynamics that must be considered in practice, the plant dynamics that
represent the true behavior of the system, and the model dynamics designed by
the engineer attempting to control the system; consider:

$\displaystyle\dot{x}_{p}$ $\displaystyle=f_{p}(x_{p})+g_{p}(x)u_{p}$ (9)
$\displaystyle y_{p}$ $\displaystyle=h_{p}(x),$

$\displaystyle\dot{x}_{m}$ $\displaystyle=f_{m}(x_{m})+g_{m}(x)u_{m}$ (10)
$\displaystyle y_{m}$ $\displaystyle=h_{m}(x),$

Where a subscript $p$ denotes that we are referring to the plant and a
subscript $m$ indicates that we are referring to the model. With the technique
of Learning Feedback Linearization, we aim to rectify the aforementioned model
mismatch by learning how to appropriately deviate the nonlinear input $u_{p}$
to better match the output of the plant to what we would expect based on the
model. In this way, we are not only learning an exactly linearizing
controller, but also forcing the plant to behave according to the dyanamics
designated by our model. For learning feedback linearization to be applicable,
we first require that the system(s) (9), (10) have the same, well-defined
relative degree:

$\displaystyle(\gamma_{1}^{p},...,\gamma_{q}^{p})=(\gamma_{1}^{m},...,\gamma_{q}^{m})$
(11)

With this assumption, we can guarantee the existence of feedback control laws
of the form:

$\begin{array}[]{lcl}u_{p}(x,v)=\beta_{p}(x)+\alpha_{p}(x)v\end{array}$ (12)
$\begin{array}[]{lcl}u_{m}(x,v)=\beta_{m}(x)+\alpha_{m}(x)v\end{array}$ (13)

With $\beta_{p}(x),\beta_{m}(x)\in\mathbb{R}^{q}$, and
$\alpha_{p}(x),\alpha_{m}(x)\in\mathbb{R}^{q\times q}$. $u_{p}$ is, as of yet,
unknown, but owing to the structure and assumptions developed here, we can
relate the dynamics of the plant and model as follows:

$\begin{array}[]{lcl}\beta_{p}(x)=\beta_{m}(x)+\Delta\beta(x)\end{array}$ (14)
$\begin{array}[]{lcl}\alpha_{p}(x)=\alpha_{m}(x)+\Delta\alpha(x)\end{array}$
(15)

Here, $\Delta\beta(x)$ and $\Delta\alpha(x)$ are two continuous functions with
parameterized estimates:

$\begin{array}[]{lcl}\Delta\beta(x)\approx\beta_{\theta_{1}}(x),\Delta\alpha(x)\approx\alpha_{\theta_{2}}(x)\end{array}$
(16)

Where $\theta_{1},\theta_{2}$ represent the parameters to be learned through
experimentation of the reinforcement learning agent. It is through learning
the approximations (16) that the model mismatch is rectified and exact
feedback linearization of the dynamic system in question can be achieved with
the control law:

$\begin{array}[]{lcl}\hat{u}_{\theta}(x,v)=[\beta_{m}(x)+\beta_{\theta_{1}}]+[\alpha_{m}(x)+\alpha_{\theta_{2}}]v\end{array}$
(17)

###  II-C Formulating the Learning Problem

####  II-C1 continuous time

We know from section II-A that the dynamics relating the inputs to the outputs
under control law (17) are of the form:

$\displaystyle{}y^{(\gamma)}=\underbrace{b_{p}(x)+A_{p}(x)\hat{u}_{\theta}(x.v)}_{W_{\theta}(x,v)}$
(18)

Where $b_{p}(x)$ and $A_{p}(x)$ are unknowns. We seek to find the parameters
$\theta$ such that $W_{\theta^{*}}(x,v)\approx v$, leading to the choice of a
point-wise loss
$\ell:\mathbb{R}^{n}\times\mathbb{R}^{q}\times\mathbb{R}^{K_{1}+K_{2}}\rightarrow\mathbb{R}$:

$\displaystyle{}\ell(x,v,\theta)=||v-W_{\theta^{*}}(x,v)||_{2}^{2}$ (19)

This relatively simple 2-norm loss measures the learned controller’s
performance in linearizing the plant dynamics as a function of the control
$\hat{u}_{\theta}$ and the state $x$ when virtual input $v$ is applied to the
linear reference model whose behavior we seek to match. Next we can define a
probablity distribution $X$ over $\mathbb{R}^{n}$ with $V$, the uniform
distribution over the set $\\{v\in\mathbb{R}^{q}:||v||\leq 1\\}$ and define
the weighted loss:

$L(\theta)=\mathbb{E}_{x\sim X,v\sim V}[l(x,v,\theta)]$ (20)

With this, we choose to formulate our learned controller to optimize over
parameters, $\theta$ to solve the problem:

$\min_{\theta}L(\theta)$ (21)

The primary concern in formulating the learning problem like this is the
unknown nature of the terms that constitute $W_{\theta^{*}}(x,v)$, however,
this can be circumvented by measuring the outputs of the plant, $y^{(\gamma)}$
as there is an equivalence between the two as defined in (18). Further detail
on the continuous time formulation of this problem, including the conditions
under which global minima can be found, can be found in [9].

####  II-C2 discrete time

While the theory above is developed generally in the continuous-time case, it
is of practical use for many systems, including the work presented here, to
formulate the learning problem in discrete-time. To do so, we first define
$\Delta t>0$ to denote the sampling rate of our system and $t_{k}=k\Delta t$
for each $k\in\mathbb{N}$. Then, defining $x(\cdot)$ as the state trajectory
of the plant with $x_{k}=x(t_{k})$ and let $u_{k}$ denote the control applied
on interval $[t_{k},t_{k+1})$. With these, we can setup the difference
equation for the plant dynamics:

$x_{k+1}=x_{k}+\underbrace{\int_{t_{k}}^{t_{k+1}}{f_{p}(x(t))+g_{p}(x(t))u_{k}d\tau}}_{F(x_{k},u_{k})}$
(22)

Now, letting
$\xi(\cdot)=(y_{1}(\cdot),...,y_{1}^{(\gamma_{1}-1)}(\cdot),...,y_{q}(\cdot),...,y_{q}^{(\gamma_{q}-1)}(\cdot))$
represent the linearized system state and defining $\xi_{k}=\xi(t_{k})$, we
can integrate (18) over $[t_{k},t_{k+1})$ and obtain a new difference equation
in terms of the linearized system states:

$\xi_{k+1}=e^{A\Delta t}\xi_{k}+H(x_{k},u_{k})$ (23)

here, $H(x_{k},u_{k})$ is, in general, no longer affine in $u_{k}$, presenting
the main issue in applying the previously developed continuous-time theory
here. Nonetheless, we can still try to enforce an approximately linear
relationship between a virtual input $v_{k}$ and the subsequent outputs. for
ease of Notation, define $\bar{A}=e^{A\Delta t}$, and $\bar{B}=\int^{\Delta
t}_{0}e^{At}Bdt$, then the normal form (8) can be rewritten as:

$\xi_{k+1}=\bar{A}\xi_{k}+\bar{B}v_{k}$ (24)

This equation defines the linear relationship we would like to learn and
enforce by our choice of control $u_{k}=\hat{u}_{\theta}(x_{k},v_{k})$. With
this being the case, we define our discrete time point-wise loss:

$\bar{\ell}(x_{k},u_{k},v_{k})=||\bar{B}v_{k}-H(x_{k},u_{k})||_{2}^{2}$ (25)

As above in the continuous case, this value can be readily calculated through
observation of the system’s outputs and numerically differentiating their
values between time-steps, this loss metric is designed to measure how well
the control $u_{k}$ truly corrects for the unknown components and
nonlinearities of the plant. We can then define the discrete time
reinforcement learning problem over the parameters of $\hat{u}_{\theta}$ as:

$\begin{array}[]{rrclcl}\displaystyle\min_{\theta}&\lx@intercol\mathbb{E}_{x_{0}\sim
X,v_{k}\sim
V,w_{k}\sim\mathcal{N}(0,\sigma_{w}^{2})}[\sum_{k=1}^{N}\bar{\ell}(x_{k},u_{k},v_{k})]\hfil\lx@intercol\\\
\textrm{s.t.}&x_{k+1}=x_{k}+F(x_{k},u_{k})\\\ &x_{0}=x_{0}\\\
&u_{k}=\hat{u}_{\theta}(x_{k},v_{k})+w_{k}\end{array}$ (26)

Where $N$ is the length of the training episodes, and $k$ represents the
current time step. In this formulation, initial conditions are sampled using
the desired state distribution $X$, and inputs are sampled from $V$ at each
time step. the additional noise term, $w_{k}$ serves to encourage exploration.
With this formulation, local optima can be found using a variety of
reinforcement learning algorithms [31, 32, 33, 34] in tandem with with
hardware experiments to evaluate the loss function.

##  III Design Considerations

###  III-A Feedback Linearization of Vehicle Dynamics

For the purposes of this work, we address only the lateral dynamics of a
vehicle, or, the dynamics from a ”top down” view. The intricacies of a
vehicle’s suspension, the possibility of tire slip, and the dynamics of the
car’s drive train will ideally be learned by the preprocessing network posited
in section III-C. With this in mind, we consider a dynamically extended
bicycle model of the vehicle dynamics:

$\begin{array}[]{lcl}\dot{x}=Vcos(\psi+\beta)\\\ \dot{y}=Vsin(\psi+\beta)\\\
\dot{\psi}=\frac{V}{l_{r}}sin(\beta)\\\ \dot{V}=a\\\ \dot{\beta}=b\end{array}$
(27)

Figure 2: Visual representation of kinematic bicycle model

Where x and y are the coordinates of the vehicle’s center of mass in an
inertial frame $(X,Y)$, $\psi$ is the inertial heading of the car, $V$ is the
velocity of the car, $l_{r}$ is the distance from the center of mass to the
vehicle’s rear axle, and $\beta$ is the angle of the velocity of the center of
mass with respect to the longitudinal axis of the car. Here, we choose $x$ and
$y$ as the outputs of the system that we seek to control and further choose
$a$ and $b$ as the inputs through which we will do so. A visualization of this
system is provided below in figure 2. A first-principles analysis of this
system reveals further:

$\beta=tan^{-1}(\frac{l_{r}}{l_{f}+l_{r}}(tan(\delta_{f}))$ (28)

where $\delta_{f}$ is the angle of steering of the front tires and $l_{f}$ is
the distance from the center of mass to the vehicle’s front axle. While
$\delta_{f}$ is more intuitively meaningful as a choice of input, the
complicated nature of its relationship to the vehicle’s dynamics make it
preferable for our formulation to control $\dot{\beta}$ directly and back-
calculate $\delta_{f}$ uniquely in the case of a steering angle limited to at
most $\pm 90$ degrees from a neutral orientation. This easily covers the range
of any standard vehicle’s max steering angle.

With this form of the dynamics, we can start deriving the system’s linearized
normal form. We first take derivatives of the outputs with respect to time
until the inputs appear:

$\begin{array}[]{lcl}x=x\\\ \dot{x}=Vcos(\psi+\beta)\\\
\ddot{x}=\dot{V}cos(\psi+\beta)-\dot{\beta}Vsin(\psi+\beta)-\frac{V}{l_{r}}sin(\psi+\beta)\\\
\ddot{x}=cos(\psi+\beta)a-Vsin(\psi+\beta)b-\frac{V}{l_{r}}sin(\psi+\beta)\end{array}$
(29)

similarly, the derivatives of y can be found:

$\begin{array}[]{lcl}y=y\\\ \dot{y}=Vsin(\psi+\beta)\\\
\ddot{y}=\dot{V}sin(\psi+\beta)+\dot{\beta}Vcos(\psi+\beta)+\frac{V}{l_{r}}sin(\psi+\beta)\\\
\ddot{y}=sin(\psi+\beta)a+Vcos(\psi+\beta)b+\frac{V}{l_{r}}sin(\psi+\beta)\end{array}$
(30)

With this, our vector relative degree is:

$\left[\begin{array}[]{ccc}\gamma_{1}\\\
\gamma_{2}\end{array}\right]=\left[\begin{array}[]{ccc}2\\\
2\end{array}\right]$ (31)

By applying the transformation $x=\xi_{1}^{1}$, and $y=\xi_{2}^{1}$, the
normal form can be written as:

$\begin{array}[]{lcl}\dot{\xi}_{1}^{1}=\xi_{2}^{1}\\\
\dot{\xi}_{2}^{1}=\xi_{3}^{1}\\\
\dot{\xi}_{3}^{1}=cos(\psi+\beta)a-Vsin(\psi+\beta)b-\frac{V}{l_{r}}sin(\psi+\beta)\\\
\dot{\xi}_{1}^{2}=\xi_{2}^{2}\\\ \dot{\xi}_{2}^{2}=\xi_{3}^{2}\\\
\dot{\xi}_{3}^{2}=sin(\psi+\beta)a+Vcos(\psi+\beta)b+\frac{V}{l_{r}}sin(\psi+\beta)\\\
\end{array}$ (32)

with corresponding decoupling matrix:

$A=\left[\begin{array}[]{ccc}cos(\psi+\beta)&-Vsin(\psi+\beta)\\\
sin(\psi+\beta)&Vcos(\psi+\beta)\end{array}\right]$ (33)

and associated drift term:

$b=\left[\begin{array}[]{ccc}-\frac{V}{l_{r}}sin(\psi+\beta)\\\
\frac{V}{l_{r}}sin(\psi+\beta)\end{array}\right]$ (34)

This yields the equations for our linearizing inputs:

$\begin{array}[]{ccc}u=-A^{-1}b+-A^{-1}v\\\ \end{array}$ (35)

where $v$ is the virtual input to the desired linear system and $u$ is the
corrected input that accounts for the nonlinearities of the system. Finally,
we arrive at the linearized dynamics relating the virtual inputs to the
outputs:

$\begin{array}[]{ccc}\ddot{x}=v_{1}\\\ \ddot{y}=v_{2}\end{array}$ (36)

It is clear from inspection that the appropriate choice of $v_{1}$ and $v_{2}$
allows us to directly control the desired output, $x$ and $y$. The resulting
linear system as expressed in (8) is

$A^{\prime}=\left[\begin{array}[]{cccccc}0&1&0&0\\\ 0&0&0&0\\\ 0&0&0&1\\\
0&0&0&0\\\ \end{array}\right],B^{\prime}=\left[\begin{array}[]{ccc}0&0\\\
1&0\\\ 0&0\\\ 0&1\\\ \end{array}\right]$ (37)
$C^{\prime}=\left[\begin{array}[]{cccccc}1&0&0&0\\\ 0&0&1&0\\\
\end{array}\right]$ (38)

###  III-B Path Planning and Tracking

The transformation of the previously nonlinear vehicle dynamics into a linear
system of the form (8) with dynamics governed by (37), (38) enables the use of
a wide range of possible linear control and path planning techniques that
present greater opportunity for guarantees in performance and safety. Central
to the path tracking approach used in this work is the continuous-time LQR:

$\begin{array}[]{rrclcl}\displaystyle\min_{u}&\lx@intercol\int_{0}^{\infty}(x^{T}Qx+v^{T}Rv)\hfil\lx@intercol\\\
\end{array}$ (39)

Where $x\in\mathbb{R}^{\gamma}$ is the system state, $v\in\mathbb{R}^{q}$ is
the input and $Q\in\mathbb{R}^{\gamma\times\gamma}$ and
$R\in\mathbb{R}^{q\times q}$ are weight matrices determined by designer. The
solution of this optimization problem can be readily found by solving the
continuous-time algebraic ricatti equation:

$\begin{array}[]{rrclcl}A^{T}P+PA-PBR^{-1}B^{T}P+Q=0\end{array}$ (40)

Solving this equation for a unique, positive definite solution, P, provides
sufficient information to construct an optimal feedback control law:

$\begin{array}[]{rrclcl}u^{*}=Fx=R^{-1}B^{T}Px\end{array}$ (41)

The ease of solution of equation (40), paired with the simple control law (41)
and guarantees of optimality in control make this ideal as a feedback control
strategy to regulate the car to the commands of a higher-level path-planner,
which is itself a discrete form of the LQR Controller:

$\displaystyle\min_{u}$
$\displaystyle\sum_{k=1}^{N-1}(x_{k}^{T}Qx_{k}+v_{k}^{T}Rv_{k})+x_{N}^{T}Q_{f}x_{N}$
(42) s.t. $\displaystyle x_{\tau+1}=A^{\prime}x_{\tau}+B^{\prime}v_{\tau}$
$\displaystyle x(T)=x_{f}$ $\displaystyle x(0)=x_{0}$
$\displaystyle|v(k)|_{\infty}\leq v_{bnds}$

Here, $N$ is the number of execution steps in the desired look-ahead time,
$T$. $\tau$ represents the current time-step, $x_{f}$ and $x_{0}$ represent
the desired final and initial locations of the vehicles linear state
respectively, $x_{\tau}$ and $v_{\tau}$ represent the linear system state and
input, respectively, at time-step $\tau$. Finally,
$Q\in\mathbb{R}^{\gamma\times\gamma}$,
$Q_{f}\in\mathbb{R}^{\gamma\times\gamma}$, and $R\in\mathbb{R}^{q\times q}$
are weight matrices determined by the designer. Further,

$\displaystyle v=[v_{1},..,v_{N-1}]$ (43) $\displaystyle x=[x_{1},...,x_{N}]$

This configuration is ideal for multiple reasons. While the unconstrained
version of (42) can be solved readily using a ricatti equation similar to
(40), we choose to formulate this system with the additional constraints of a
starting and ending location, Other constraints are added to enforce
mechanical limitations like only having the ability to accelerate so quickly.
This effectively forces the planner to account for the boundary conditions of
the car’s starting location, mechanical limitations, and where it should be at
the end of its look-ahead period. The addition of constraints to enforce
waypoint tracking between the start and finish locations can be used to
further constrain the planner to a desired path. In the exactly linearizing
case, both the state path and inputs that we calculate in this approach can be
used limiting the need for the continuous time LQR solution (41), However, in
the approximate controller case, we will need a more active form of feedback
control to correct for the unknown true dynamics of the system. It is
convenient to note that in both cases, (42) a linearly constrained quadratic
programming problem that is easily solvable by most off-the-shelf solvers
quickly and thus is ideal for a waypoint following-based path planning
approach. Specifically, this formulation can be iteratively solved as the
system evolves to update the planned path as the vehicle successively
approaches and passes new waypoints, akin to an MPC approach.

###  III-C Preprocessing Network

The gym environment ‘CarRacing-v0‘ environment takes in tuples of actions in
the form $\\{s,g,b\\}$, corresponding to the steering, gas, and brake applied
respectively. In an end-to-end reinforcement learning task, there exists no
need to investigate the detailed effects and properties of individual actions,
as the learning algorithm would implicitly learn some representations of the
actions, such as a Q functions, through the training process [35].

In this task, however, the gym environment only offers the vehicle dynamics
that the LFBL controller is applied to. Since the gym environments original
reward function is unused, the learning algorithm cannot capture the action
representations as stated above. Additionally, according to the bicycle model,
the LFBL seeks to control the steering angle and the acceleration directly on
the vehicle, but the car racing dynamics require gas and brake parameters
instead. The mismatch in control inputs makes it necessary to train a
preprocessing network with supervised learning.

Since relationship between linear accelearation and the control inputs is not
too complicated (see Figure 1), a shallow feed-forward neural network can
approximate the function well. The input layer has size $(1\times 200)$
followed by a leakyReLU activation, and the output has size $(200\times 1)$
with identity output activation. The Adam optimizer is used with a learning
rate of 0.05 over the MSE loss.

###  III-D Model-free Deep RL Training

We utilize deep reinforcement learning to train our LFBL controller since
neural networks are capable of learning and representing the complex
relationship in nonlinear control systems. Since the accurate true dynamics of
our system is unknown, we further restrict our attention to Soft Actor-Critic
(SAC), a state-of-the-art model-free deep RL algorithm. SAC is an off-policy
algorithm that optimizes over a stochastic policy with entropy regularization,
which augments its objective to be

$J(\pi)=\sum_{t=0}^{T}\mathbb{E}_{(s_{t},a_{t})\sim\rho_{\pi}}[r(s_{t},a_{t})+\alpha
H(\pi(\cdot|s_{t}))]$ (44)

where $H(\pi(\cdot|s_{t}))$ is the entropy term that measures the randomness
of the policy under $s_{t}$ and $\alpha$ is a temperature hyper-parameter that
determines the relative importance of the entropy [36]. Detailed
implementations and novelty designs [37] of SAC will not be mentioned in this
work. The main reasons that we use SAC in training our linearizing controller
are (1) it attains good balance between performance and sample efficiencey and
(2) it is in-built in the OpenAI Spinning Up library.

###  III-E Overall Control Architecture

To briefly summarize the overall control architecture posited in this paper,
we direct the reader to figure 1 which is a visual representation of how the
path and corresponding control signals are created and fed to the environment.
In general, the entire system is guided by a path planner that calculates an
optimal path from a desired starting point to a desired end-point, as-in (42).
Then the LFBL controller can be used to calculate an input to the nonlinear
dynamics of the car that will behave as-in the desired linear relationship,
(7). The output of this controller, acceleration, is then converted into gas
and brake signals that are fed directly to the racing environment in tandem
with the other output of the linearizing controller, steering angle. The
effect of these inputs is observed and used to further train the LFBL
controller. Note that, in the case of the exact learned controller desribed
below, the preprocessing network is not needed, as the dynamics being learned
is strictly of the form of the LFBL controller.

##  IV Experimentation, Results, and Discussion

###  IV-A Exact Learned Controller

We first sought to learn a controller that starts with a linearizing
controller based on a set of nominal dynamics that can be exactly linearized
relative to the true dynamics being learned. The setup for running experiments
to this end involves the instantiation of two sets of dynamic equations of the
form (27). The key difference is the differing choice of $l_{r}$. This
inherently affects how the inertial heading evolves and has downstream
consequences for how the outputs we seek to control evolve. In particular,
this mostly effects the drift term. Thus the problem is for the agent to learn
a corrective drift term that accurately accounts for the difference in $l_{r}$
and causes the true dynamics to behave exactly as the nominal dynamics would
dictate. For the tests performed, the nominal dynamics were instantiated with
an $l_{r}$ value of .5, idicating a distance between the front wheels and the
center of mass of .5 meters.

With this setup we began training. The LQR planner discussed in section III-B
was configured to plan a path for the car from a starting location of
$[x,y]=[0,0]$ at $t=0$ to a final location of $[x,y]=[5,5]$ at $t=5$s with a
time step of .02s, resulting in episodes of 249 steps of the environment.
because the car begins aligned with the y axis, this involves turning, which
will in turn require that the agent learn to correct for turning as well as
acceleration. The step reward is calculated as the 2-norm distance of the
vehicle’s linear state from the desired state calculated by the planner at
that time. the sum of these values over an episode is therefore a good
indication of the ability of the controller to follow the desired path and is
consequently a good proxy for the quality of the learned controller.

Figure 3: Plot comparing the path we want the agent to follow, the agent’s
true path and the path of the nominal linearizing controller with no learning

To begin, we removed the ability of the agent to actively effect the
controller in order to set a baseline that reflects the performance of the
controller as it was nominally designed with $lr=.5$. We found that in an
episode of 249 steps the total loss for an episode was constant, as there was
no way for the agent to interact with, and correct, the controller. The
baseline average episode return value we observed was $-1.4\times 10^{3}$. We
next tried to run the algorithm using the default parameters of the SAC
implementation being used and received a max average episode return of -350 at
epoch 56 of 200. This is slightly less than a four-fold reduction, indicating
significant improvements in the performance of the learned controller, a plot
for the learning curve of this agent, as well as a visualization of the path
it followed vs the path it was intended to follow are in figures 3 and 4,
respectively. Once the performance of a naively tuned agent was found, we
began a hyperparameter sweep to determine the best performing set of
parameters. We opted to run these sweeps over a single variable, keeping the
rest of the hyperparameters as the default setting used in the previous
experiment. Together, we ran tests varying learning rate, the batch size of
training updates, the discount factor, $\gamma$, as well as the polyak
iterpolation factor, $\rho$, the result of most of these experiments was the
same; within roughly 10 eopchs of starting the experiment, the agent would
converge to an Average episode return fluctuating between -700 and -900,
although it would occassionally show chaotic behavior where it either
performed exceptionally well, as in the trial reported above, or blowing up
and returning NaN values after several epochs. This seems to indicate that the
performance of the agent is sensitive to the seed used in the training
process, which was selected at random for each trial. We further note that
these trials were run in roughly one hour per trial on both a personal laptop
and desktop.

From inspection of 4, a few things are clear. While the desired path is
relatively smooth, the path that is followed by the nominal controller with no
learning performs quite poorly, diverging from the initial path very quickly,
performing chaotic maneuvers and not arriving at the correct final location.
This is to be expected as the controller is designed not to match the model
that is actually being controlled. More interesting is the path followed by
the learned controller, reported above to have an average episode return of
-350. It is clear that this controller does not follow the path very closely,
makes erroneous maneuvers, and fails to reach the specified endpoint. It does,
however, reach roughly the correct y-coordinate, $y=5$ and manages to stay
much closer to the desired path than the the controller with no learning.
There might be a few reasons for this lackluster performance. As previously
mentioned, this trial appears to be the product of a good initial seed without
any special hyperparameter tuning. Thus, performing a second hyperparameter
sweep using this initial seed will likely improve performance significantly.

Figure 4: Plot comparing the path we want the agent to follow, the agent’s
true path and the path of the nominal linearizing controller with no learning

###  IV-B Approximate Learned Controller

The next case to consider is the approximate learned controller. here the true
dynamics are modeled by the car racing environment discussed in section III.
Beyond simply learning a corrective term for 27, this set of true dynamics
also models more complicated behaviors relating to the application of brakes,
steering, and gas as well as accounting for the delays in their effect on the
acceleration and steering that we seek to control. This is where the
preprocessing network is used to learn those more complicated relationships
and relate them to the acceleration and steering angle that we command from
our linearized controller. This involves not only a well-trained linearizing
controller, but a well trained preprocessing network. As such, we though it
best to decouple the problem of training the controller and preprocessing
networks before we could begin experiments to this end in-earnest. While an
initial version of the preprocessing network has been trained as per section
IV-C, the fact that we are still in the process of training an optimally
performant linearizing controller has limited our ability to test the
performance of the controller and pre-processing network combined.

###  IV-C Preprocessing Network Training

The aim of the preprocessing network is to take in the desirable linear
acceleration that the feedback linearizer outputs, and return the tuple
$\\{gas,brake\\}$ which can be directly sent as a control action to the gym
environment.

The training data set is collected by sampling random actions from the
‘CarRacing-v0‘ environment with the steering angle fixed to be zero. Once an
action vector is sampled, it is fixed and kept applied to the environment for
certain number of steps (usually 10). The acceleration is calculated by taking
the average of the differences in velocity between consecutive steps. Our
final data set consists of 5000 $\\{acceleration,gas,break\\}$ observations,
where the last two variables serve as the label in the supervised learning
task. A plot of the learning curve for the initial preprocessing network
implementation is shown in figure 5. While it appears to have convereged by
iteration 20 of training, we have not yet had the opportunity to test it in
tandem with the linearizing controller.

Figure 5: Plot depicting the learning curve of the preprocessing network.

##  V Conclusions and Future Work

While traditional methods of geometric control can drastically simplify the
control of complex nonlinear systems, they often require a great deal of
knowledge about the structure and parameters that govern the dynamics of the
system to be controlled. In stark contrast to this, model free methods of
reinforcement learning offer the ability to learn controllers for dynamic
systems with almost no prior knowledge of the system’s dynamics, but at the
cost of requiring large data sets and a lot of computer power. The method of
learning feedback linearization is devised by [9] to fuse these two sets of
techniques to effectively benefit from the advantages of both. A feedback
linearizing controller is designed using a nominal set of dynamics, and model
mismatch between the plant and controller is learned, and corrected for, in
the output of the linearizing controller. the result is a controller that
exactly linearizes the relationship between the inputs and outputs of the
system. This technique is implemented and used here for the purpose of
learning the dynamics of a 4-wheeled motor vehicle. The resulting linearized
dynamics are used to design a path planning and control algorithm based on a
Linear Quadratic Regulator, that outputs a sequence of states for the
linearized system to follow as well as the sequence of controls needed to
achieve those states. While this is sufficient for the exact linearization
case, the approximately learned controller will have extra elements to its
dynamics that will cause the vehicle to diverge from the optimal path and
requiring a more active feedback scheme through the use of the continuous time
Linear Quadratic Regulator that calculates appropriate control inputs at every
time step, instead of a-priori at the beginning of an episode. We further
posit an extension to the method of learning feedback linearization by the
introduction a preprocessing network; trained in a supervised manor with data
collected from the environment that we use to model the car dynamics in the
approximate learned controller case. The purpose of the preprocessing network
is to convert the exactly linearized outputs of our learned feedback
linearization controller into the inputs required by the environment.
Specifically we seek to convert the acceleration value output by our
linearizing controller into steering and brake signals that are put into the
’CarRacing-v0’. As of the writing of this report, the results of this work are
as follows: a custom gym environment has been created to test the exactly
linearizing case of model mismatch in two instances of the same set of
equations to test and verify the performance of the LFBL method in a clearly
defined case. Further, an initial implementation of the preprocessing network
has been trained using data from the target racing environment, but we have
been unable to begin experieents on the complete control scheme because of
difficulties in the aforementioned exactly linearizing case.

There are multiple next steps to be performed in continuing this work beyond
the progress reported here. Namely, once a hyperparameter sweep is performed
with a good initial seed, we expect that a much better performing agent will
be learned, we will then begin experiments with the true dynamics in the
aforementioned car racing environment. beyond that, once we can demonstrate
some basic degree of performance in the car racing environment we can begin
tuning the path planning algorithm to optimise more closely for speed and
general performance. This would likely involve the iterative use of the path
planner to form an MPC-like approach.

## Acknowledgment

The Authors would like to thank Tyler Westenbroek and Valmik Prabhu for their
help in planning experiements and working with the learning feedback
linearization method.

## References

  * [1] Shankar Sastry  “Nonlinear systems: analysis, stability, and control”  Springer Science & Business Media, 1999
  * [2] Alberto Isidori  “Nonlinear control systems”  Springer Science & Business Media, 2013
  * [3] Dimitri P Bertsekas and John N Tsitsiklis  “Neuro-dynamic programming”  Athena Scientific, 1996
  * [4] Richard S Sutton and Andrew G Barto  “Reinforcement learning: An introduction”, 2018
  * [5] Richard S Sutton, David A McAllester, Satinder P Singh and Yishay Mansour  “Policy gradient methods for reinforcement learning with function approximation”  In _Advances in neural information processing systems_ , 2000, pp. 1057–1063
  * [6] John Schulman et al.  “Trust region policy optimization”  In _International conference on machine learning_ , 2015, pp. 1889–1897
  * [7] Timothy P Lillicrap et al.  “Continuous control with deep reinforcement learning”  In _arXiv preprint arXiv:1509.02971_ , 2015
  * [8] John Schulman et al.  “Proximal policy optimization algorithms”  In _arXiv preprint arXiv:1707.06347_ , 2017
  * [9] Tyler Westenbroek et al.  “Feedback Linearization for Unknown Systems via Reinforcement Learning”  In _arXiv preprint arXiv:1910.13272_ , 2019
  * [10] Phillipe Martin, Richard M Murray and Pierre Rouchon  “Flat systems, equivalence and trajectory generation”  California Institute of Technology, 2003
  * [11] Rudolf Emil Kalman  “Contributions to the theory of optimal control”  In _Bol. soc. mat. mexicana_ 5.2, 1960, pp. 102–119
  * [12] Francesco Borrelli, Alberto Bemporad and Manfred Morari  “Predictive control for linear and hybrid systems”  Cambridge University Press, 2017
  * [13] Jessy W Grizzle, Gabriel Abba and Franck Plestan  “Asymptotically stable walking for biped robots: Analysis via systems with impulse effects”  In _IEEE Transactions on automatic control_ 46.1 IEEE, 2001, pp. 51–64
  * [14] Aaron D Ames, Kevin Galloway, Koushil Sreenath and Jessy W Grizzle  “Rapidly exponentially stabilizing control lyapunov functions and hybrid zero dynamics”  In _IEEE Transactions on Automatic Control_ 59.4 IEEE, 2014, pp. 876–891
  * [15] Daniel Mellinger and Vijay Kumar  “Minimum snap trajectory generation and control for quadrotors”  In _2011 IEEE International Conference on Robotics and Automation_ , 2011, pp. 2520–2525  IEEE
  * [16] Shankar Sastry and Marc Bodson  “Adaptive control: stability, convergence and robustness”  Courier Corporation, 1989
  * [17] John J Craig, Ping Hsu and S Shankar Sastry  “Adaptive control of mechanical manipulators”  In _The International Journal of Robotics Research_ 6.2 Sage Publications Sage UK: London, England, 1987, pp. 16–28
  * [18] Sosale Shankara Sastry and Alberto Isidori  “Adaptive control of linearizable systems”  In _IEEE Transactions on Automatic Control_ 34.11 IEEE, 1989, pp. 1123–1131
  * [19] Kwanghee Nam and ARISTOTLE Araposthathis  “A model reference adaptive control scheme for pure-feedback nonlinear systems”  In _IEEE Transactions on Automatic Control_ 33.9 IEEE, 1988, pp. 803–811
  * [20] Ioannis Kanellakopoulos, Petar V Kokotovic and A Stephen Morse  “Systematic design of adaptive controllers for feedback linearizable systems”  In _1991 American Control Conference_ , 1991, pp. 649–654  IEEE
  * [21] Jonas Umlauft, Thomas Beckers, Melanie Kimmel and Sandra Hirche  “Feedback linearization using Gaussian processes”  In _2017 IEEE 56th Annual Conference on Decision and Control (CDC)_ , 2017, pp. 5249–5255  IEEE
  * [22] Girish Chowdhary, Hassan A Kingravi, Jonathan P How and Patricio A Vela  “Bayesian nonparametric adaptive control using gaussian processes”  In _IEEE Transactions on Neural Networks and Learning Systems_ 26.3 IEEE, 2014, pp. 537–550
  * [23] Girish Chowdhary, Hassan A Kingravi, Jonathan P How and Patricio A Vela  “Bayesian nonparametric adaptive control of time-varying systems using Gaussian processes”  In _2013 American Control Conference_ , 2013, pp. 2655–2661  IEEE
  * [24] “Autonmous Racing Lueagues/Competitions”  URL: https://diyrobocars.com/autonomous-racing-leaguescompetitions/
  * [25] “AWS DeepRacer”, 2020  URL: https://aws.amazon.com/deepracer/
  * [26] “F1tenth”, 2020  URL: https://f1tenth.org/index.html
  * [27] “RoboRace”, 2020  URL: https://roborace.com/
  * [28] G. Williams et al.  “Aggressive driving with model predictive path integral control”  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , 2016, pp. 1433–1440  DOI: 10.1109/ICRA.2016.7487277
  * [29] U. Rosolia and F. Borrelli  “Learning Model Predictive Control for Iterative Tasks. A Data-Driven Control Framework”  In _IEEE Transactions on Automatic Control_ 63.7, 2018, pp. 1883–1896  DOI: 10.1109/TAC.2017.2753460
  * [30] Melanie N. Lukas Hewing  “Cautious Model Predictive Control using Gaussian Process Regression”  In _arXiv preprint arXiv:1705.10702_ , 2019
  * [31] Richard S. Sutton and Andrew G. Barto  “Introduction to Reinforcement Learning”  Cambridge, MA, USA: MIT Press, 1998
  * [32] John Schulman et al.  “Trust region policy optimization”  In _International Conference on Machine Learning_ , 2015, pp. 1889–1897
  * [33] John Schulman et al.  “Proximal Policy Optimization Algorithms”  In _CoRR_
  * [34] David Silver et al.  “Deterministic Policy Gradient Algorithms”  In _Proceedings of the 31st International Conference on Machine Learning_ , Proceedings of Machine Learning Research, 2014, pp. 387–395
  * [35] Volodymyr Mnih et al.  “Playing Atari with Deep Reinforcement Learning”, 2013  arXiv:1312.5602 [cs.LG]
  * [36] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel and Sergey Levine  “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”, 2018  arXiv:1801.01290 [cs.LG]
  * [37] Josh Achiam  “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”, 2018  URL: https://spinningup.openai.com/en/latest/algorithms/sac.html