# Closed-Loop Error Learning Control for Uncertain Nonlinear Systems With
Experimental Validation on a Mobile Robot

Erkan Kayacan The information, data, or work presented herein was funded in
partby the Advanced Research Projects Agency-Energy (ARPA-E), U.S.Department
of Energy, under Award Number DE-AR0000598.The author is with the School of
Mechanical $\&$ Mining Engineering, the University of Queensland, Brisbane.
QLD 4072 Australia. e-mail<EMAIL_ADDRESS>
###### Abstract

This paper develops a Closed-Loop Error Learning Control (CLELC) algorithm for
feedback linearizable systems with experimental validation on a mobile robot.
Traditional feedback and feedforward controllers are designed based on the
nominal model by using Feedback Linearization Control (FLC) method. Then, an
intelligent controller is designed based on sliding mode learning algorithm
that utilizes closed-loop error dynamics to learn the system behavior. The
controllers are working in parallel, and the intelligent controller can
gradually replace the feedback controller from the control of the system. In
addition to the stability of the sliding mode learning algorithm, the closed-
loop stability of an $n$th order feedback linearizable system is proven. The
simulation results demonstrate that CLELC algorithm can improve control
performance (e.g., smaller rise time, settling time and overshoot) in the
absence of uncertainties, and also provides robust control performance in the
presence of uncertainties as compared to traditional FLC method. To test the
efficiency and efficacy of CLELC algorithm, the trajectory tracking problem of
a tracked mobile robot is studied in real-time. The experimental results
demonstrate that CLELC algorithm ensures high-accurate trajectory tracking
performance than traditional FLC method.

###### Index Terms:

Feedback linearization, learning control, mobile robot, nonlinear system,
sliding mode learning algorithm, uncertain system.

##  I Introduction

Control of uncertain nonlinear systems is one of the most important topics in
modern control engineering [1, 2, 3, 4, 5, 6, 7]. Controllers intend to
achieve the best control performance in the presence of the worst
uncertainties in robust control approaches, and a high controller gain is the
general method to handle the effect of uncertainties in nonlinear control
theory [8, 9]. However, these methods result in substantial control action,
and very powerful actuators are thus demanded to perform unnecessarily
excessive control actions [10]. Furthermore, the robust control performance is
generally established by sacrificing the nominal control performance because
the nominal control performance is not taken into account [11, 12]. Therefore,
a control method is required to remain or improve nominal control performance
in the absence of uncertainties, and exhibit robust control performance in the
presence of uncertainties [13, 14, 15, 16, 17].

Autonomous guidance, navigation and control of mobile robots in unstructured,
off-road is one of the major problems in field robotics [18, 19, 20]. Mobile
robots in off-road terrain face to different surface materials and terrain
topographies; therefore, learning-based control techniques are necessary for
mobile robots to learn the unmodeled surface conditions (e.g., grass, sand,
and snow) and complex robot dynamics [21]. Modeling of these effects is so
arduous because it is not always possible to know the terrain type and wheel-
terrain interactions. Even if models exist, the identification of model
parameters is cumbersome. Therefore, researchers have focused on learning-
based controllers for mobile robots [22, 23, 24]. Online adaptive controller
based on a kinematic model with wheel slip angles was developed in [25], where
observers estimated the slip angles. Modeled and identified uncertainties
partially improved mobile robot path tracking performance; however, it was not
sufficient. To model and identify uncertainties fully, a framework consisting
of moving horizon estimation and model predictive control methods was proposed
for articulated unmanned ground vehicles [26, 27, 28] and mobile robots [29,
30]. In these works, an adaptive kinematic model was derived by adding
traction parameters for longitudinal and side slips into the traditional
kinematic model. Nonlinear moving horizon estimation method estimated these
traction parameters and nonlinear model predictive controller designed based
on the adaptive kinematic model with these traction parameters estimates
generated control signals. Since parameter estimation in nonlinear moving
horizon estimator is carried out in the arrival cost, the estimates are
assumed to be changing very slowly, and the most recent estimates do not have
any information on the measurements in the estimation horizon. Therefore, our
learning approach must (i) take the most recent measurements into account,
(ii) be robust against fast changing working conditions and (iii) enable the
representation of complex disturbance characteristics.

As distinct from previous studies on sliding mode learning algorithms [31],
the Closed-Loop Error Learning Control (CLELC) algorithm developed in this
paper does not require a high controller gain in feedback control action to
guarantee system stability so that it does not cause chattering effects on
control signals. Moreover, the CLELC algorithm demonstrates two remarkable
features. First, it does not cause any adverse effects on the system in the
absence of uncertainties, and it additionally improves control performance
when compared to nominal control performance obtained by Feedback
Linearization Control (FLC). Second, the proposed method exhibits robust
control performance for feedback linearizable systems with uncertainties.

The main contributions of this study are as follows: A novel control method
termed as a CLELC algorithm is developed and implemented in this paper. In
this novel control scheme, closed-loop error dynamics are utilized to learn
system behavior for adaptation. After a short duration, the intelligent
controller takes the overall control action while the output of the feedback
controller converges to zero. Moreover, the overall system stability for a
feedback linearizable system with uncertainties is proven based on Lyapunov
stability theory under the bounded error, reference and feedforward action
rates. The stability analysis shows that an $n$th order feedback linearizable
system controlled by the CLELC algorithm is stable. Along with the theoretical
results, this study demonstrates trajectory tracking-test results of the
developed CLELC algorithm on a tracked mobile robot.

The organization of the paper is as follows: The problem formulation is given
in Section II. The CLELC algorithm is formulated Section III. The intelligent
controller is designed and the basics of the sliding mode learning algorithm
are given in Sections III-A and III-B, respectively, while the closed-loop
system stability is proven in Section III-C. Simulation results for a third-
order nonlinear system and experimental validation on a mobile robot are given
in Sections IV and V, respectively. Finally, conclusions of the study are
summarized in Section VI.

##  II Problem Formulation

An $n$th order feedback linearizable system with uncertainties is written as
below:

$\displaystyle\dot{x}_{i}$ $\displaystyle=$ $\displaystyle x_{i+1}$
$\displaystyle\dot{x}_{n}$ $\displaystyle=$ $\displaystyle
a(\textbf{x})+b(\textbf{x})u+\Delta(\textbf{x},u)$ (1)

where $i=1,2,\dots,n-1)$, $\textbf{x}=[x_{1},x_{2},\ldots,x_{n}]\in R^{n}$ is
the state vector, $u\in R$ is the control input, $a(\textbf{x})$ and
$b(\textbf{x})$ are smooth nonlinear functions, and $\Delta(\textbf{x},u)$ is
the term for system uncertainty where $\Delta(\textbf{x},u)=\Delta
a(\textbf{x})+\Delta b(\textbf{x})u$.

Traditional feedback linearization control (FLC) law is formulated below:

$u=b^{-1}(\textbf{x})\Big{(}-a(\textbf{x})+u_{b}+u_{f}\Big{)}$ (2)

where $u_{b}$ and $u_{f}$ are the feedback and feedforward control inputs, and
formulated below:

$\displaystyle u_{b}$ $\displaystyle=$
$\displaystyle\textbf{k}\textbf{e}=k_{1}e_{1}+k_{2}e_{2}+\ldots+k_{n}e_{n}$
(3) $\displaystyle u_{f}$ $\displaystyle=$ $\displaystyle\dot{r}_{n}$ (4)

where $\textbf{k}=[k_{1},k_{2},\ldots,k_{n}]$ is the feedback controller gain
vector and positive, i.e., $k_{i}(i=1,2,\dots,n)>0$, and
$\textbf{e}=[e_{1},e_{2},\ldots,e_{n}]^{T}=[r_{1}-x_{1},r_{2}-x_{2},\ldots,r_{n}-x_{n}]^{T}$
is the error vector where $\textbf{r}=[r_{1},r_{2},\ldots,r_{n}]^{T}$ is the
reference vector and defined as
$[\dot{r}_{1},\dot{r}_{2},\ldots,\dot{r}_{n}]^{T}=[r_{2},r_{3},\ldots,r_{n+1}]^{T}$
so that the reference $r_{1}$ must be continuously $(n+1)$ times
differentiable.

If FLC law (2) is employed to control an nth-order feedback linearizable
system with uncertainties (II), the closed-loop error dynamics are calculated
below:

$\dot{e}_{n}+k_{n}{e}_{n}+k_{n-1}e_{n-1}+\ldots+k_{1}e_{1}=-\Delta(\textbf{x},u)$
(5)

Equation (5) shows that the error dynamics cannot be driven to zero under FLC
control law (2) in the presence of uncertainties.

##  III Closed-Loop Error Learning Control Algorithm

In this study, a CLELC algorithm is developed where the feedback, feedforward
and intelligent controllers are working in parallel as illustrated in Fig. 1.
The total control law for a CLELC algorithm is formulated as follows:

$u=b^{-1}(\textbf{x})\Big{(}-a(\textbf{x})+u_{b}+u_{f}+u_{n}\Big{)}$ (6)

where $u_{b}$ and $u_{f}$ are respectively the traditional feedback and
feedforward control inputs, and $u_{n}$ is the output of the intelligent
controller. The total control input is formulated as follows:

$u_{t}=u_{b}+u_{f}+u_{n}$ (7)

The details of the intelligent controller consisting of an adaptive neuro-
fuzzy structure and a sliding mode learning algorithm are given in the next
subsections.

Figure 1: Closed-loop error learning control scheme.

###  III-A Adaptive Neuro-Fuzzy Structure

The fuzzy _if-then_ rule of a zeroth-order Takagi-Sugeno-Kang model with $I$
input variables where the consequent part is a crisp number can be defined as
follows:

$R_{r}:\;\;\textrm{If}\;e_{1}\;\textrm{is}\;A_{1k}\;\ldots\;\textrm{and}\;e_{i}\;\textrm{is}\;A_{ik},\;\;\textrm{then}\;\;f_{r}=d_{r}$
(8)

where $e_{i}(i=1,\ldots,I)$ are the inputs of the Takagi-Sugeno-Kang model,
$A_{ik}$ is the $k$th fuzzy membership function $(k=1,\ldots,K)$ corresponding
to the input $i$th input variable. $K$ is the number of membership functions
for the $i$th input. The consequent part of the rules $f_{r}(r=1,\dots,N)$ is
the output function.

In adaptive neuro-fuzzy structure, the crisp inputs $e_{i}$ are mapped into
fuzzified values using membership functions [32]. The membership values for
the inputs $\mu_{ik}(e_{i})$ are defined by using Gaussian membership
functions as below:

$\mu_{ik}(e_{i})=\exp\Bigg(-\frac{1}{2}\Big{(}\frac{e_{i}-c_{ik}}{\sigma_{ik}}\Big{)}^{2}\Bigg{missing})$
(9)

where $c_{ik}$ and $\sigma_{ik}$ are respectively the mean and standard
deviation of the membership functions to be adjusted.

The firing strengths of the rules $w_{r}$ are calculated by using prod-t norm
operator as follows:

$w_{r}=\mu_{A1}(e_{1})\ast\mu_{A2}(e_{2})\ast\ldots\ast\mu_{AI}(e_{I})$ (10)

The prod-t norm operator is preferred due to the simplicity for the stability
analysis and the normalized firing strengths of the rules are computed as
below:

$\widetilde{w}_{r}=\frac{w_{r}}{\sum\limits_{r=1}^{N}{w}_{r}}$ (11)

The output signal of the neuro-fuzzy structure $u_{n}$ is computed as follows:

$u_{n}(t)=\sum\limits_{r=1}^{N}\widetilde{w}_{r}f_{r}$ (12)

The following vectors can be specified:
$\widetilde{\textbf{w}}=\left[\widetilde{w_{1}}\;\widetilde{w_{2}}\;...\;\widetilde{w_{N}}\right]^{T}$,
and $\textbf{f}=\left[f_{1}\;f_{2}\;...\;f_{N}\right]$.

###  III-B Sliding Mode Learning Algorithm

A novel sliding mode learning algorithm for the adaptations of the neuro-fuzzy
structure is designed to ensure the system stability in this paper. In
contradistinction to previous studies, the closed-loop error dynamics of a
feedback linearizable system are defined as the sliding surface for the
sliding mode learning algorithm as below:

$\displaystyle s$ $\displaystyle=$
$\displaystyle\Big{(}\frac{d}{dt}+\lambda\Big{)}^{n}e_{1}$ (13)
$\displaystyle=$
$\displaystyle\dot{e}_{n}+\frac{n\lambda}{1!}e_{n}+\frac{n(n-1)}{{2!}}\lambda^{2}e_{n-1}+\ldots+\lambda^{n}e_{1}$

where $n$ is the order of a feedback linearizable system and $\lambda$ is the
slope of the sliding surface. The condition $s=0$ guarantees that the error is
zero when the system is on the sliding surface, and the $n$th order system is
stable.

The sliding surface is rewritten by taking the feedback control law (3) into
consideration as follows:

$s=\dot{e}_{n}+u_{b}$ (14)

where the controller gain vector in the feedback control law is defined as

$\textbf{k}=[k_{1},\ldots,k_{n-1},k_{n}]=\Big{[}\lambda^{n},\ldots,\frac{n(n-1)}{{2!}}\lambda^{2},\frac{n\lambda}{1!}\Big{]}$
(15)

######  Assumption 1.

The total input rate and the second time-derivative of state $x_{n}$ are
assumed to be bounded:

$\mid\dot{u}_{t}\mid<B_{\dot{u}_{t}},\quad\textrm{and}\quad\mid\ddot{x}_{n}\mid<B_{\ddot{x}_{n}}$
(16)

where $B_{\dot{u}_{t}}$ and $B_{\ddot{x}_{n}}$ are considered as positive
constants. Moreover, the learning rate $\alpha$ is assumed to be selected
larger than the sum of these positive constants

$\alpha>B_{\dot{u}_{t}}+B_{\ddot{x}_{n}}$ (17)

In the light of Assumption 1, since the second time-derivative of $x_{n}$ is
bounded, the second time-derivative of $r_{n}$ must also be bounded.

######  Theorem 1.

The parameters adaptation laws of the neuro-fuzzy structure are given by the
following equations:

$\dot{c}_{ik}=\dot{e}_{i}+(e_{i}-c_{ik})\alpha\textrm{sgn}\left(s\right)$ (18)
$\dot{\sigma}_{ik}=-\bigg{(}\sigma_{ik}+\frac{(\sigma_{ik})^{3}}{(e_{i}-c_{ik})^{2}}\bigg{)}\alpha\textrm{sgn}\left(s\right)$
(19)

$\dot{f}_{r}=\frac{\widetilde{\textbf{w}}_{r}}{\widetilde{\textbf{w}}^{T}_{r}\widetilde{\textbf{w}}_{r}}\alpha\textrm{sgn}\left(s\right)$
(20)

where $\alpha$ is a sufficiently positive constant as stated in Assumption 1.
This ensures that (i) the feedback control action approaches to zero for a
given arbitrary initial condition $u_{b}(0)$ and (ii) $\dot{x}_{n}$ can track
$\dot{r}_{n}$ so that $\dot{e}_{n}$ approaches to zero for a given arbitrary
initial condition $\dot{e}_{n}(0)$.

###### Proof.

The time derivative of Gaussian membership function (9) is written as follows:

$\dot{\mu}_{ik}(e_{i})=-M_{ik}\dot{M}_{ik}\mu_{ik}(e_{i})$ (21)

where

$\displaystyle
M_{ik}=\frac{e_{i}-c_{ik}}{\sigma_{ik}},\quad\dot{M}_{ik}=\frac{(\dot{e}_{i}-\dot{c}_{ik})\sigma_{ik}-(e_{i}-c_{ik})\dot{\sigma}_{ik}}{\sigma_{ik}^{2}}$
(22)

Combination of (18), (19), and (22) gives:

$K_{r}=\sum\limits_{i=1}^{I}M_{ik}\dot{M}_{ik}=I\alpha\textrm{sgn}\left(s\right)$
(23)

The time derivative of (10) is written as follows:

$\dot{w}_{r}=-K_{r}w_{r}$ (24)

The time derivative of (11) is written considering (24) as follows:

$\dot{\widetilde{w}}_{r}=-\widetilde{w}_{r}K_{r}+\widetilde{w}_{r}\sum\limits_{r=1}^{N}\widetilde{w}_{r}K_{r}$
(25)

The stability of the sliding mode learning algorithm is investigated by the
following Lyapunov function below:

$V=\frac{1}{2}\left(u_{b}+\dot{e}_{n}\right)^{2}$ (26)

The time derivative of the proposed Lyapunov function (26) is computed below:

$\dot{V}=(u_{b}+\dot{e}_{n})(\dot{u}_{b}+\ddot{e}_{n})\\\ $

Taking (7) into consideration, $\dot{u}_{b}$ is inserted into the equation
above. Then, it is re-organized consideration (14) as follows:

$\dot{V}=s(\dot{u}_{t}-\dot{u}_{f}-\dot{u}_{n}+\ddot{e}_{n})$ (27)

The time derivative of the output of the neuro-fuzzy structure (12) is
calculated as follows:

$\dot{u}_{n}=\sum\limits_{r=1}^{N}\dot{f}_{r}\widetilde{w}_{r}+f_{r}\dot{\widetilde{w}}_{r}$
(28)

If (25) is inserted into the equation above, (28) can be obtained:

$\dot{u}_{n}=\sum\limits_{i=1}^{N}\bigg{\\{}\dot{f}_{r}\widetilde{w}_{r}+f_{r}\Big{(}-\widetilde{w}_{r}K_{r}+\widetilde{w}_{r}\sum\limits_{i=1}^{N}\widetilde{w}_{r}K_{r}\Big{)}\bigg{\\}}$
(29)

If (23) is inserted into the aforementioned equation, (29) can be obtained:

$\dot{u}_{n}=\sum\limits_{r=1}^{N}\bigg{\\{}\dot{f}_{r}\widetilde{w}_{r}+f_{r}I\alpha\textrm{sgn}\left(s\right)\Big{(}-\widetilde{w}_{r}+\widetilde{w}_{r}\sum\limits_{r=1}^{N}\widetilde{w}_{r}\Big{)}\bigg{\\}}$
(30)

The aforementioned equation is rewritten by taking the definition
$\sum\limits_{i=1}^{N}\widetilde{w}_{r}=1$ into account as follows:

$\dot{u}_{n}=\sum\limits_{r=1}^{N}\dot{f}_{r}\widetilde{w}_{r}$ (31)

By using (20), (31) is obtained as follows:

$\dot{u}_{n}=\alpha\textrm{sgn}\left(s\right)$ (32)

Substitution of (32) into (III-B) gives:

$\dot{V}=s\Big{[}\dot{u}_{t}-\dot{u}_{f}-\alpha\textrm{sgn}\left(s\right)+\ddot{e}_{n}\Big{]}$
(33)

If the aforementioned equation is re-arranged considering
$\ddot{e}_{n}=\ddot{r}_{n}-\ddot{x}_{n}=\dot{u}_{f}-\ddot{x}_{n}$

$\dot{V}=s\Big{[}\dot{u}_{t}-\alpha\textrm{sgn}\left(s\right)-\ddot{x}_{n}\Big{]}$
(34)

As remarked in Assumption 1, if it is assumed that the total input rate
$\dot{u}_{t}$ and the second time-derivative of state $\ddot{x}_{n}$ are upper
bounded, it is obtained as follows:

$\dot{V}\leq\mid s\mid(B_{\dot{u}_{t}}+B_{\ddot{x}_{n}}-\alpha)$ (35)

If the learning rate $\alpha$ is large enough, i.e., $\alpha\geq
B_{\dot{u}_{t}}+B_{\ddot{x}_{n}}$, as stated in Theorem 1, the time derivative
of the Lyapunov function yields

$\dot{V}<0$ (36)

So that the sliding mode learning algorithm is stable, the feedback control
action $u_{b}$ approaches to zero and $\dot{x}_{n}$ can track $\dot{r}_{n}$.
This implies that the intelligent controller is capable of learning the system
behavior. ∎

######  Remark 1.

Sliding mode learning algorithm learns the system behaviour very fast so that
selection of initial conditions for the adaptation laws (18)-(20) does not
play a very critical role in system performance. However, if the initial
conditions for the adaptation laws are very large, then parameter boosting
might occur and cause instability of the training algorithm.

###  III-C System Stability Analysis

######  Assumption 2.

The rate of the uncertainty term in the system dynamics (II) is assumed to be
bounded:

$\mid\dot{\Delta}(\textbf{x},u)\mid<B_{\dot{\Delta}(\textbf{x},u)}$ (37)

where $B_{\dot{\Delta}(\textbf{x},u)}$ is a positive constant.

######  Theorem 2 (System stabiltiy).

The CLELC algorithm (6) is applied to the feedback linearizable system (II).
If the learning rate $\alpha$ is large enough as stated in Assumption 2, i.e.,
$\alpha>B_{\dot{\Delta}(\textbf{x},u)}$, the sliding surface $s$ approaches to
zero for an arbitrary initial condition $s(0)$ within finite time $t_{h}$,
which is estimated as

$t_{h}\leq\frac{\mid s(0)\mid}{\alpha-B_{\dot{\Delta}(\textbf{x},u)}}$ (38)

This purports that the sliding surface $s(t)$ approaches to zero for all
$t>t_{h}$.

###### Proof.

The stability of an $n$th order feedback linearizable system is checked by
following Lyapunov function:

$V=\frac{1}{2}s^{2}$ (39)

The time derivative of the Lyapunov function is calculated as follows:

$\dot{V}=s\dot{s}$ (40)

The sliding surface rate $\dot{s}$ is obtained by taking (14) into account.
Then, it is inserted into (40) as follows:

$\dot{V}=s(\ddot{e}_{n}+\dot{u}_{b})$ (41)

If the control law for a CLELC algorithm (6) is applied to the feedback
linearizable system with uncertainties (II), it is obtained

$\dot{x}_{n}=u_{b}+u_{f}+u_{n}+\Delta(\textbf{x},u)$ (42)

Then, the closed-loop error dynamics are obtained by inserting the feedforward
control action (4)

$\dot{e}_{n}=-u_{b}-u_{n}-\Delta(\textbf{x},u)$ (43)

Time derivative of (43) is taken and (41) is rewritten as below:

$\dot{V}=s\big{(}-\dot{u}_{n}-\dot{\Delta}(\textbf{x},u)\big{)}$ (44)

If the time derivative of the neuro-fuzzy output (32) is inserted into the
aforementioned equation, it is computed below:

$\dot{V}=s\big{(}-\alpha\textrm{sgn}\left(s\right)-\dot{\Delta}(\textbf{x},u)\big{)}$
(45)

As stated in Assumption 2, the rate of the uncertainty term is upper bounded

$\dot{V}\leq\mid s\mid\big{(}-\alpha+B_{\dot{\Delta}(\textbf{x},u)}\big{)}$
(46)

If the learning rate $\alpha$ is large enough, i.e.,
$\alpha>B_{\dot{\Delta}(\textbf{x},u)}$, as remarked in Theorem 2, the time
derivative of the Lyapunov function yields

$\dot{V}<0$ (47)

so that the sliding surface approaches to zero in a stable manner so that the
closed-loop system is stable.

It is possible now to be shown that such a convergence takes place in finite
time for an arbitrary initial condition. The sliding surface rate is satisfied
by the sliding surface trajectory $s(t)$ is as below:

$\dot{s}(t)=-\alpha\textrm{sgn}\left(s(t)\right)-\dot{\Delta}(\textbf{x},u)$
(48)

the solution of the sliding surface rate with an initial condition $s(0)$
satisfies for any $t\leq t_{h}$

$s(t)-s(0)=\int^{t}_{0}\dot{s}(\xi)d\xi=\int^{t}_{0}\big{(}-\alpha\textrm{sgn}\left(s(\xi)\right)-\dot{\Delta}(\textbf{x},u)\big{)}d\xi$
(49)

the solution takes zero value at time $t=t_{h}$, i.e., $s(t_{h})=0$, and the
arbitrary initial condition is inserted into the equation above, i.e.,
$\textrm{sgn}\left(s(\xi)\right)=\textrm{sgn}\left(s(0)\right)$, therefore

$\displaystyle\underbrace{s(t_{h})}_{0}-s(0)$ $\displaystyle=$
$\displaystyle\int^{t_{h}}_{0}\big{(}-\alpha\textrm{sgn}\left(s(0)\right)-\dot{\Delta}(\textbf{x},u)\big{)}d\xi$
$\displaystyle-s(0)$ $\displaystyle=$
$\displaystyle-\alpha\textrm{sgn}\left(s(0)\right)t_{h}-\int^{t_{h}}_{0}\dot{\Delta}(\textbf{x},u)d\xi$
(50)

Both sides of the aforementioned equation are multiplied by
$-\textrm{sgn}\left(s(0)\right)$, it is obtained as below:

$\displaystyle\mid s(0)\mid$ $\displaystyle=$ $\displaystyle\alpha
t_{h}+\textrm{sgn}\left(s(0)\right)\int^{t_{h}}_{0}\dot{\Delta}(\textbf{x},u)d\xi$
(51) $\displaystyle\geq$ $\displaystyle\alpha
t_{h}-B_{\dot{\Delta}(\textbf{x},u)}t_{h}$

A finite time $t_{h}$ is found as

$t_{h}\leq\frac{\mid s(0)\mid}{\alpha-B_{\dot{\Delta}(\textbf{x},u)}}$ (52)

This purports that a sliding motion is maintained on $s(t)=0$ $\forall
t>t_{h}$. ∎

######  Remark 2.

As stated in Theorems 1 and 2, the learning rate $\alpha$ must be large enough
to guarantee the stability of the learning algorithm and the system stability;
however, it is to be noted that a larger $\alpha$ may cause the chattering
effect. Therefore, it is so crucial to determine the learning rate $\alpha$.
In this study, the sgn functions are replaced by the following equation to
decrease the chattering effect $\textrm{sgn}\left(s\right)=\frac{s}{\mid
s\mid+\delta}$ where $\delta=0.05$. It is important to state that the
selection of $\delta$ is very critical and must be carried out by taking the
learning rate $\alpha$ into account to reduce the chattering effect: (i) if
$\delta$ is small, it may not be enough to reduce the chattering effect, (ii)
if $\delta$ is large, it may cause steady-state error. Therefore, an
appropriate value is selected considering anticipated disturbance in this
paper.

##  IV Simulation Results

The following third-order numerical system is considered for the simulation
study:

$\displaystyle\left[\dot{x}_{1},\dot{x}_{2},\dot{x}_{3}\right]^{T}=\left[x_{2},x_{3},a(\textbf{x})+b(\textbf{x})u+\Delta(\textbf{x},u)\right]^{T}$
(53)

where $a(\textbf{x})=-2x_{1}-x_{2}-\sin{(x_{3})}+e^{x_{1}}$ and
$b(\textbf{x})=1$.

The simulation study runs for a total of $20$ s with a sampling time of $0.01$
s. The number of membership functions for each input to the adaptive neuro-
fuzzy structure is set to $3$. The system states are initialized as
$\textbf{x}(0)=[1,-1,-10]^{T}$. The system is a third order system; therefore,
the sliding surface is obtained as $s=\dot{e}_{3}+3\lambda
e_{3}+3\lambda^{2}e_{2}+\lambda^{3}e_{1}$. Moreover, the slope of the sliding
surface $\lambda$ is selected as $3$; therefore, the sliding surface is
obtained as $s=\dot{e}_{3}+9e_{3}+27e_{2}+27e_{1}$ and the controller gain
vector is obtained as $\textbf{k}=[k_{1},k_{2},k_{3}]=[27,27,9]$ as formulated
in (15). The reference signal $r$ is equal to zero, i.e., $\textbf{r}=0$. The
disturbance is imposed on the system at $10$ s and formulated as
$\Delta(\textbf{x},u)=5\sin{(t)}$. As remarked in Theorem 2, the learning rate
$\alpha$ must be larger than the maximum value of the uncertainty rate, i.e.,
$\alpha>B_{\dot{\Delta}(\textbf{x},u)}$ to guarantee the system stability.
Therefore, the learning rate $\alpha$ is set to $25$, i.e.,
$\alpha=25>B_{\dot{\Delta}(\textbf{x},u)}=5$.

The performance of the CLELC algorithm is shown in Figs. 2(a)-2(c). At the
beginning of the simulation between $t=0-10$ s, the control performances of
the FLC and CLELC algorithms are compared while there is no uncertainty in the
system. The CLELC results in smaller rise and settling times, and overshoot
than the traditional FLC in the absence of uncertainties. It can be concluded
that the CLELC algorithm improves the transient response performance of the
system. In next step, a sinusoidal disturbance
$\Delta(\textbf{x},u)=5\sin{(t)}$ is imposed on the system between $t=10-20$ s
to evaluate the robustness of the CLELC algorithm against uncertainty. The
traditional FLC is not robust against uncertainties after the disturbance is
imposed on the system. The CLELC algorithm learns system behavior online by
adjusting its parameters and controls the system without any steady-state
error. Thanks to the CLELC scheme, the developed control algorithm results in
robust control performance in the presence of uncertainties.

The sliding surface of the sliding mode learning algorithm must converge to
zero in finite time, which is proven in Theorem 2. Taking (52) into account,
the finite time is found as $t_{h}\leq 4.5$ seconds, where $s(0)=-90$,
$\alpha=25$ and $B_{\dot{\Delta}(\textbf{x},u)}=5$. As can be seen from Figs.
2(a)-2(c), the states converges to zero less than $4.5$ seconds, which
confirms the formulation in (52).

The control signals for the CLELC algorithm are shown in Fig. 2(d). The
intelligent controller takes the overall control of the control signal in a
very short duration; therefore, the feedback control input $u_{b}$ approaches
to zero. This implies that the CLELC algorithm learns the system dynamics.
Note that the feed-forward control input is set to zero throughout the
simulation study since the reference is equal to zero.

(a) Responses of state $x_{1}$.

(b) Responses of state $x_{2}$.

(c) Responses of state $x_{3}$.

(d) Control signals.

Figure 2: Simulation results.

##  V Experimental Validation

###  V-A Mobile Robot

Figure 3: Mobile robot.

The mobile robot shown in Fig. 3 has been equipped with sensors. Plant imaging
and phenotyping sensors, as well as a Real-Time Kinematic (RTK) differential
Global Navigation Satellite System (GNSS) are mounted on the mast on the
robot. A Septentrio Altus APS-NR3 GNSS (Septentrio Satellite Navigation NV,
Belgium) has been employed to acquire the position of the mobile robot with an
accuracy of $0.04$ m at a $5$-Hz sampling rate. More information about the
actuators and sensors mounted on the mobile robot can be found in [29].

The CLELC algorithm is implemented in LabVIEWTM (v2015, National Instrument
Corporation, USA) and executed in real time on the onboard computer. It
gathers the GNSS data, linear and angular velocities, and regulates the speed
of the tracks by sending signals to the motion controller. The control input
generated by the CLELC algorithm is sent to the motion controller via NI Myrio
(National Instrument Corporation, USA), and linear and angular velocities
measurements are received by the same way. Position measurements are received
via Bluetooth. The sampling frequency of the CLELC algorithm is set to $5$-Hz.

###  V-B Feedback Linearization Control of a Mobile Robot

The schematic diagram of the mobile robot is illustrated in Fig. 4. The mobile
robot is a unicycle type robot and formulated as [33, 34, 35]:

$\displaystyle\left[\dot{p_{x}},\dot{p_{y}},\dot{\theta}\right]^{T}=\left[v\cos{(\theta)},v\sin{(\theta)},\omega\right]^{T}$
(54)

where $p_{x}$ and $p_{y}$ are respectively the position of the mobile robot in
x- and y-axes, $\theta$ is the heading angle, $v$ is the linear velocity, and
$\omega$ is the angular velocity.

Figure 4: Schematic illustration of the mobile robot.

The feedback linearization procedure is illustrated for the unicycle model
(54) in literature [33]. The linearizing states are defined as $x_{1}=p_{x}$,
$x_{2}=p_{y}$. If $x_{1}$ and $x_{2}$ are differentiated taking (54) into
account,

$\displaystyle\left[\begin{array}[]{c}\ddot{x}_{1}\\\
\ddot{x}_{2}\end{array}\right]=\dot{v}\left[\begin{array}[]{c}\cos{(\theta)}\\\
\sin{(\theta)}\end{array}\right]+v\dot{\theta}\left[\begin{array}[]{r}-\sin{(\theta)}\\\
\cos{(\theta)}\end{array}\right]$ (61)

Let us define linear acceleration as $\xi=\dot{v}$ and consider
$\dot{\theta}=\omega$. The aforementioned equation is re-arranged as follows:

$\displaystyle\left[\begin{array}[]{c}\ddot{x}_{1}\\\
\ddot{x}_{2}\end{array}\right]=\left[\begin{array}[]{l
r}\cos{(\theta)}&-v\sin{(\theta)}\\\
\sin{(\theta)}&v\cos{(\theta)}\end{array}\right]\left[\begin{array}[]{c}\xi\\\
\omega\end{array}\right]$ (68)

If the linear velocity is not equal to zero, i.e., $v\neq 0$, the matrix is
non-singular. Thus, $\xi$ and $\omega$ are obtained as follows:

$\displaystyle\left[\begin{array}[]{c}\xi\\\
\omega\end{array}\right]=\left[\begin{array}[]{l
r}\cos{(\theta)}&-v\sin{(\theta)}\\\
\sin{(\theta)}&v\cos{(\theta)}\end{array}\right]^{-1}\left[\begin{array}[]{c}\ddot{x}_{1}\\\
\ddot{x}_{2}\end{array}\right]$ (75)

The system is linearized and described by two chains of integrator
$\left[\ddot{x}_{1},\ddot{x}_{2}\right]^{T}=\left[u_{1},u_{2}\right]^{T}$ so
that the feedback linearization law is obtained as follows:

$\dot{v}=u_{1}\cos{\theta}+u_{2}\sin{\theta},\quad\textrm{and}\quad\omega=\frac{u_{2}\cos{\theta}-u_{1}\sin{\theta}}{v}$
(76)

The new states are defined as
$\textbf{x}=[x_{1},x_{2},x_{3},x_{4}]^{T}=[p_{x},p_{y},\dot{p}_{x},\dot{p}_{y}]^{T}$
and the new state-space equations are written as follows:

$\displaystyle\begin{matrix}\dot{x}_{1}&=&x_{3}\\\ \dot{x}_{3}&=&u_{1}\\\
\end{matrix}\qquad\text{and}\qquad\begin{matrix}\dot{x}_{2}&=&x_{4}\\\
\dot{x}_{4}&=&u_{2}\\\ \end{matrix}$ (77)

It is assumed that the mobile robot tracks a smooth trajectory
$(r_{1}(t),r_{2}(t))$ and the desired linear velocity along this trajectory
never goes to zero. The control laws are written as follows:

$\displaystyle\left[\begin{array}[]{c}u_{1}\\\
u_{2}\end{array}\right]=\left[\begin{array}[]{c}\ddot{r}_{1}+k_{1}({r}_{1}-x_{1})+k_{3}({r}_{3}-x_{3})\\\
\ddot{r}_{2}+k_{2}({r}_{2}-x_{2})+k_{4}({r}_{4}-x_{4})\end{array}\right]$ (82)

These generated signals are fed to the transformation matrix in (76) to obtain
the actual inputs, i.e., the desired linear and angular velocities, for a
mobile robot.

###  V-C Implementation of the CLELC algorithm

The CLELC laws for a mobile robot are written by utilizing (6) and (82) as
follows:

$\displaystyle\left[\begin{array}[]{c}u_{1}\\\
u_{2}\end{array}\right]=\left[\begin{array}[]{c}\ddot{r}_{1}+k_{1}({r}_{1}-x_{1})+k_{3}({r}_{3}-x_{3})+u_{n1}\\\
\ddot{r}_{2}+k_{2}({r}_{2}-x_{2})+k_{4}({r}_{4}-x_{4})+u_{n2}\end{array}\right]$
(87)

The number of membership functions for each input to each adaptive neuro-fuzzy
structure is set to $3$ and the learning rate for the sliding mode learning
algorithm $\alpha$ is set to $5$.

After feedback linearization is applied, the mobile robot dynamics consists of
two single-input-single-output systems which are second-order systems.
Therefore, the sliding surfaces are obtained as $s_{1}=\dot{e}_{3}+2\lambda
e_{3}+\lambda^{2}e_{1}$ and $s_{2}=\dot{e}_{4}+2\lambda
e_{4}+\lambda^{2}e_{2}$. The slope of the sliding surface $\lambda$ is set to
$0.3$; therefore, the controller gain vector is obtained as
$\textbf{k}=[k_{1},k_{2},k_{3},k_{4}]=[0.09,0.09,0.6,0.6]$ as formulated in
(15).

The heading angle reference is derived from (54) as follows:

$\theta_{r}=\arctan 2{(r_{4},r_{3})}+k\pi\quad k=0,1$ (88)

To increase the tracking accuracy, feedforward actions are derived in a time-
interval $t\in[0,T]$ by using the system model (54) and the desired linear
velocities in x- and y-axes $(r_{3}(t),r_{4}(t))$ as follows [36]:

$v_{d}=\pm\sqrt{(r_{3})^{2}+(r_{4})^{2}}\quad\textrm{and}\quad\omega_{d}=\frac{\dot{r_{4}}r_{3}-\dot{r_{3}}r_{4}}{(r_{3})^{2}+(r_{4})^{2}}$
(89)

###  V-D Experimental Results

The start and end points of the reference trajectory and the initial point of
the mobile robot are shown in Fig. 5(a). The traditional FLC and the CLELC can
reach to the reference trajectory and stay on-track. The Euclidean errors for
the FLC and CLELC are shown in Fig. 5(b). The mean values of Euclidean errors
for the mobile robot controlled by the FLC and CLELC after staying on-track
are respectively $0.12$ $m$ and $0.03$ $m$ approximately. This purports that
the CLELC results in more accurate trajectory tracking performance than the
FLC, which demonstrates the capability of the CLELC algorithm. In the previous
works, a nonlinear model predictive controller has been designed and
implemented on the same mobile robot in [37]. Trajectory tracking error has
been around $0.05$ $m$. Moreover, the kinematic model was augmented with two
traction parameters and a nonlinear moving horizon estimator was used to
estimate these traction parameters. Then, a nonlinear model predictive
controller based on the adaptive kinematic model has been designed and
implemented on the same mobile robot in [29]. Trajectory tracking error has
been around $0.04$ $m$. Consequently, the CLELC algorithm gives more accurate
trajectory tracking performance than previous implemented controllers on the
same mobile robot.

The reference and measured heading angles are shown in Fig. 5(c). The
reference heading angle is calculated from the reference trajectory as
formulated in (88). The mobile robot controlled by the CLELC is capable of
tracking the heading angle reference accurately. The same consequence can be
obtained from the heading angle error as shown in Fig. 5(d).

The feedback and intelligent control signals fed to the feedback linearization
law (76) are respectively shown for $u_{1}$ and $u_{2}$ in Figs. 5(e)-5(f).
The outputs of the feedback controllers converge to zero while the intelligent
controllers learn the mobile robot behavior online. The intelligent
controllers take over the control, thus becoming the leading controllers.

The output of the feedback linearization law (76) results in the desired
linear and angular velocities fed to the mobile robot. The linear velocity
reference and measurements of the mobile robot are shown in Fig. 5(g). The
linear velocity reference of the mobile robot is changing around a constant
speed because the linear velocity throughout the reference trajectory is
constant. The angular velocity reference and measurements are shown in Fig.
5(h). The minimum and maximum angular velocities are limited by $-0.1$ and
$0.1$ rad/s due to the capability of the mobile robot. The performance of the
low-level controller is sufficient to track the reference signals despite the
high noisy measurements.

(a) Reference trajectory.

(b) Euclidean error.

(c) Heading angle reference and measurements.

(d) Heading angle error.

(e) Feedback and intelligent control signals for $u_{1}$.

(f) Feedback and intelligent control signals for $u_{2}$.

(g) Linear velocity reference and measurements.

(h) Angular velocity reference and measurements.

Figure 5:  Experimental results on the mobile robot.

##  VI Conclusions

In this study, a novel CLELC algorithm has been developed for the control of
feedback linearizable systems with uncertainties and experimentally validated
on a mobile robot. The simulation results show that the CLELC algorithm has
exhibited better control performance in terms of smaller rise time, settling
time and overshoot in the absence of uncertainties as compared to the
traditional FLC. Moreover, it can drive the error to zero in the presence of
uncertainties. The experimental results on a tracked mobile robot show that
the CLELC algorithm ensures more accurate trajectory tracking performance when
compared to the FLC. The mean values of the Euclidean errors for the FLC and
CLELC after staying on-track are respectively $0.12$ m and $0.03$ m
approximately.

As a future study, the proposed algorithm can be developed for uncertain
multi-input-multi-output nonlinear systems and the constraints on the error
and reference rates in the stability analysis can be lightened.

## References

  * [1] E. Kayacan, E. Kayacan, H. Ramon, and W. Saeys, “Velocity control of a spherical rolling robot using a grey-pid type fuzzy controller with an adaptive step size,” _IFAC Proceedings Volumes_ , vol. 45, no. 22, pp. 863 – 868, 2012, 10th IFAC Symposium on Robot Control.
  * [2] W. H. Chen, J. Yang, L. Guo, and S. Li, “Disturbance-observer-based control and related methods: An overview,” _IEEE Transactions on Industrial Electronics_ , vol. 63, no. 2, pp. 1083–1095, Feb 2016.
  * [3] E. Kayacan, J. M. Peschel, and G. Chowdhary, “A self-learning disturbance observer for nonlinear systems in feedback-error learning scheme,” _Engineering Applications of Artificial Intelligence_ , vol. 62, pp. 276 – 285, 2017.
  * [4] I. Matraji, A. Al-Durra, A. Haryono, K. Al-Wahedi, and M. Abou-Khousa, “Trajectory tracking control of skid-steered mobile robot based on adaptive second order sliding mode control,” _Control Engineering Practice_ , vol. 72, pp. 167 – 176, 2018.
  * [5] B. Xiao, Q. Dong, D. Ye, L. Liu, and X. Huo, “A general tracking control framework for uncertain systems with exponential convergence performance,” _IEEE/ASME Transactions on Mechatronics_ , vol. 23, no. 1, pp. 111–120, Feb 2018.
  * [6] M. Van, “An enhanced robust fault tolerant control based on an adaptive fuzzy pid-nonsingular fast terminal sliding mode control for uncertain nonlinear systems,” _IEEE/ASME Transactions on Mechatronics_ , vol. 23, no. 3, pp. 1362–1371, June 2018.
  * [7] P. Yu, M. Wu, J. She, K. Liu, and Y. Nakanishi, “Robust tracking and disturbance rejection for linear uncertain system with unknown state delay and disturbance,” _IEEE/ASME Transactions on Mechatronics_ , vol. 23, no. 3, pp. 1445–1455, June 2018.
  * [8] E. Kayacan, “Multiobjective $h_{\infty}$ control for string stability of cooperative adaptive cruise control systems,” _IEEE Transactions on Intelligent Vehicles_ , vol. 2, no. 1, pp. 52–61, March 2017.
  * [9] Z. Sun, J. Zheng, Z. Man, and H. Wang, “Robust control of a vehicle steer-by-wire system using adaptive sliding mode,” _IEEE Transactions on Industrial Electronics_ , vol. 63, no. 4, pp. 2251–2262, April 2016.
  * [10] B. Ren, Q. C. Zhong, and J. Dai, “Asymptotic reference tracking and disturbance rejection of ude-based robust control,” _IEEE Transactions on Industrial Electronics_ , vol. 64, no. 4, pp. 3166–3176, April 2017.
  * [11] H. Liu, W. Zhao, Z. Zuo, and Y. Zhong, “Robust control for quadrotors with multiple time-varying uncertainties and delays,” _IEEE Transactions on Industrial Electronics_ , vol. 64, no. 2, pp. 1303–1312, Feb 2017.
  * [12] I. Guenoune, F. Plestan, A. Chermitti, and C. Evangelista, “Modeling and robust control of a twin wind turbines structure,” _Control Engineering Practice_ , vol. 69, pp. 23 – 35, 2017.
  * [13] E. Kayacan and T. I. Fossen, “Feedback linearization control for systems with mismatched uncertainties via disturbance observers,” _Asian Journal of Control_ , vol. 21, no. 3, pp. 1064–1076, 2019.
  * [14] J. Huang, S. Ri, L. Liu, Y. Wang, J. Kim, and G. Pak, “Nonlinear disturbance observer-based dynamic surface control of mobile wheeled inverted pendulum,” _IEEE Transactions on Control Systems Technology_ , vol. 23, no. 6, pp. 2400–2407, Nov 2015.
  * [15] M. Amini, M. Shahbakhti, S. Pan, and J. Hedrick, “Bridging the gap between designed and implemented controllers via adaptive robust discrete sliding mode control,” _Control Engineering Practice_ , vol. 59, pp. 1 – 15, 2017\.
  * [16] M. P. Aghababa, “Sliding-mode control composite with disturbance observer for tracking control of mismatched uncertain ndof nonlinear systems,” _IEEE/ASME Transactions on Mechatronics_ , vol. 23, no. 1, pp. 482–490, Feb 2018.
  * [17] C. Huang, F. Naghdy, and H. Du, “Observer-based fault tolerant controller for uncertain steer-by-wire systems using the delta operator,” _IEEE/ASME Transactions on Mechatronics_ , pp. 1–1, 2018.
  * [18] H. Mousazadeh, “A technical review on navigation systems of agricultural autonomous off-road vehicles,” _Journal of Terramechanics_ , vol. 50, no. 3, pp. 211 – 232, 2013.
  * [19] M. Deremetz, R. Lenain, B. Thuilot, and V. Rousseau, “Adaptive trajectory control of off-road mobile robots: A multi-model observer approach,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2017, pp. 4407–4413.
  * [20] M. Begnini, D. W. Bertol, and N. A. Martins, “A robust adaptive fuzzy variable structure tracking control for the wheeled mobile robot: Simulation and experimental results,” _Control Engineering Practice_ , vol. 64, pp. 27 – 43, 2017.
  * [21] C. J. Ostafew, A. P. Schoellig, and T. D. Barfoot, “Robust constrained learning-based nmpc enabling reliable mobile robot path tracking,” _The International Journal of Robotics Research_ , vol. 35, no. 13, pp. 1547–1563, 2016\.
  * [22] F. G. Rossomando, C. Soria, and R. Carelli, “Sliding mode neuro adaptive control in trajectory tracking for mobile robots,” _Journal of Intelligent & Robotic Systems_, vol. 74, no. 3, pp. 931–944, Jun 2014.
  * [23] C. Yu and X. Chen, “Trajectory tracking of wheeled mobile robot by adopting iterative learning control with predictive, current, and past learning items,” _Robotica_ , vol. 33, no. 7, p. 1393–1414, 2015.
  * [24] C. J. Ostafew, A. P. Schoellig, T. D. Barfoot, and J. Collier, “Learning-based nonlinear model predictive control to improve vision-based mobile robot path tracking,” _Journal of Field Robotics_ , vol. 33, no. 1, pp. 133–152, 2016\.
  * [25] C. Cariou, R. Lenain, B. Thuilot, and M. Berducat, “Automatic guidance of a four-wheel-steering mobile robot for accurate field operations,” _Journal of Field Robotics_ , vol. 26, no. 6-7, pp. 504–518, 2009.
  * [26] E. Kayacan, J. M. Peschel, and E. Kayacan, “Centralized, decentralized and distributed nonlinear model predictive control of a tractor-trailer system: A comparative study,” in _2016 American Control Conference (ACC)_ , July 2016, pp. 4403–4408.
  * [27] E. Kayacan, E. Kayacan, I.-M. Chen, H. Ramon, and W. Saeys, _On the Comparison of Model-Based and Model-Free Controllers in Guidance, Navigation and Control of Agricultural Vehicles_. Cham: Springer International Publishing, 2018, pp. 49–73.
  * [28] E. Kayacan, W. Saeys, H. Ramon, C. Belta, and J. Peschel, “Experimental validation of linear and nonlinear mpc on an articulated unmanned ground vehicle,” _IEEE/ASME Transactions on Mechatronics_ , vol. 23, no. 5, pp. 2023–2030, Oct 2018.
  * [29] E. Kayacan, S. N. Young, J. M. Peschel, and G. Chowdhary, “High-precision control of tracked field robots in the presence of unknown traction coefficients,” _Journal of Field Robotics_ , vol. 35, no. 7, pp. 1050–1062.
  * [30] E. Kayacan, Z. Zhang, and G. Chowdhary, “Embedded high precision control and corn stand counting algorithms for an ultra-compact 3d printed field robot,” in _Proceedings of Robotics: Science and Systems_ , Pittsburgh, Pennsylvania, June 2018.
  * [31] E. Kayacan, “Sliding mode learning control of uncertain nonlinear systems with lyapunov stability analysis,” _Transactions of the Institute of Measurement and Control_ , vol. 41, no. 6, pp. 1750–1760, 2019.
  * [32] M. S. Ayas and I. H. Altas, “Fuzzy logic based adaptive admittance control of a redundantly actuated ankle rehabilitation robot,” _Control Engineering Practice_ , vol. 59, pp. 44 – 54, 2017.
  * [33] G. Oriolo, A. D. Luca, and M. Vendittelli, “Wmr control via dynamic feedback linearization: design, implementation, and experimental validation,” _IEEE Transactions on Control Systems Technology_ , vol. 10, no. 6, pp. 835–852, Nov 2002.
  * [34] M. Seder, M. Baotić, and I. Petrović, “Receding horizon control for convergent navigation of a differential drive mobile robot,” _IEEE Transactions on Control Systems Technology_ , vol. 25, no. 2, pp. 653–660, March 2017.
  * [35] H. Xiao, Z. Li, C. Yang, L. Zhang, P. Yuan, L. Ding, and T. Wang, “Robust stabilization of a wheeled mobile robot using model predictive control based on neurodynamics optimization,” _IEEE Transactions on Industrial Electronics_ , vol. 64, no. 1, pp. 505–516, Jan 2017.
  * [36] E. Kayacan and G. Chowdhary, “Tracking error learning control for precise mobile robot path tracking in outdoor environment,” _Journal of Intelligent & Robotic Systems_, Aug 2018.
  * [37] S. N. Young, E. Kayacan, and J. M. Peschel, “Design and field evaluation of a ground robot for high-throughput phenotyping of energy sorghum,” _Precision Agriculture_ , vol. 20, no. 4, p. 697–722, August 2019.

|  Erkan Kayacan (S’12-M’16-SM’19) received the B.Sc. degree in mechanical
engineering and the M.Sc. degree in system dynamics and control from Istanbul
Technical University, Istanbul, Turkey in 2008 and 2010, respectively. He
received the Ph.D. degree in Mechatronics, Biostatistics and Sensors from
University of Leuven (KU Leuven), Leuven, Belgium in 2014. He is currently a
Lecturer with the School of Mechanical and Mining Engineering at the
University of Queensland (UQ), Australia. Prior to UQ, he was a Postdoctoral
Researcher with Delft University of Technology, University of Illinois at
Urbana-Champaign, and Massachusetts Institute of Technology. His research
interests include real-time optimization-based control and estimation methods,
nonlinear control theory, learning algorithms and machine learning with a
heavy emphasis on applications to autonomous systems and field robotics. Dr.
Kayacan is a recipient of the Best Systems Paper Award at Robotics: Science
and Systems (RSS) in 2018.
---|---


# Diffusion Models Enable Zero-Shot Pose Estimation for Lower-Limb Prosthetic
Users

Tianxun Zhou  Muhammad Nur Shahril Iskandar  Keng-Hwee Chiam

###### Abstract

The application of 2D markerless gait analysis has garnered increasing
interest and application within clinical settings. However, its effectiveness
in the realm of lower-limb amputees has remained less than optimal. In
response, this study introduces an innovative zero-shot method employing image
generation diffusion models to achieve markerless pose estimation for lower-
limb prosthetics, presenting a promising solution to gait analysis for this
specific population. Our approach demonstrates an enhancement in detecting key
points on prosthetic limbs over existing methods, and enables clinicians to
gain invaluable insights into the kinematics of lower-limb amputees across the
gait cycle. The outcomes obtained not only serve as a proof-of-concept for the
feasibility of this zero-shot approach but also underscore its potential in
advancing rehabilitation through gait analysis for this unique population.

###### keywords:

transtibial , transfemoral , amputee , gait , locomotion , biomechanics ,
movement science

[inst1]organization=Bioinformatics Institute,addressline=30 Biopolis Street,
postcode=138671, country=Singapore

[inst2]organization=Nanyang Technological University,addressline=1 Nanyang
Walk, postcode=637616, country=Singapore

##  1 Introduction

Human gait, especially in lower limb amputees, is a complex motor task
requiring coordinated movement of various body segments. Extensive research
has established that individuals in these populations frequently exhibit
distinct gait abnormalities and physiological challenges. For example, it is
known that gait asymmetry [1, 2] is prevalent, where the prosthetic and intact
limbs display different walking patterns. The presence of gait asymmetry may
contribute to the development of degenerative joint disease [3]. Previous
studies on physiological differences have also revealed that above-knee
amputations lead to an increase in oxygen consumption by approximately 49%
[4], in addition to an approximately 65% higher energy expenditure when
compared to individuals without amputations [5]. These issues could hinder
mobility and functionality, ultimately diminishing the quality of life [6].
Therefore, clinicians are often keen to understand the gait patterns of lower
limb amputees both in general and at an individual level, in order to improve
the quality of life of prosthetic users through better rehabilitation
monitoring, providing better-fitted prosthetics, reducing gait abnormalities
and more.

Gait analysis offers a systematic approach to identify pathological gait
patterns associated with neurological [7], musculoskeletal [8], and other
disorders by collecting various gait parameters data [9, 10]. This allows
clinicians to identify deviations from normal gait patterns and tailor
interventions to address the specific issue. Gait analysis can be broadly
classified as either observational[11], or quantitative.

In observational gait analysis (OGA), clinicians typically rely on a checklist
to note the presence of pathological gait characteristics [2, 11]. For
instance, studies have shown that individuals with lower-limb amputations
often exhibit a lateral trunk lean towards the prosthetic limb [2, 12]. When
clinicians observe such characteristics during OGA, they simply mark a tick on
the checklist. Subsequently, they may address these issues by implementing
interventions to reduce the extent of lateral trunk lean through targeted
training. These metrics are often qualitative, which may introduce errors in
terms of inter- and intra-rater reliability [13] and are recommended to
supplement their clinical assessment with some form of quantitative
measurement [14, 15].

Quantitative gait assessments have been facilitated by technologies such as
instrumented motion analysis [16], force platforms, electromyography [1], and
wearable sensors. These tools allow clinicians to systematically quantify a
spectrum of gait parameters, enabling them to effectively monitor progress
throughout the rehabilitation journey. For instance, Sjödahl et al. [17] used
force platforms and motion-capture systems to identify increased hip flexion
at the beginning of the stance phase on the intact limb, offering timely
feedback to the patient. Furthermore, intricate spatiotemporal variables such
as step length and cadence could be scrutinized in meticulous detail. The
study revealed a reduction in variability on the prosthetic limb, indicative
of a more stabilized and symmetrical gait pattern post-treatment. Such
nuanced, comprehensive and timely analysis would not be possible if clinicians
were tasked with manually tracking each individual step.

Therefore, although trained clinicians may possess the expertise to identify
pathological gait patterns in OGA, the introduction of quantification brings
objectivity into the assessment, facilitating more accurate and precise
measurements of diverse gait parameters. This also becomes crucial when
comparing results among different clinicians, ensuring consistency and
reducing subjective interpretation. Furthermore, by collecting standardized
and objective data, researchers can analyze gait patterns across a larger
sample of amputees, compare rehabilitation strategies, and evaluate the
effectiveness of different prosthetic devices.

While trained clinicians may possess the expertise to discern pathological
gait patterns through OGA, the integration of quantification brings greater
objectivity into the assessments. This would enable more precise and accurate
measurements of a diverse range of gait parameters. This is particularly
crucial when comparing results across different clinicians, ensuring
consistency and mitigating subjective interpretation. Moreover, through the
collection of standardized and objective data, researchers gain the ability to
analyze gait patterns across a broader spectrum of amputees, compare
rehabilitation strategies, and assess the efficacy of various prosthetic
devices with unprecedented granularity.

Traditionally, obtaining quantitative measurements demands specialized, often
costly equipment, which confines data collection to controlled laboratory
environments, subsequently limiting the broader applicability of findings [18,
19]. In recent years, an alternative arises through the utilization of high-
speed video recording on commercial video cameras. Advances in deep learning
have enabled rapid progress in human pose estimation from videos [20, 21, 22,
23]. Pose estimation software such as OpenPose [24] are able to estimate the
position of keypoints such as joints on images of humans without the need for
markers. With multiple cameras, it is possible to triangulate the keypoints to
obtain 3D coordinates [25]. Such markerless pose estimation techniques may
potentially be the key to cost-effective systems for automated clinical gait
analysis. Several studies have examined the accuracy of available pose
estimation methods specifically for gait analysis applications [10, 26, 27,
28].

Current markerless pose estimation models face challenges in accurately
localizing joints on prosthetic limbs [29]. This limitation stems from their
training on datasets primarily featuring able-bodied individuals, resulting in
poor generalization to unseen prosthetic limbs. Custom models tailored for
prosthetic keypoint identification have been proposed, yet their applicability
across diverse settings is constrained by the wide array of prosthetic
appearances and limited training data [29, 30]. Achieving robust
generalization across varied settings necessitates training on extensive
datasets akin to those used in general human pose estimation, like the COCO
dataset underlying OpenPose. However, the resource-intensive manual labelling
of such datasets presents a formidable barrier in terms of cost and time and
would likely be beyond the reach of rehabilitation labs.

This paper introduces a novel zero-shot pose estimation method leveraging
existing pre-trained image generative diffusion models and pose estimation
frameworks to achieve accurate pose estimation for lower limb prosthetic
users. Recent advances in image generation artificial intelligence, notably
with denoising diffusion models, have enabled the generation of remarkably
realistic images conditioned on diverse inputs, including text descriptions
and sample images [31]. Leveraging the capabilities of these image generation
models, we transform prosthetic limbs in images into representations
resembling able-bodied limbs while preserving their position and shape. This
technique empowers existing pose estimation models to make accurate keypoint
estimations on lower-limb prosthetics without further training.

To our knowledge, this is the first work that achieves zero-shot pose
estimation for lower-limb prosthetic users. The contributions of this paper
are twofold. First, we provide quantification of the errors made by one of the
most widely used pre-trained pose estimation software, OpenPose, on lower-limb
prosthetic users. Second, we demonstrate a working method for accurate zero-
shot pose estimation on lower-limb prosthetic users without the need for any
data collection, labelling or training.

##  2 Methods

We analyzed publicly accessible videos of individuals with unilateral lower-
limb prosthetic walking from video sharing website YouTube [32]. For each
video, an image generative model, ControlNet, was utilized to generate
synthetic images corresponding to every frame of the video. Subsequently,
OpenPose was applied to these newly synthesized images, allowing for the
extraction of anatomical keypoints. Inverse kinematics was performed based on
the 2D coordinates generated. We then evaluated the performance based on a
custom model created using DeepLabCut (DLC). Gardiner et al. [33] have shown
that using publicly available videos produces results that are comparable with
published data from controlled laboratory studies.

###  2.1 Dataset

The video search was conducted by a single researcher with terms include:
gait, walking, amputee, transtibial, and transfemoral. A total of 16 videos
containing two subjects with different amputation levels; a female
transfemoral and a male transtibial amputee, were downloaded. Essentially,
each subject has two sets of four videos (Supplementary Fig. 1), recorded at
different camera positions (i.e., anterior, posterior, left and right). Both
subjects are amputees on the right side. As the source of videos was publicly
available, the etiology for amputation, time since amputation and type of
prosthetic used were unknown. The videos were downloaded at a resolution of
1280 x 720 and 30 frames per second. Subsequently, each video was trimmed to 6
seconds (180 frames) as the initial videos included the subject changing
walking direction, before following the zero-shot pose estimation pipeline.

###  2.2 Zero-shot pose estimation pipeline

The pipeline for zero-shot pose estimation is as follows. Given a video, for
each frame, we first apply edge detection using Canny edge detection to
generate an edge map. The edge map is then used as the conditional control
into a text-to-image generative diffusion model. In this case, we use
ControlNet proposed by Zhang & Agrawala [31] which applies conditional
controls using inputs such as edge maps, segmentation maps, and keypoints to
pretrained large diffusion models to support additional input conditions other
than prompt text. The generated image by the diffusion model conforms to the
edge map to a large extent but transforms the prosthetic limbs into able-
bodied limbs. The generated image is then passed to a standard pose estimation
model, for example, OpenPose, trained on large human pose estimation datasets
to output the pixel locations of body keypoints. The frames were rescaled to
512 x 512 to adhere to the default ControlNet configuration.

Figure 1: (A) The typical workflow when using pose estimation such as
OpenPose. This is also a case example where the default OpenPose fails to
adequately identify keypoints on the prosthetic limb.(B) The proposed zero-
shot workflow involves first acquiring the edge map of the original image and
subsequently generating a synthetic image, which can then be utilized for pose
estimation purposes.

###  2.3 Edge map conditioned diffusion model

Image generative diffusion models are currently state-of-the-art models for
generating synthetic images. They are inspired by the diffusion process where
molecules move from high to low-concentration areas, leading to a
homogenization of the distribution over time. In image generation, the
diffusion process is applied to the image pixels, where random noise is added
over a number of time steps until the image is indistinguishable from random
Gaussian noise.

The diffusion process is typically implemented using a sequence of invertible
transformations, such as Gaussian diffusion or Langevin dynamics. During
training, the model learns to reverse the diffusion process given the time
step and the noisy image at that time step. During inference, new images can
be generated by sampling from random Gaussian noise and applying the learnt
reverse diffusion step-wise to obtain a clean generated image.

To generate images conditioned upon other features such as class, text
description, or sample image, the conditioning information is provided to the
diffusion model as an additional input. The conditioning information is
typically encoded into a fixed-length vector representation using a separate
neural network, such as a text encoder, which is then used as an input to the
diffusion model.

During training, the diffusion model is trained to maximize the log-likelihood
of the observed data given the conditioning information. During inference, the
conditioned diffusion model is used to generate images by sampling from the
diffusion process given the conditioning information. The conditioning
information is used to initialize the diffusion process and to guide the
sampling process at each time step.

Overall, conditioning diffusion models for image generation involves modifying
the training and inference procedures to incorporate the conditioning
information, and training a separate network to encode the conditioning
information into a fixed-length vector that can be used as input to the
diffusion model.

ControlNet is a neural network architecture proposed by [31] that applies
task-specific controls to large image diffusion models such as Stable
Diffusion. ControlNet does so by changing the inputs of diffusion network
blocks. ControlNet makes a copy of the network block and applies a 1x1
convolution layer before and after the block. The corresponding diffusion
model network block weights are frozen during training to avoid overfitting to
the smaller conditional training set and preserve the model’s pre-trained
high-quality image generation capabilities. The control condition is fed to
the first 1x1 convolution layer and the output of that layer is added to the
original input of the network block. This modified input is then passed
through the network block and the subsequent 1x1 convolution layer. The
resulting output is added to the original output of the network block [31].

On the Stable Diffusion model, which follows a U-Net architecture, ControlNet
is implemented to control each level of the U-Net. It creates trainable copies
of the 12 encoding blocks and 1 middle block of Stable Diffusion. The outputs
of the ControlNet are added to the 12 skip-connections and 1 middle block of
the U-Net. We refer interested readers to the detailed explanation in the
original paper.

ControlNets have been trained to perform tasks based on various conditions,
this includes generating images that are controlled by Canny edges, Hough
lines, HED edges, human pose keypoints, segmentation maps, depth maps and
more.

Specifically relevant to this application, the Canny edge conditioned
ControlNet was trained with edge maps that were obtained by processing 3M
images (with captions) using the Canny edge detector. The positive prompts
used to generate images for this study were, “an able-body person walking,
intact lower limbs, 2 legs, full-body portrait, realistic”, while the negative
prompts were, “cyborg, amputee, panfuturism”. The code for ControlNet is
created by [31] and available at https://github.com/lllyasviel/ControlNet.

###  2.4 Pretrained pose estimation model

Human pose estimation models aim to estimate the pose of humans in 2D images,
typically by detecting body keypoints. Deep learning has led to significant
advances in human pose estimation, and models such as OpenPose and AlphaPose,
which have been trained on large datasets (COCO human keypoints dataset),
demonstrated good performance on various benchmarks. Several studies also
investigated the accuracy of these pretrained pose estimation models for
clinical applications including gait analysis. Here we introduce OpenPose in
more detail, which is the widely used model in pose estimation for clinical
and biomechanics studies [34, 28, 35, 36].

The OpenPose model uses a multi-stage convolutional neural network (CNN)
architecture to process images and estimate human poses [24]. The keypoint
detection is performed in two stages: the first stage generates a coarse
heatmap of body keypoints and parts affinity field, and the second stage
refines the heatmap to produce a more accurate estimate of the keypoint
locations. The model then uses the key point heatmap and parts affinity field
to link up the keypoints into the pose skeleton. The model can detect various
body keypoints, including those of the head, neck, shoulders, elbows, wrists,
hips, knees, and ankles. OpenPose is designed to work in real-time and is
capable of estimating human poses in images or videos with multiple people,
even in complex scenes with occlusions and overlapping body parts. It is also
able to estimate the pose of people in different orientations and viewpoints.

Another pose estimation model used in this study was DeepLabCut (DLC) version
2.3.0. DLC allows users to label and train with their own key points. Its
architecture consists of a residual neural network (ResNet-50) with deep
convolutional and deconvolutional neural network layers to predict the
keypoints using feature detection. We train a DLC model using videos from the
transtibial amputee. Ground truth labels for 8 lower-body keypoints (left and
right of hip, knee, ankle and toes) were obtained by manual annotation of
every frame for one set of the transtibial videos. Using the labels of these 4
videos as ground truth, we evaluated the accuracy of the custom model (using
DLC) trained on 20 labelled frames until the training loss had plateaued.

The data, expressed in mean (and standard deviation in parenthesis), are
reported in pixels due to unknown measurements and the lack of a calibration
procedure in the video. Results from our preliminary study were deemed
satisfactory which showed that the custom model had a mean absolute error
(MAE) of 7.89 $\pm$ 3.11 pixels for the coordinates (Supplementary Table 5)
and 1.61∘ $\pm$ 1\. 19∘ for kinematics when evaluated on one set of
transtibial videos (Supplementary Table 6). Therefore, a custom model
following the same steps was taken to obtain the ground truth coordinates for
the other 12 videos. The resulting keypoints were visually examined frame by
frame to ensure accuracy before we effectively treated as manual labels.

###  2.5 Error quantification on keypoints coordinates

In certain frames, the visibility of keypoints may be occluded leading to
lower likelihood scores given by OpenPose. Consequently, frames with a
likelihood below 0.50 were removed. Cubic spline interpolation was applied to
address the resulting gaps in the data, and a low-pass Butterworth filter (4th
order, 6 Hz) was subsequently applied to attenuate high-frequency noise. These
procedures have also been used in previous studies [37, 26].

###  2.6 Error quantification on joint kinematics

Assuming that the left and right camera views are positioned perpendicularly
relative to the sagittal plane, lower-limb joint angles of the limb closest to
the camera can be calculated using the keypoints coordinates via inverse
kinematics [10, 28, 26, 27]. The hip angle was calculated involving the
keypoint vectors of the hip, knee and a relative vertical line to the hip.
Similarly, the knee angle was computed by using the keypoint vectors of the
hip, knee and ankle. In both cases, a positive value indicates flexion while a
negative value indicates extension. The ankle angle was calculated using the
keypoints vectors of the knee, ankle and toe. In addition, the ankle angle is
defined by the foot with respect to a 90∘ line to the tibia. The kinematic
data for the entire time-series was initially calculated, followed by time
normalization from 0-100% of the gait cycles. Gait cycles were defined as
consecutive occurrences of heel strikes by the same foot, which were manually
identified. There were at least 4 gait cycles observed for all videos.

To quantify the error of pose estimation, we use the MAE which is the mean
Euclidean distance between ground truth and predicted keypoints. Both
coordinates and kinematics data were quantified using MAE. There are several
previous works in the literature that quantify the accuracy of markerless pose
estimation models such as OpenPose for the purpose of gait analysis.
Supplementary Table 4 summarizes existing data reported on the accuracy of
clinical gait applications of markerless pose estimation on able-bodied
individuals. Original values reported are used if provided in the papers,
otherwise, the values were estimated by reading off from figures.

##  3 Results

The experimental findings demonstrate significant improvement in pose
estimation accuracy achieved by our zero-shot method when compared to the
established pose estimation software, OpenPose. This is a crucial improvement
needed for practical gait analysis, such as comparing joint angles through the
gait cycle, that is of interest to clinicians. Additionally, we also discern
and quantify variations in pose estimation performance across different
prosthetic types.

###  3.1 Obvious misidentification in OpenPose resolved by zero-shot method

Our findings reveal that when OpenPose is directly applied to the original
image, a substantial proportion of frames exhibit either an inability to
identify or a conspicuous misidentification of lower-limb keypoints (Fig. 2).
Supplementary Table 1 provides a detailed breakdown of these occurrences.
Specifically, transfemoral images display a higher incidence of such
challenges compared to their transtibial counterparts. Notably, our zero-shot
method demonstrates remarkable efficacy in mitigating the number of affected
frames across all camera perspectives. It is also important to note that the
majority of observed keypoint challenges are concentrated in the sagittal
camera view, particularly when the prosthetic limb is in closer proximity to
the camera.

Figure 2: Values are expressed as percentages based on a total of 180 frames.
Each graph shows the percentage of frames where lower-limb keypoints were not
identified fully. Each colour represents the different camera positions.

###  3.2 Substantial improvement in accuracy of quantitative measures using
zero-shot method

The OpenPose method exhibited coordinate errors that were about twice as large
(Table 1) on the prosthetic limb (right) compared to the intact limb (left). A
detailed breakdown of the individual keypoints coordinates errors can be found
in Supplementary Table 2.

The zero-shot method consistently outperformed the OpenPose method in
estimating keypoints on the prosthetic limb by a large margin for joint
coordinates. The mean absolute error (MAE) measured for joint coordinates
decreases by 37% for transtibial prosthetic limbs and 76% for transfemoral
prosthetic limbs. Even though the OpenPose method yielded slightly lower
coordinate errors on the intact limb, the differences observed were relatively
minor, with a mean difference of less than 2 pixels.

Similarly for joint kinematics, the errors against ground truth using the
zero-shot method consistently outperformed the OpenPose method on the
prosthetic limb by a large margin. The improvement is particularly noteworthy
for ankle angles and transfemoral knee angles. The comparison of joint
kinematics is shown in Fig. 4.

Table 1: The overall mean absolute error (MAE) and standard deviations (SD), in parentheses of keypoint coordinates error (MAE) after processing using the default OpenPose and zero-shot method. For all videos, the right leg is the prosthetic limb while the left leg is the intact limb. Bold font indicates the lower value between OpenPose and the Zero-shot method. Metrics (Units) | Amputation Type | Leg | OpenPose (MAE) | Zero-shot (MAE)
---|---|---|---|---
Joint Coordinates (pixels) | Transfemoral | Prosthetic | >100 (>100) | 23.7 (35.61)
Intact | 10.22 (5.36) | 11.96 (7.58)
Transtibial | Prosthetic | 24.07 (15.74) | 15.18 (6.66)
Intact | 12.01 (3.54) | 12.27 (7.60)
Joint Kinematics (∘) | Transfemoral | Prosthetic | 29.64 (26.07) | 6.66 (6.50)
Intact | 5.71 (4.77) | 5.23 (5.10)
Transtibial | Prosthetic | 10.23 (9.08) | 6.32 (4.49)
Intact | 2.90 (2.11) | 3.25 (2.54)
Figure 3: Normalized joint angle kinematics for (A) transfemoral and (B)
transtibial amputee, where each row represents the hip, knee and ankle angle.
Each colour represents the pose estimation used: ground truth (green),
OpenPose (blue) and the zero-shot method (red). Positive hip and knee joint
angles indicate flexion, negative angles indicate extension. Positive ankle
joint angles indicate dorsiflexion, and negative angles indicate
plantarflexion.

Although the open-source videos lacked explicit calibration or measurement
information, an estimate of the subject’s height enabled us to approximate the
metric error. Assuming a subject height of approximately 180 cm, we deduced
that 1 pixel corresponded to approximately 0.3 cm. These assumptions enabled
us to draw comparisons with previous findings. Our results for OpenPose on
subjects with intact limbs are consistent with existing literature
(Supplementary Table 4). Clinicians aiming to replicate our study may consider
incorporating calibration steps at the beginning of the video recording. This
would allow them to gather additional gait parameters, such as stride length
and step width, which were not analyzed in our study due to the utilization of
open-source videos.

###  3.3 Comparison of performance between different prosthetic types

The two main types of lower-limb prosthetics, transfemoral and transtibial
differ in appearance and in characteristics of gait. The performance of pose
estimation and gait analysis on transfemoral and transtibial amputees are
compared.

As observed in Table 1, when comparing transfemoral and transtibial amputees,
the pose estimation of the prosthetic limb exhibited poorer performance in the
transfemoral amputee. Even with the application of the zero-shot method, the
coordinate errors on the prosthetic limb were twice as substantial in the
transfemoral amputee. Whereas in the transtibial amputee, the zero-shot method
was able to reduce the coordinate errors on the prosthetic limb to a
comparable level as the intact limb.

The hip angle curve exhibited satisfactory results for both transtibial and
transfemoral amputees when employing both the OpenPose and zero-shot methods.
Particularly, the zero-shot method displayed the ability to reduce the MAE on
the prosthetic limb for the transfemoral condition by about 3 degrees.

In the knee angle, the OpenPose method was able to replicate the kinematic
curvature throughout the gait cycle in the transtibial amputee (Fig. 3) but
less so in the transfemoral amputee (Fig. 3). Thereafter, the application of
the zero-shot method resulted in an improvement in replicating the knee angle
kinematic throughout the gait cycle for the transfemoral amputee (Fig. 3).

In comparison to the hip and knee angles measured using OpenPose, the ankle
angle exhibited higher error values. This observation is further supported by
the higher error in ankle keypoint coordinate (Supplementary Table 3). When
the zero-shot method was applied, a slight improvement in replicating the
ankle kinematic curve for the transtibial amputee was observed, but no
improvement was evident for the transfemoral amputee.

###  3.4 Gait anomalies in prosthetic limb revealed by zero-shot method

Previous studies have indicated significant kinematic differences between
prosthetic and intact limbs in lower-limb amputees [38]. Notably, the
kinematic gait cycle observed in our study exhibited a similar pattern to
those reported in previous studies [39, 2, 40].

For instance, as seen in Fig. 4, our zero-shot approach revealed that the hip
extension of the transtibial amputee was reduced. Additionally, within the
10-50% range of the gait cycle, a decreased range of motion was observed in
the knee angle. In the case of the transfemoral amputee as seen in Fig. 5, a
noticeable phase shift in both hip and knee angles was observed, indicating a
leftward displacement in the kinematic patterns throughout the gait cycle when
compared to the intact limb’s kinematics. In comparison, OpenPose fails to
capture some of these findings, especially in the transfemoral case due to
greater inaccurate joint angle kinematics. This demonstrates the effectiveness
of the zero-shot method over OpenPose for identifying and quantifying key
anomalies during gait analysis and provides the means for clinicians to
uncover and measure key gait markers during rehabilitation.

Figure 4: Normalized kinematics results for transtibial, where the top and
bottom row represent hip, knee and ankle kinematics, respectively, on the left
(blue; intact) and right (red; prosthetic) limb. (A) Kinematics comparing the
ground truth (solid line) and OpenPose (dotted line). (B) Kinematics comparing
the ground truth (solid line) and zero-shot method (dotted line). Positive hip
and knee joint angles indicate flexion, negative angles indicate extension.
Positive ankle joint angles indicate dorsiflexion, negative angles indicate
plantarflexion. Some key differences in kinematics between the prosthetic and
intact limbs are marked with arrows Figure 5: Normalized kinematics results
for transfemoral, where the top and bottom row represent hip, knee and ankle
kinematics, respectively, on the left (blue; intact) and right (red;
prosthetic) limb. (A) Kinematics comparing the ground truth (solid line) and
OpenPose (dotted line). (B) Kinematics comparing the ground truth (solid line)
and zero-shot method (dotted line). Positive hip and knee joint angles
indicate flexion, negative angles indicate extension. Positive ankle joint
angles indicate dorsiflexion, negative angles indicate plantarflexion. Some
key differences in kinematics between the prosthetic and intact limbs are
marked by arrows

##  4 Discussion

###  4.1 Improved pose estimation accuracy enables quantitative analysis of
gait biomechanics in lower-limb amputees

Gait analysis for lower-limb prosthetic users without the need for specialized
equipment has the potential to transform rehabilitation for this population,
by providing an automated, consistent and quantitative assessment of the
presence and severity of pathological gait. This opens up the possibility of
rapid clinical feedback, reliable longitudinal tracking of progress, objective
comparison between rehabilitation strategies and more. The method proposed in
this work is a step towards this goal through accurate pose estimation from
markerless videos acquired from low-cost commercial cameras.

The results obtained in our investigation demonstrate the effectiveness of the
zero-shot method in reducing both the coordinates and kinematics errors when
compared to applying OpenPose on the original images. Although the ankle
kinematics exhibited a relatively higher error value, the hip and knee
kinematics generated using the zero-shot method remain valuable for
clinicians. Such kinematic information enables clinicians to discern
disparities in kinematic patterns between intact and prosthetic limbs, thereby
facilitating the planning and monitoring of individualized rehabilitation
programs. Presently, it is evident that OpenPose encounters difficulties in
accurately detecting images featuring prosthetic limbs, with higher error
rates observed for prosthetic limbs compared to intact limbs (Fig. 2). This
observation aligns with the study by Cimorelli et al.[29], who similarly
reported challenges when utilizing OpenPose for lower-limb amputees,
consequently restricting the applicability of markerless motion capture in
this population.

Our findings demonstrate that the implementation of the zero-shot method leads
to enhanced keypoint detection by greatly reducing the number of problematic
frames, subsequently reducing coordinate and kinematic errors. This
improvement has practical implications for gait analysis, particularly in
deriving kinematic angles throughout the gait cycle. This information is
crucial for clinicians to comprehend the gait symmetry of individuals. From a
clinical standpoint, the analysis of kinematics throughout the gait cycle
provides the ability to discern asymmetrical walking patterns between the
prosthetic and intact limb. When compared to OpenPose, we have shown the zero-
shot method to be more effective in identifying and quantifying significant
anomalies during gait analysis. By incorporating the identified anomalies and
utilizing the insights gained from the zero-shot method, the clinician can
design an individualized rehabilitation regimen that addresses specific gait
deviations and promotes the restoration of balanced walking patterns. This
personalized approach enhances the potential for successful rehabilitation
outcomes in the lower-limb amputees’ population.

###  4.2 Limitation, possible solutions and future work

Figure 6: (A) In the original image, the left limb is positioned in front.
However, upon generating the synthetic image, the left leg was positioned
backwards, and this discrepancy persisted after the application of OpenPose,
where the dark blue line represents the left leg. (B) Case example where the
generation of the synthetic image fails to change the prosthetic limb to an
intact limb. (C). Case example where the generation of the synthetic image
successfully changes the prosthetic limb to an intact limb.

One drawback in the application of the zero-shot method is the increased
swapping of lower-limb keypoints in the sagittal plane (Fig. 2). This occurs
when the edge map generation process fails to preserve the information
regarding the leg’s proximity to the camera. Humans can perceive the relative
positions of objects from 2D images based on overlapping regions. For
instance, if the left leg is closer to the camera, the left leg will naturally
occlude parts of the right leg that are directly behind when viewing from the
camera. Subsequently, when the edge detector fails to identify the outline of
the leg in the region where the two legs overlap, it will result in an
ambiguous edge map. This ambiguity may potentially result in a swap between
the two legs in the synthetic image. Fig. 6 provides an illustrative example
of the observed phenomenon. Nevertheless, this issue can be readily resolved
by manually identifying and correcting the swapped frames or by utilizing
other cues, such as the asynchronous movement of the ipsilateral arm and leg
swing. This observation is reinforced by the finding that, although the zero-
shot method requires a higher number of frames to be swapped compared to
OpenPose, it still leads to a decrease in error. It is worth highlighting that
the default OpenPose on the original image may also cause the lower-limb
keypoints to swap, similar to previous studies [28, 41]. Moreover, the zero-
shot method resulted in the reduction of frames where keypoints were not
detected in OpenPose due to the non-resemblance of intact limbs. This further
demonstrates the efficacy of the zero-shot method in converting the prosthetic
limb to a human-like representation, thereby allowing the application of pose
estimation.

It is important to acknowledge that not all frames can be successfully
transformed into images resembling able-bodied individuals (Fig. 6B). This
indicates certain limitations with the edge map. Nevertheless, even when
comparing frames at approximately the same gait cycle, the generation of
synthetic images yields inconsistent results (Fig. 6C). This inconsistency is
more pronounced in the case of transfemoral amputees, likely attributed to the
dissimilarity between the prosthetic limb and a human limb, posing challenges
for the model in identifying the appropriate image type. Despite inputs of
positive prompts into ControlNet, emphasizing the presence of two intact
limbs, the model may still encounter difficulties. However, it is worth noting
that the majority of frames can still be successfully transformed, as
evidenced by the overall reduction in coordinate error, especially for the
prosthetic limb. Another limitation pertains to the wide range of prosthetic
options available in the market. Our testing of the diffusion model was
limited to a specific subset of prosthetic users, which may introduce some
variability in the results. Nevertheless, we anticipate that any deviations
from our findings will be minor and not substantially different.

The main advantage of the proposed method is its ability to perform zero-shot
pose estimation for prosthetic users without the need for any manual labelling
and model training. However, the zero-shot ability comes with the limitations
of slow processing speed due to the need to run inference on the large image
generative diffusion model. Thus, there is a trade-off between inference time
versus the time required for manual labelling for prosthetic joints and
finetuning or training a custom detection model. With personal computing
hardware (Nvidia GTX Titan X), the speed of generating 512 x 512 synthetic
images is approximately 8 - 10 seconds per image which presents the main
bottleneck for the workflow. This means for a 24 FPS video, the inference time
needed is 200x that of the video length. Given a walking speed of 1 m/s, a 10
m walk test may require 10 seconds of video recording and may take more than
30 minutes to process which precludes real-time analysis and presents
limitations for practical clinical usage.

To address this, we propose 2 methods to speed up inference. First, clinicians
may run pose estimation on the original video to identify poorly estimated
frames before employing the image-generative zero-shot workflow on the poorly
estimated frames only. Second, clinicians may drop selected frames to reduce
the number of images to be generated. For the purpose of gait analysis, the
gait cycle is obtained by averaging over many gaits. In such use cases, it is
possible to remove selected frames and perform interpolation without affecting
the resultant gait cycle significantly. Nevertheless, the zero-shot method is
still faster than creating a custom-trained image set which includes manual
annotations and a longer training time for custom pose estimation models for
each individual.

Another potential alternative for regenerating an intact limb image for lower-
limb amputees is inpainting a masked section of the image with diffusion
models [42]. This technique involves selectively highlighting specific parts
of an image that require modification. However, this option was not explored
in our study due to the continuous movement of the limbs in each image, which
necessitates manual masking and thereby increases post-processing time. Future
research could investigate alternative methods for automating the masking of
the prosthetic limb in each frame. By generating images for smaller masked
regions, it may be possible to expedite the inference time.

##  5 Conclusion

The proposed zero-shot method presents a novel approach to applying markerless
pose estimation techniques in the context of lower-limb amputees. Our results
demonstrate that this approach improves the detection of keypoints on
prosthetic limbs, thereby enabling clinicians to gain valuable insights into
the subject’s kinematics throughout the gait cycle. Although it is important
to acknowledge that the method is not flawless, its successful implementation
showcases the proof-of-concept for such techniques. Future investigations can
focus on streamlining the workflow to enable real-time analysis, thereby
enhancing its practical utility in clinical settings.

## References

  * [1] C. M. Powers, S. Rao, J. Perry, Knee kinetics in trans-tibial amputee gait, Gait & Posture 8 (1) (1998) 1–7.
  * [2] A. Esquenazi, Gait Analysis in Lower-Limb Amputation and Prosthetic Rehabilitation 25 (1) 153–167.  doi:10.1016/j.pmr.2013.09.006.
URL https://linkinghub.elsevier.com/retrieve/pii/S1047965113000739

  * [3] L. Yang, P. Dyer, R. Carson, J. Webster, K. B. Foreman, S. Bamberg, Utilization of a lower extremity ambulatory feedback system to reduce gait asymmetry in transtibial amputation gait, Gait & posture 36 (3) (2012) 631–634.
  * [4] C. T. Huang, J. R. Jackson, N. B. Moore, P. R. Fine, K. V. Kuhlemeier, G. H. Traugh, P. T. Saunders, Amputation: Energy cost of ambulation 60 (1) 18–24.  arXiv:420566.
  * [5] G. H. Traugh, P. J. Corcoran, R. L. Reyes, Energy expenditure of ambulation in patients with above-knee amputations 56 (2) 67–71.  arXiv:1124978.
  * [6] R. Gailey, K. Allen, J. Castles, J. Kucharick, M. Roeder, Review of secondary physical conditions associated with lower-limb, Journal of Rehabilitation Research & Development 45 (1-4) (2008) 15–30.
  * [7] Y. Celik, S. Stuart, W. Woo, A. Godfrey, Gait analysis in neurological populations: Progression in the use of wearables 87 9–29.  doi:10.1016/j.medengphy.2020.11.005.
URL https://linkinghub.elsevier.com/retrieve/pii/S1350453320301697

  * [8] I. K. Jalata, T.-D. Truong, J. L. Allen, H.-S. Seo, K. Luu, Movement Analysis for Neurological and Musculoskeletal Disorders Using Graph Convolutional Neural Network 13 (8) 194.  doi:10.3390/fi13080194.
URL https://www.mdpi.com/1999-5903/13/8/194

  * [9] L. Lonini, Y. Moon, K. Embry, R. J. Cotton, K. McKenzie, S. Jenz, A. Jayaraman, Video-Based Pose Estimation for Gait Analysis in Stroke Survivors during Clinical Assessments: A Proof-of-Concept Study 6 (1) 9–18.  doi:10.1159/000520732.
URL https://www.karger.com/Article/FullText/520732

  * [10] M. Moro, G. Marchesi, F. Odone, M. Casadio, Markerless gait analysis in stroke survivors based on computer vision and deep learning: A pilot study, in: Proceedings of the 35th Annual ACM Symposium on Applied Computing, ACM, pp. 2097–2104.  doi:10.1145/3341105.3373963.
URL https://dl.acm.org/doi/10.1145/3341105.3373963

  * [11] C. Ridao-Fernández, E. Pinero-Pinto, G. Chamorro-Moriana, Observational Gait Assessment Scales in Patients with Walking Disorders: Systematic Review 2019 1–12.  doi:10.1155/2019/2085039.
URL https://www.hindawi.com/journals/bmri/2019/2085039/

  * [12] M. J. Highsmith, C. R. Andrews, C. Millman, A. Fuller, J. T. Kahle, T. D. Klenow, K. L. Lewis, R. C. Bradley, J. J. Orriola, Gait training interventions for lower extremity amputees: a systematic literature review, Technology & Innovation 18 (2-3) (2016) 99–113.
  * [13] S. J. Hillman, S. C. Donald, J. Herman, E. McCurrach, A. McGarry, A. M. Richardson, J. E. Robb, Repeatability of a new observational gait score for unilateral lower limb amputees 32 (1) 39–45.  doi:10.1016/j.gaitpost.2010.03.007.
URL https://linkinghub.elsevier.com/retrieve/pii/S0966636210000779

  * [14] M. Saleh, G. Murdoch, In defence of gait analysis. observation and measurement in gait assessment, The Journal of bone and joint surgery. British volume 67 (2) (1985) 237–241.
  * [15] J. M. Wilken, R. Marin, Gait analysis and training of people with limb loss, Care of the Combat Amputee (2009) 535–52.
  * [16] D. J. Sanderson, P. E. Martin, Lower extremity kinematic and kinetic adaptations in unilateral below-knee amputees during walking, Gait & Posture 6 (2) (1997) 126–136.
  * [17] C. Sjödahl, G.-B. Jarnlo, B. Söderberg, B. Persson, Kinematic and kinetic gait analysis in the sagittal plane of trans-femoral amputees before and after special gait re-education, Prosthetics and orthotics international 26 (2) (2002) 101–112.
  * [18] S. Blair, M. J. Lake, R. Ding, T. Sterzing, Magnitude and variability of gait characteristics when walking on an irregular surface at different speeds 59 112–120.  doi:10.1016/j.humov.2018.04.003.
URL https://linkinghub.elsevier.com/retrieve/pii/S0167945717305407

  * [19] B. Van Hooren, J. T. Fuller, J. D. Buckley, J. R. Miller, K. Sewell, G. Rao, C. Barton, C. Bishop, R. W. Willy, Is motorized treadmill running biomechanically comparable to overground running? a systematic review and meta-analysis of cross-over studies, Sports medicine 50 (2020) 785–813.
  * [20] M. G. Bernal-Torres, H. I. Medellín-Castillo, J. C. Arellano-González, Development of a new low-cost computer vision system for human gait analysis: A case study 095441192311636doi:10.1177/09544119231163634.
URL http://journals.sagepub.com/doi/10.1177/09544119231163634

  * [21] Y. Chen, Y. Tian, M. He, Monocular human pose estimation: A survey of deep learning-based methods 192 102897.  doi:10.1016/j.cviu.2019.102897.
URL https://linkinghub.elsevier.com/retrieve/pii/S1077314219301778

  * [22] E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, B. Schiele, DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Model.  arXiv:1605.03170.
URL http://arxiv.org/abs/1605.03170

  * [23] A. Mathis, S. Schneider, J. Lauer, M. W. Mathis, A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives 108 (1) 44–65.  doi:10.1016/j.neuron.2020.09.017.
URL https://linkinghub.elsevier.com/retrieve/pii/S0896627320307170

  * [24] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, Y. Sheikh, OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields 43 (1) 172–186.  doi:10.1109/TPAMI.2019.2929257.
URL https://ieeexplore.ieee.org/document/8765346/

  * [25] S. D. Uhlrich, A. Falisse, L. Kidzinski, J. Muccini, M. Ko, A. S. Chaudhari, J. L. Hicks, S. L. Delp, OpenCap: 3D human movement dynamics from smartphone videos.  doi:10.1101/2022.07.07.499061.
URL http://biorxiv.org/lookup/doi/10.1101/2022.07.07.499061

  * [26] B. Van Hooren, N. Pecasse, K. Meijer, J. M. N. Essers, The accuracy of markerless motion capture combined with computer vision techniques for measuring running kinematics sms.14319doi:10.1111/sms.14319.
URL https://onlinelibrary.wiley.com/doi/10.1111/sms.14319

  * [27] E. P. Washabaugh, T. A. Shanmugam, R. Ranganathan, C. Krishnan, Comparing the accuracy of open-source pose estimation methods for measuring gait kinematics 97 188–195.  doi:10.1016/j.gaitpost.2022.08.008.
URL https://linkinghub.elsevier.com/retrieve/pii/S0966636222004738

  * [28] J. Stenum, C. Rossi, R. T. Roemmich, Two-dimensional video-based analysis of human gait using pose estimation 17 (4) e1008935.  doi:10.1371/journal.pcbi.1008935.
URL https://dx.plos.org/10.1371/journal.pcbi.1008935

  * [29] A. Cimorelli, A. Patel, T. Karakostas, R. J. Cotton, Portable in-clinic video-based gait analysis: Validation study on prosthetic users.  doi:10.1101/2022.11.10.22282089.
URL http://medrxiv.org/lookup/doi/10.1101/2022.11.10.22282089

  * [30] A. Mathis, P. Mamidanna, K. M. Cury, T. Abe, V. N. Murthy, M. W. Mathis, M. Bethge, DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning 21 (9) 1281–1289.  doi:10.1038/s41593-018-0209-y.
URL https://www.nature.com/articles/s41593-018-0209-y

  * [31] L. Zhang, M. Agrawala, Adding Conditional Control to Text-to-Image Diffusion Models, preprint at https://arxiv.org/abs/2302.05543.  doi:10.48550/ARXIV.2302.05543.
  * [32] Mission gait.
URL https://www.youtube.com/@MissionGait

  * [33] J. Gardiner, N. Gunarathne, D. Howard, L. Kenney, Crowd-Sourced Amputee Gait Data: A Feasibility Study Using YouTube Videos of Unilateral Trans-Femoral Gait 11 (10) e0165287.  doi:10.1371/journal.pone.0165287.
URL https://dx.plos.org/10.1371/journal.pone.0165287

  * [34] N. Nakano, T. Sakura, K. Ueda, L. Omura, A. Kimura, Y. Iino, S. Fukashiro, S. Yoshioka, Evaluation of 3D Markerless Motion Capture Accuracy Using OpenPose With Multiple Video Cameras 2 50.  doi:10.3389/fspor.2020.00050.
URL https://www.frontiersin.org/article/10.3389/fspor.2020.00050/full

  * [35] K. Sato, Y. Nagashima, T. Mano, A. Iwata, T. Toda, Quantifying normal and parkinsonian gait features from home movies: Practical application of a deep learning–based 2D pose estimator 14 (11) e0223549.  doi:10.1371/journal.pone.0223549.
URL https://dx.plos.org/10.1371/journal.pone.0223549

  * [36] K. Abe, K.-I. Tabei, K. Matsuura, K. Kobayashi, T. Ohkubo, OpenPose-based Gait Analysis System For Parkinson’s Disease Patients From Arm Swing Data, in: 2021 International Conference on Advanced Mechatronic Systems (ICAMechS), IEEE, pp. 61–65.  doi:10.1109/ICAMechS54019.2021.9661562.
URL https://ieeexplore.ieee.org/document/9661562/

  * [37] G. Serrancoli, P. Bogatikov, J. P. Huix, A. F. Barbera, A. J. S. Egea, J. T. Ribe, S. Kanaan-Izquierdo, A. Susin, Marker-Less Monitoring Protocol to Analyze Biomechanical Joint Metrics During Pedaling 8 122782–122790.  doi:10.1109/ACCESS.2020.3006423.
URL https://ieeexplore.ieee.org/document/9131774/

  * [38] T. Varrecchia, M. Serrao, M. Rinaldi, A. Ranavolo, S. Conforto, C. De Marchis, A. Simonetti, I. Poni, S. Castellano, A. Silvetti, A. Tatarelli, L. Fiori, C. Conte, F. Draicchio, Common and specific gait patterns in people with varying anatomical levels of lower limb amputation and different prosthetic components 66 9–21.  doi:10.1016/j.humov.2019.03.008.
URL https://linkinghub.elsevier.com/retrieve/pii/S0167945718307577

  * [39] A. D. Koelewijn, A. J. Van Den Bogert, Joint contact forces can be reduced by improving joint moment symmetry in below-knee amputee gait simulations 49 219–225.  doi:10.1016/j.gaitpost.2016.07.007.
URL https://linkinghub.elsevier.com/retrieve/pii/S0966636216301448

  * [40] H. Bateni, S. J. Olney, Kinematic and kinetic variations of below-knee amputee gait, JPO: Journal of Prosthetics and Orthotics 14 (1) (2002) 2–10.
  * [41] L. Needham, M. Evans, D. P. Cosker, L. Wade, P. M. McGuigan, J. L. Bilzon, S. L. Colyer, The accuracy of several pose estimation methods for 3D joint centre localisation 11 (1) 20673.  doi:10.1038/s41598-021-00212-x.
URL https://www.nature.com/articles/s41598-021-00212-x

  * [42] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, L. Van Gool, Repaint: Inpainting using denoising diffusion probabilistic models, arXiv preprint arXiv:2201.09865 (2022).