# NEW ADVANCES AT THE INTERSECTION OF BRAIN-INSPIRED LEARNING AND DEEP LEARNING IN AUTONOMOUS VEHICLES AND ROBOTICS

EDITED BY : Guang Chen, Pascual Campoy, Changhong Fu and Caixia Cai PUBLISHED IN : Frontiers in Neurorobotics

#### Frontiers eBook Copyright Statement

The copyright in the text of individual articles in this eBook is the property of their respective authors or their respective institutions or funders. The copyright in graphics and images within each article may be subject to copyright of other parties. In both cases this is subject to a license granted to Frontiers. The compilation of articles constituting this eBook is the property of Frontiers.

Each article within this eBook, and the eBook itself, are published under the most recent version of the Creative Commons CC-BY licence. The version current at the date of publication of this eBook is CC-BY 4.0. If the CC-BY licence is updated, the licence granted by Frontiers is automatically updated to the new version.

When exercising any right under the CC-BY licence, Frontiers must be attributed as the original publisher of the article or eBook, as applicable.

Authors have the responsibility of ensuring that any graphics or other materials which are the property of others may be included in the CC-BY licence, but this should be checked before relying on the CC-BY licence to reproduce those materials. Any copyright notices relating to those materials must be complied with.

Copyright and source acknowledgement notices may not be removed and must be displayed in any copy, derivative work or partial copy which includes the elements in question.

All copyright, and all rights therein, are protected by national and international copyright laws. The above represents a summary only. For further information please read Frontiers' Conditions for Website Use and Copyright Statement, and the applicable CC-BY licence.

ISSN 1664-8714 ISBN 978-2-88963-971-7 DOI 10.3389/978-2-88963-971-7

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office<EMAIL_ADDRESS>
# NEW ADVANCES AT THE INTERSECTION OF BRAIN-INSPIRED LEARNING AND DEEP LEARNING IN AUTONOMOUS VEHICLES AND ROBOTICS

Topic Editors: Guang Chen, Tongji University, China Pascual Campoy, Polytechnic University of Madrid, Spain Changhong Fu, Tongji University, China Caixia Cai, Agency for Science, Technology and Research (A\*STAR), Singapore

Citation: Chen, G., Campoy, P., Fu, C., Cai, C., eds. (2020). New Advances at the Intersection of Brain-Inspired Learning and Deep Learning in Autonomous Vehicles and Robotics. Lausanne: Frontiers Media SA. doi: 10.3389/978-2-88963-971-7

# Table of Contents

*04 Robust Learning Control for Shipborne Manipulator With Fuzzy Neural Network*

Zhiqiang Xu, Wanli Li and Yanran Wang

*15 Mobile Robot Path Planning Based on Ant Colony Algorithm With A\* Heuristic Method*

Xiaolin Dai, Shuai Long, Zhiwen Zhang and Dawei Gong

*24 Neural Network Based Uncertainty Prediction for Autonomous Vehicle Application*

Feihu Zhang, Clara Marina Martinez, Daniel Clarke, Dongpu Cao and Alois Knoll

*41 SVM-Based Classification of sEMG Signals for Upper-Limb Self-Rehabilitation Training*

Siqi Cai, Yan Chen, Shuangyuan Huang, Yan Wu, Haiqing Zheng, Xin Li and Longhan Xie

*51 Toward a Brain-Inspired System: Deep Recurrent Reinforcement Learning for a Simulated Self-Driving Agent*

Jieneng Chen, Jingye Chen, Ruiming Zhang and Xiaobin Hu

*60 Deep Recurrent Neural Networks Based Obstacle Avoidance Control for Redundant Manipulators*

Zhihao Xu, Xuefeng Zhou and Shuai Li

*73 A Novel Model for Arbitration Between Planning and Habitual Control Systems*

Farzaneh Sheikhnezhad Fard and Thomas P. Trappenberg

Florian Mirus, Peter Blouw, Terrence C. Stewart and Jörg Conradt

# Robust Learning Control for Shipborne Manipulator With Fuzzy Neural Network

#### Zhiqiang Xu, Wanli Li\* and Yanran Wang

School of Mechanical Engineering, Tongji University, Shanghai, China

The shipborne manipulator plays an important role in autonomous collaboration between marine vehicles. In real applications, a conventional proportional-derivative (PD) controller is not suitable for the shipborne manipulator to conduct safe and accurate operations under ocean conditions, due to its bad tracing performance. This paper presents a real-time and adaptive control approach for the shipborne manipulator to achieve position control. This novel control approach consists of a conventional PD controller and fuzzy neural network (FNN), which work in parallel to realize PD+FNN control. Qualitative and quantitative tests of simulations and real experiments show that the proposed PD+FNN controller achieves better performance in comparison with the conventional PD controller, in the presence of uncertainty and disturbance. The presented PD+FNN eliminates the requirements for precise tuning of the conventional PD controller under different ocean conditions, as well as an accurate dynamics model of the shipborne manipulator. In addition, it effectively implements a sliding mode control (SMC) theory-based learning algorithm, for fast and robust control, which does not require matrix inversions or partial derivatives. Furthermore, simulation and experimental results show that the angle compensation deviation of the shipborne manipulator can be improved in the range of ±1 ◦ .

#### Edited by:

Caixia Cai, Agency for Science, Technology and Research (A\*STAR), Singapore

#### Reviewed by:

Yingbai Hu, Technische Universität München, Germany Dawei Gong, Juilliard School, United States

#### \*Correspondence:

Wanli Li<EMAIL_ADDRESS>
Received: 14 January 2019 Accepted: 13 March 2019 Published: 04 April 2019

#### Citation:

Xu Z, Li W and Wang Y (2019) Robust Learning Control for Shipborne Manipulator With Fuzzy Neural Network. Front. Neurorobot. 13:11. doi: 10.3389/fnbot.2019.00011 Keywords: shipborne manipulator, real-time adaptive control, conventional PD controller, fuzzy neural network, sliding mode control, experiment verification

## 1. INTRODUCTION

The shipborne manipulator has become the most important tool in achieving autonomous cargo reloading between marine vehicles. With the use of the shipborne manipulator, onboard physical labor can be greatly reduced. However, unpredictable ship motion has a great impact on the maneuverability of the manipulator in real applications due to the complexity of the marine environment. If the sea state reaches Level-4, i.e., the height of a sea wave is larger than 1.52 m and wind speed exceeds 10.8 m/s, arm movement of the manipulator is extremely limited because of the influence of sling inertia and non-linear ship pose variation. In this case, the operational capacity of the manipulator is reduced by more than 50% or the manipulator is even temporarily suspended.

For the shipborne manipulator, changing boom inclination is realized through the expansion and contraction of its amplitude cylinder. The energy-saving and vibration-damping function of its accumulator is of great significance for improving manipulator control. In literature, the cylinder-accumulator in a manipulator has been studied in different types of engineering applications (Xiao et al., 2014; Shen et al., 2015; Zhao et al., 2017; Xia et al., 2018).

Specifically, the accumulator is a key component in the design of hydraulic hybrid structures, which ensures acceptable shock absorption performance and system energy consumption. Its different designs are used in other construction machines such as a rock drill (Yang et al., 2017), Scraper (Junke and Zhen, 2017), and Fast Forging Press (Zhang et al., 2016). However, there are few investigations related to the influence of the accumulator on the valve control system. According to the valve-controlled cylinder-accumulator model, the accumulator is used as an energy-saving and oil damping source in parallel with the rodless cavity of the amplitude cylinder. **Figure 1** shows the details of connecting the accumulator with the cylinder.

In literature, the conventional PD controller is often used to control different types of manipulators (Cervantes and Alvarez-Ramirez, 2001; Alvarez-Ramirez et al., 2003; Su et al., 2010). However, it is not suitable for controlling the hydraulic system discussed in this work, due to its high-order non-linearity, time-varying and hysteresis characteristics. It cannot control the hydraulic system in time and is vulnerable to environmental interference. Additionally, a significant steady-state error still exists, even if plenty of time is used to tune the appropriate

values for the conventional PD controller. In order to control the manipulator, calculating torque is the simplest control approach, but this approach relies on the accurate mechanical model of the system. To overcome this issue, the model-free approach has gained respectable attention since it does not require the precise model of the system and is more robust in response to uncertainty and disturbance.

The fuzzy logic controller (FLC) is widely applied to handle uncertainty and disturbance in many systems (Hasanien and Matar, 2015; Dabbaghjamanesh et al., 2016; Vaidyanathan and Azar, 2016), especially in different kinds of robots (Fu et al., 2013; Tai et al., 2016; Sarabakha et al., 2018). However, the FLC also requires a lot of time in order to tune the proper parameters to achieve a satisfied control performance. Recently, the FLC has been combined with an artificial neural network (ANN), i.e., fuzzy neural network (FNN), to overcome the aforementioned weakness of the FLC. In literature, the FNN has been successfully applied in identification and non-linear system controlling (Lin et al., 2015; Tang et al., 2017; He and Dong, 2018). At the same time, the sliding mode control (SMC) theory-based algorithm has been presented as a faster learning approach for tuning the FNN parameters, due to its faster convergence speed and higher robustness to uncertainty and disturbance (Lin et al., 2014). Moreover, the FNN, trained with the SMC theory-based algorithm, has been successfully used in controlling a spherical rolling robot (Kayacan et al., 2013), a robotic arm (Wai and Muthusamy, 2013), and a gyroscope (Yan et al., 2017).

In this work, an FNN is proposed to work in parallel with a conventional PD controller to achieve a PD+FNN controller. It not only overcomes the original defects of conventional PD control but also significantly enhances self-learning abilities and adaptabilities. In order to validate the proposed control strategy, simulation and experimental tests have been implemented. The qualitative and quantitative results show that the presented strategy is feasible and practical. In addition, it outperforms the conventional PD controller. The main contributions of this work are listed below:

The organization of this paper is as follows: In section 2, the dynamic model of the shipborne manipulator is introduced. In section 3, the PD+FNN control strategy is described. In section 4, different simulation tests are conducted in order to verify the proposed control strategy. In section 5, the real experimental tests on the swaying platform are performed to validate the proposed controller. Finally, conclusions are drawn in section 6.

#### 2. DYNAMIC MODEL

**Table 1** shows the simulation parameters. **Figure 2** shows the flow direction of the oil. The accumulator is linked with the rodless cavity of the cylinder. When the cylinder extends, the accumulator and the valve supply oil to the cylinder simultaneously, making the cylinder stretch out faster. The coupling dynamics model of the parallel accumulator of the valve-controlled cylinder system is established based on the flow continuity equation and the dynamic equation.

#### 2.1. Cylinder Dynamic Equation

The cylinder dynamic equation can be obtained by ignoring the cylinder cavity pressure, which is defined as:

$$pA = m\_l \ddot{\mathbf{x}} + B\_P \dot{\mathbf{x}} + k\_z \mathbf{x} \,, \tag{1}$$

where p denotes the working pressure, N/m2 ; A denotes the action area of rodless cavity, m2 ; mt denotes the mass of the piston, kg; x denotes the displacement of pistol, m; BP denotes the viscosity damping coefficient, N · s/m; kz denotes the spring stiffness, N/m.

#### 2.2. Cylinder Flow Equation

The cylinder flow continuity equation is:

$$Q\_L = A\dot{\mathbf{x}} + \mathbf{C}\_l \mathbf{p} \,, \tag{2}$$

where Ci denotes the internal leakage coefficient, m5 /N · s; QL denotes the total flow into rodless cavity of cylinder, m3 /s, which is defined as:

$$Q\_L = Q\_1 + Q\_X \,, \tag{3}$$

where Q1 denotes the oil flow from valve, m3 /s; QX denotes the oil flow from the accumulator, m3 /s. Accumulator energy release process can be regarded as interference according to Gaussian distribution, i.e.,:

$$Q\_{\overline{X}} = \mathcal{N}\left(\mu\_{\nu}, \sigma\_{\nu}^{2}\right), \tag{4}$$

where µv and σv are the mean value and the standard deviation of the accumulator output flow, respectively.

#### 2.3. Valve Flow Equation

The spool flow is a function of the working pressure and the displacement of the spool. The spool can be viewed as a zeroopen four-way spool valve. The valve flow equation is defined as follows:

$$Q\_1 = C\_d \omega \chi\_\nu \sqrt{\frac{2}{\rho} \left(p\_S - p\_1\right)} = k\_q \chi\_\nu \,, \tag{5}$$

where Cd denotes the flow coefficient of valve; w denotes the valve area gradient, m2 /m; xv denotes the displacement of valve core, m; xv = kv ·µ, kv denotes the spool scale factor; µ denotes current signal; ρ denotes the oil density, kg/m3 ; ps denotes the system pressure, N/m2 ; p1 denotes the pressure of the rodless cavity, N/m2 ; kq denotes the flow gain coefficient, m2 /s, which is defined as:

$$k\_q = C\_d w \sqrt{\frac{2}{\rho} \left(p\_S - p\_1\right)}\,,\tag{6}$$

Assume that the state of the system is x1 = x, x2 = ˙x, the dynamic model of the valve-controlled cylinder system can be defined as follows:

$$\begin{cases}
\dot{\mathbf{x}}\_1 = \mathbf{x}\_2 \\
\dot{\mathbf{x}}\_2 = \theta\_1 \mathbf{x}\_1 + \theta\_2 \mathbf{x}\_2 + \mathbf{g}\mu + d \\
\mathbf{x} = \mathbf{x}\_1
\end{cases},\tag{7}$$

where

$$\begin{cases} \theta\_1 = -\frac{k\_t}{m\_l} \\ \theta\_2 = -\left(\frac{A^2 + C\_i B\_P}{m\_l C\_i}\right) \\ g = \frac{k\_q k\_\vartheta A}{m\_l C\_i} \\ d = \frac{\mathcal{N}(\mu\_\vartheta, \sigma\_\vartheta^2)A}{m\_l C\_i} \end{cases} \tag{8}$$

#### 3. PD+FNN CONTROL STRATEGY

#### 3.1. Overview of Control Strategy

**Figure 3** shows the presented PD+FNN control strategy, in which the conventional PD controller works in parallel with the fuzzyneuro controller, as the FNN block shows in **Figure 3**. The PD controller is utilized to not only trace the target value by system error, i.e., e = xpref − xp, but also to provide learning errors to train the FNN online. The FNN is supposed to improve the control accuracy and offset the effects of system interference.

#### 3.2. Fuzzy Neural Network Construction

The proposed FNN consists of two input signals, i.e., x1 = e and x2 = ˙e, and one output signal xf . Takagi-Sugeno-Kang (TSK) fuzzy model (Lin et al., 2015; Precup et al., 2015) is used in which the antecedent part is the fuzzy set and the consequent part consists of only crisp numbers. The rth rule of a zero-order TSK model with two input variables x1 and x2 can be defined as follows:

$$\text{IF } \mathfrak{x}\_1 \text{ is } M\_{1\dot{t}} \text{ and } \mathfrak{x}\_2 \text{ is } M\_{2\dot{\jmath}}, \text{ THEN } f\_{\ddot{\jmath}} = d\_{\ddot{\imath}\dot{\jmath}}\text{ }, \tag{9}$$

where fij is the time-varying parameter of the consequent part. dij is the coefficient of the output function for the rth rule, and M1i and M2j are fuzzy sets. Therefore, the inputs can be represented as µ1i and µ2j , respectively. The firing strength of the rth rule is computed as the T-norm (multiplication) of the member functions (MFs) in the antecedent part (Imanberdiyev and Kayacan, 2018):

$$W\_{ij} = \mu\_{1i}(\mathbf{x}\_1)\mu\_{2j}(\mathbf{x}\_2)\,,\tag{10}$$

The output signal of the system can be derived using the normalized values of the firing strength W˜ ij with the following form (Biglarbegian et al., 2010):

$$u\_f = \sum\_{i=1}^{I} \sum\_{j=1}^{J} f\_{ij} \tilde{W}\_{ij} \, , \tag{11}$$

where J and I represent the number of MFs for x2 and x1, respectively. W˜ ij is expressed as follows:

$$
\tilde{W}\_{\vec{ij}} = \frac{W\_{\vec{ij}}}{\sum\_{i=1}^{I} \sum\_{j=1}^{J} W\_{\vec{ij}}} \,. \tag{12}
$$

Overall control input u to the system is defined as follows:

$$\mathbf{x}\_{\nu} = \mathbf{x}\_{\mathfrak{c}} - \mathbf{x}\_{\mathfrak{f}} \,, \tag{13}$$

where xc and xf are the control signals produced by the PD controller and the FNN controller, respectively.

#### 3.3. Triangular Fuzzy MFs

In the FNN, the fuzzy MFs play the important role of overcoming environmental interference. These MFs have already shown promising results for control (Khanesar et al., 2015) and identification (Khanesar et al., 2011) purposes. In this work, typical triangular fuzzy MFs are chosen in order to achieve a faster and robust control performance. The MF is defined as follows:

$$\mu(\mathbf{x}) = \begin{cases} \left(1 - \left|\frac{\mathbf{x} - \mathbf{c}}{d}\right|\right) \text{ if } c - d < \mathbf{x} < c + d\\ \mathbf{0} & \text{otherwise} \end{cases},\tag{14}$$

where x is the input, d and c are the width and the center of the MF. The stability proof can be found in Kayacan and Khanesar (2015) according to sliding mode control theory.

## 3.4. Sliding Mode Control Theory-Based Training Approach

In this paper, SMC based parameter update rules are proposed to guarantee the stability of the system and provide favorable robustness. By utilizing the principles of the SMC theory, the zero dynamics of the learning error coordinate xc(t) can be described as a time-varying sliding surface Sc in the following form:

$$S\_{\mathfrak{c}}(\boldsymbol{\chi}\_{f}, \boldsymbol{\chi}\_{\boldsymbol{\nu}}) = \boldsymbol{\chi}\_{\mathfrak{c}}(t) = \boldsymbol{\chi}\_{f}(t) + \boldsymbol{\chi}\_{\boldsymbol{\nu}}(t) = \boldsymbol{0} \,. \tag{15}$$

If this condition is satisfied, the FNN structure is trained to become the non-linear regulator which assists the parallel controller (in our case PD controller), and the desired performance of the system can be obtained. Therefore, the sliding surface for the non-linear system under control is given by

$$\mathcal{S}\_{\mathbb{P}}(e,\dot{e}) = \dot{e} + \chi e \,, \tag{16}$$

with χ > 0 being a positive parameter which defines the desired trajectory of the error signal.

The time-varying parameter of the consequent part ˙ fij is updated based on the following adaptation law (Kayacan and Khanesar, 2015):

$$\dot{f}\_{\hat{i}\hat{j}} = -\frac{\tilde{W}\_{\hat{i}\hat{j}}}{\prod^{T}\prod^{T}}\alpha\text{sign}(\boldsymbol{\chi}\_{\mathcal{E}}),\tag{17}$$

where

$$\prod = \left(\sum\_{i=1}^{I} \sum\_{j=1}^{J} \tilde{W}\_{\vec{v}}\right),\tag{18}$$

the learning rate α > 0 is updated based on the following equation:

$$
\dot{\alpha} = \left| \chi\_{\mathfrak{c}} \right|, \tag{19}
$$

The adaptation law for the premise part is given as follows (Kayacan and Khanesar, 2015):

$$\dot{c}\_{1i} = -\gamma\_1 \left| d\_{1i} \right| \left( 1 - T\_{1i} \right) \text{sgn} \left( \mathbf{x}\_1 - c\_{1i} \right) \times H \left( \mathbf{x}\_1, c\_1 - d\_1, c\_1 + d\_1 \right),$$

$$\dot{d}\_{1i} = -\gamma\_1 \frac{(1 - T\_{1i}) \left| d\_{1i} \right|}{\left| \chi\_1 - c\_{1i} \right|} \text{sgn} \left( d\_{1i} \right) \times H \left( \chi\_1, c\_1 - d\_1, c\_1 + d\_1 \right), \tag{21}$$

$$\dot{c}\_{2i} = -\gamma\_1 \left| d\_{2j} \right| \left( 1 - T\_{2j} \right) \text{sgn} \left( \mathbf{x}\_2 - c\_{2i} \right) \times H \left( \mathbf{x}\_2, c\_2 - d\_2, c\_2 + d\_2 \right), \tag{22}$$

$$\dot{d}\_{2j} = -\gamma\_1 \frac{\left(1 - T\_{2j}\right) \left|d\_{2j}\right|}{\left|\mathbf{x}\_1 - c\_{2j}\right|} \text{sgn}\left(d\_{2j}\right) \times H\left(\mathbf{x}\_2, c\_2 - d\_2, c\_2 + d\_2\right),\tag{23}$$

where

$$H(\mathbf{x}, \mathbf{c}, d) = \begin{cases} \mathbf{x} \text{ if } \mathbf{c} - d < \mathbf{x} < \mathbf{c} + d\\ \mathbf{0} \text{ otherwise} \end{cases},\tag{24}$$

$$\begin{cases} T\_{1,i} = \begin{vmatrix} \frac{x\_1 - c\_i}{d\_i} \end{vmatrix} \\ T\_{2,i} = \begin{vmatrix} \frac{x\_2 - c\_i}{d\_i} \end{vmatrix} \end{cases} \tag{25}$$

For the γ1, it needs to be selected as positive (Kayacan and Khanesar, 2015).

#### 4. SIMULATION AND RESULTS ANALYSIS

#### 4.1. Simulation Parameter

The control gains for the PD controller are chosen as follows: kp = 10, kd = 5.

#### 4.2. Simulation Results

**Table 1** shows the simulation parameters. **Figures 4**–**7** shows the simulation results without noise. The adaptive learning capabilities of the PD+FNN structure can provide superior performance in different conditions. It is able to solve limitations such as the lack of modeling and existing uncertainties in the environment and is therefore more suitable for real-time applications.

As seen from **Figure 4**, the PD+FNN controller has a faster and more stable response performance. **Figure 5** shows that the controller has a better adaptive learning property to lessen the

error gradually. As shown in **Figure 6**, although the PD controller ensures the error signal is bounded in the neighborhood of zero, significant steady-state errors that occur from internal or external interferences cannot be eliminated. Compared to the PD controller, the PD+FNN controller eliminates the steady-state error.

**Figure 7** shows curves of the overall signal (which is defined as xv = xc − xf ), the output of FNN (xf ), and the output of the conventional PD controller (xc). As seen in **Figure 7**, the overall control signal is close to the conventional PD controller at the beginning of the simulation, then the FNN learns the dynamics of the system and takes responsibility for the system. Simultaneously, the output of the PD controller tends to go to zero.

As discussed in the section 2.2, the accumulator energy release process can be regarded as interference according to Gaussian distribution. In order to create different noise levels, four different mean values of the accumulator output flow are chosen, i.e., µv = {100, 150, 200, 300} [L/min]. For the standard deviation, it is selected as σv = 70 L/min.

**Figure 8** shows the manipulator position errors under PD controller with the noise level from 0.12 to 0.24 m.

**Figure 8** shows that the PD controller cannot handle steady-state errors that occur from internal or external interferences. However, the PD+FNN controller can eliminate steady-state errors through an adaptive learning algorithm, and can be used in a real time control to cope with noisy measurements and uncertainties in the system more effectively.

**Figure 9** shows the output signals of PD+FNN in different levels of noise. **Figure 9**, shows that the overall control signal is determined by the FNN controller, and the output signal from PD tends to be close to zero. Therefore, the FNN learns the

FIGURE 8 | Euclidean error in different levels of noise, (A) is µv = 100 L/min, σv = 70 L/min, (B) is µv = 150 L/min, σv = 70 L/min, (C) is µv = 200 L/min, σv = 70 L/min, (D) is µv = 300 L/min, σv = 70 L/min.

Xu et al. Robust Learning Control for Manipulator

dynamics of the system and ultimately takes responsibility for the system.

## 5. EXPERIMENTAL VERIFICATION

## 5.1. Introduction of Experimental Equipment

The crane experimental model was placed on a swaying table to simulate the sea state. The relationship between each part of manipulators is shown in **Figure 10**.

The sensor and controller parameters are shown in **Table 2**.

#### 5.2. Analysis of Step Signal Response

The test results are shown in **Figure 11** with the step extension signal.

The release process of the accumulator is shown in **Figure 11**. When the cylinder is extended, the accumulator releases the oil non-linearly. The cylinder protrudes quickly, and the extension time is about 0.25 s. The color block diagram on the right side of **Figure 11** shows that the energy release rate of the accumulator is significantly correlated to the rodless chamber pressure and the cylinder extension speed.

#### 5.3. Experimental Strategy and Data Analysis

The swaying table was controlled with a sinusoidal signal in the frequency of 0.11Hz and an amplitude of ±7 ◦ . The comparison experiment of automatic wave compensation was performed using a PD algorithm and PD+FNN to control the angular at 0◦ .

**Figure 12** shows the wave compensation results of the conventional PD controller. During the wave compensation process, the cylinder pressure is stable, and the fluctuation range is approximately 90–105 Bar. However, the expansion and contraction movement of the cylinder is irregular due to the vicious cycle caused by the accumulator intervening. TABLE 2 | Parameters of components.

The performance of the cylinder control varies greatly, and the delay is severe. The wave compensation angle error is ±3 ◦ . **Figure 13** shows that the accumulator flow output is irregular.

**Figure 14** shows the experimental results of the PD+FNN controller. The optimization strategy effectively solves the speed shock caused by the accumulator. The cylinder control signal is consistent with the motion of the swinging table, which largely reduces the vibration of the cylinder. The wave compensation effect has been improved, and the angle compensation deviation is stable at ±1 ◦ .

**Figure 15** shows the highest value of the accumulator flow output concentrated in the negative angle of the cylinder (cylinder extension).

## 6. CONCLUSION

In this work, a novel control strategy, i.e., PD+FNN approach, is designed to control a shipborne manipulator. Specifically, it is able to handle the high-order non-linearity, time-varying and hysteresis characteristics of the valve-controlled cylinder under the intervention of the accumulator. In addition, it can solve the overshoot generated by the wave compensation process when the accumulator releases energy and the cylinder reacts quickly in the extended stage. Moreover, the presented control strategy is capable of solving the problem of pressure fluctuation. The control precision is improved compared to using a conventional PD controller. Qualitative and quantitative tests on the simulation and real experiments have shown that the proposed controller is capable of significantly reducing

steady state-errors and in overcoming the disturbances caused by the accumulator and uncertainties. The deviation angle of compensation is ±1 ◦ instead of ±3 ◦ compared to the conventional PD controller. We believe that the results of this work will motivate a wider use of the proposed PD+FNN approach, for autonomous collaboration of marine vehicles with a shipborne manipulator.

## DATA AVAILABILITY

All datasets generated for this study are included in the manuscript and/or the supplementary files.

## AUTHOR CONTRIBUTIONS

ZX and WL conceived and designed the experiments. YW performed the experiments. ZX and YW analyzed the data and wrote the paper.

## REFERENCES

Engineering (ICMSC), 2017 International Conference on, Saint Petersburg, 250–254.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Xu, Li and Wang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Mobile Robot Path Planning Based on Ant Colony Algorithm With A∗ Heuristic Method

#### Xiaolin Dai 1,2, Shuai Long1 , Zhiwen Zhang1 and Dawei Gong1,2 \*

*1 School of Mechatronics Engineering, University of Electronic Science and Technology of China, Chengdu, China, 2 Center of Robot, University of Electronic Science and Technology of China, Chengdu, China*

This paper proposes an improved ant colony algorithm to achieve efficient searching capabilities of path planning in complicated maps for mobile robot. The improved ant colony algorithm uses the characteristics of A∗ algorithm and MAX-MIN Ant system. Firstly, the grid environment model is constructed. The evaluation function of A∗ algorithm and the bending suppression operator are introduced to improve the heuristic information of the Ant colony algorithm, which can accelerate the convergence speed and increase the smoothness of the global path. Secondly, the retraction mechanism is introduced to solve the deadlock problem. Then the MAX-MIN ant system is transformed into local diffusion pheromone and only the best solution from iteration trials can be added to pheromone update. And, strengths of the pheromone trails are effectively limited for avoiding premature convergence of search. This gives an effective improvement and high performance to ACO in complex tunnel, trough and baffle maps and gives a better result as compare to traditional versions of ACO. The simulation results show that the improved ant colony algorithm is more effective and faster.

#### Edited by:

*Caixia Cai, Agency for Science, Technology and Research (A*∗*STAR), Singapore*

#### Reviewed by:

*Wenchao Gao, Institute for Infocomm Research (A*∗*STAR), Singapore Ran Duan, Hong Kong Polytechnic University, Hong Kong Bonan Huang, Northeastern University, China*

#### \*Correspondence:

*Dawei Gong<EMAIL_ADDRESS>
Received: *20 December 2018* Accepted: *25 March 2019* Published: *16 April 2019*

#### Citation:

*Dai X, Long S, Zhang Z and Gong D (2019) Mobile Robot Path Planning Based on Ant Colony Algorithm With A* ∗ *Heuristic Method. Front. Neurorobot. 13:15. doi: 10.3389/fnbot.2019.00015* Keywords: path planning, ant colony algorithm, A∗ algorithm, bending suppression, retraction mechanism

## INTRODUCTION

Path planning is a key issue in the field of mobile robot research. Its main purpose is to find an optimal or suboptimal, safe and collision-free path from the starting point to the target point in the environment with obstacle (Cheng et al., 2010; Deepak et al., 2012; Zhou et al., 2013). According to the degree of intelligence in the process of path planning, mobile robot path planning can be divided into traditional path planning and intelligent path planning. The traditional path planning algorithm includes simulated annealing algorithm (Miao and Tian, 2013), potential function theory (Cetin and Yilmaz, 2014; Nair et al., 2015), fuzzy logic algorithm (Li et al., 2013; Jiang and Li, 2014; Bakdi et al., 2016) and so on. However, these traditional methods can't be further improved in path search efficiency and path optimization.  The ant colony algorithm has the advantages of strong robustness, good global optimization ability and inherent parallelism. Moreover, it easily combines with multiple heuristic algorithms to improve the performance of algorithms. So it is widely used in path planning.

However, due to the randomness of probabilistic transfer and the inappropriateness of pheromone intensity update, the traditional ACO will easily fall into the local optimum and tend to poor convergence. To this end, many scholars delivered a variety of improved methods to solve problems regarding pheromone update and path search strategy (Stützle and Hoos, 2000; Zeng et al., 2016; Zhao et al., 2016; Zhang et al., 2017). In Stützle and Hoos (2000), an Ant Colony System (ACS) algorithm was proposed to speed up the convergence rate of ACO by updating pheromones on the path of the optimal ant of each generation. In Zhao et al. (2016), by adaptively changing the volatilization rate and adjusting the pheromone updating formula, the search ability of the ant colony and the convergence rate of the algorithm were improved. In Zhao et al. (2016), some intelligent algorithm was proposed to generate an initial path, which can be transformed into the initial pheromone distribution to avoid blind search of ant colony. In Zhang et al. (2017), the path information (such as the crowded path and the steep path weight) was added into the initial pheromone matrix, which could affects the efficiency of the algorithm. In Zhao et al. (2016), the heuristic function was adjusted to improve the convergence rate of the algorithm according to the target point. In Zeng et al. (2016), it unlimited step length of finding optimal path so that the improved ACO could find a shorter path and its convergence was better. In addition, many scholars have combined the ant colony algorithm with other (intelligent) algorithms (He et al., 2016b; Liu et al., 2016; Yen and Cheng, 2016; He and Zhang, 2017) to improve the convergence rate and the smooth of path. In Liu et al. (2016), the geometric method was used to optimize path. Also in Liu et al. (2016), the force factor in the artificial potential field method is transformed into local diffusion pheromone to improve the ability of the ant colony algorithm to find the obstacle. In Yen and Cheng (2016), the fuzzy ant colony optimization method was proposed to minimize the iterative learning error.

In this paper, an effective version of ant colony algorithm is achieved. It utilizes the evaluation function of A∗ algorithm to improve the heuristic information of Ant colony algorithm, which accelerates the convergence speed during the search. And MAX–MIN Ant System is used to make the global search ability better by updating the path pheromone of the optimal network. At the same time, the bending suppression operator is introduced to improve heuristic information, which aims to optimize the smoothness of the path. The problem of deadlock is solved by using the retraction mechanism. All these procedures not only give an effective improvement and better performance to ACO, but also give the best results as compare to traditional versions of the algorithm (Zhao et al., 2016) and ACO in complex tunnel, trough and baffle maps. The simulation results show that the proposed algorithm is effective and fast.

#### MATERIALS AND METHODS

#### Environment Model

The work environment is built by using the grid model, which divides the robot working space into N ∗N squares. As shown in **Figure 1**, the gray grids are represented as obstacles (the grid with barriers) and the white grids are represented as free grid squares

(the robot can move). In order to identify obstacles, the white grid cell is represented by 0 and the gray grid unit is represented by 1. The grid method is simple and effective to create and maintain grid model. Moreover, the grid method have strong adaptability for obstacle. This method is convenient for computer storage and processing.

The grid model was placed into two-dimensional coordinate system. And then serial number method is adopted to mark each grid. In N ∗N grid map, the starting node is named after Start and the target node is named after Goal. The position coordinates x, y corresponding to any grid whose grid number is R as follow:

$$\begin{cases} \chi = \begin{cases} \bmod(R,N) - 0.5 & \text{if } mod \,(R,N) \text{!} = 0\\ N + mod \,(R,N) - 0.5 & \text{otherwise} \end{cases} \end{cases} \tag{1}$$

Where mod is the surplus operation, ceil rounds the elements to the nearest integers toward infinity.

#### Ant Colony Algorithm

Heuristic Strategy With Direction Information

In the traditional ACO, the probability of the next node is selected by roulette wheel method as follows:

$$\begin{cases} P\_{\vec{\eta}}^{k}(t) = \begin{cases} \frac{\left(\tau\_{\vec{\eta}}(t)\right)^{a} \cdot \left(\eta\_{\vec{\eta}}(t)\right)^{\beta}}{\sum\_{i \in \mathit{allow}\_{k}} \left(\tau\_{\vec{\eta}}(t)\right)^{a} \cdot \left(\eta\_{\vec{\eta}}(t)\right)^{\beta}} & s \in \mathit{allow}\_{k} \\ 0 & s \notin \mathit{allow}\_{k} \end{cases} \tag{2}$$

$$\begin{aligned} \eta\_{\vec{\eta}}(t) &= \frac{1}{d\_{\vec{\eta}}} \\ d\_{\vec{\eta}} &= \sqrt{\left(\chi\_{\vec{\jmath}} - \chi\_{\vec{\imath}}\right)^{2} + \left(\chi\_{\vec{\jmath}} - \chi\_{\vec{\imath}}\right)^{2}} \end{aligned} $$

Where τij is the pheromone trail of the path grid i to grid j, and ηij is the heuristic information of the path grid i to grid j. α is the stimulating factor of pheromone concentration which determine the relative influence of the pheromone trail. β is the stimulating factor of visibility which determine the relative influence of the heuristic information. dij is the distance between node i and node j. (xi , yi) and (xj , yj) is the coordinates of grid i and grid j. allowk is the collection of grids which ants can choose when ants in the grid i (in other words, they are the grids around the grid i except the obstacle grid and taboo grid).

#### Coverage and Updating Strategy

According to the traditional ACO, the next node is decided by the roulette wheel method and it is repeated until the target point is obtained. After each iteration is completed, pheromone trails are updated in line with the length of path planning. For each trial during pheromone update, all imperfect pheromones evaporates and only the best pheromones are updated to trials history, because it enables ants to neglect all substandard pheromone trails and improve its coverage efficiency to find a shorter path. Formula (3) is used to update the pheromone quantity on each vertex at the end of each cycle:

$$\begin{cases} \mathsf{r}\_{\vec{l}\vec{j}} &= (1 - \rho)\,\mathsf{r}\_{\vec{l}\vec{j}} + \Delta\mathsf{r}\_{\vec{l}\vec{j}}\\ \Delta\mathsf{r}\_{\vec{l}\vec{j}} = \sum\_{k=1}^{m} \Delta\mathsf{r}\_{\vec{l}\vec{j}}^{k} & , \ 0 < \rho < 1 \end{cases} \tag{3}$$

where m is the number of ants. ρ is pheromone evaporation rate. 1τ k ij represents the value of pheromone that the ant k leaves in the path of grid i to grid j. This article uses the ant-cycle-system model, and 1τ k ij is defined as follows:

$$
\Delta \pi\_{ij}^k(k) = \begin{cases}
\begin{array}{l}
\text{Q}\_1/L\_k\left(t\right) \text{ if } \arcsin\left(i,j\right) \text{ is used by k in iteration t} \\
0 \text{ otherwise}
\end{array}
\tag{4}
$$

Where Q1 is a constant. LK (t) is the length of the path that the ant k is looking for.

#### Improved Ant Colony Algorithm

The traditional ACO has the following shortcomings: Due to the lack of initial pheromone and the unapparent difference of the heuristic value between adjacent grids, it usually requires a longer search time, which leads to the slow convergence rate. When grid model is complex, the robot maybe fall into a deadlock state in which the robot cannot move to the surrounding grids. In the grid map, the path planning with traditional ACO may have more bending times and big cumulative bending angle. To solve the above problems, this paper makes the following improvements.

#### Heuristic Information Based on A∗ Algorithm

A ∗ algorithm (Duchon et al., 2014) is the most effective direct search method for solving the shortest path in static road network. It is developed on the basis of Dijkstra algorithm, which can avoid blind search to improve search efficiency. In this paper, A∗ algorithm is used as the heuristic information of path searching to improve the convergence speed of the algorithm and obtain the better path. The bending suppression operator is added to the heuristic information to reduce bending times and cumulative bending angle.

The heuristic cost of A∗ algorithm is expressed by the estimated function, and the estimated function equation f (n) is as follows:

$$f\left(n\right) = g\left(n\right) + h\left(n\right)\tag{5}$$

$$h\left(n\right) = \left(\left(n\_{\rm x} - \mathrm{g}\_{\rm x}\right)^{2} + \left(n\_{\rm y} - \mathrm{g}\_{\rm y}\right)^{2}\right)^{1/2}$$

$$\mathrm{g}\left(n\right) = \left(\left(n\_{\rm x} - \mathrm{s}\_{\rm x}\right)^{2} + \left(n\_{\rm y} - \mathrm{s}\_{\rm y}\right)^{2}\right)^{1/2}$$

where g(n) is the minimum cost from the source node to the current node. h(n) is the minimum cost from the current node to the destination node. nx and ny are the coordinates of the current node n . gx and gy are the coordinates of the target node g, sx, and sy are the coordinates of the initial node s.

The estimated function of A∗ algorithm is used as heuristic information to search for global optimal path in ant colony algorithm, and the bending suppression operator is added to the heuristic value of ant colony algorithm to reduce the number of bending times and the large cumulative turning angle. The improved heuristic information formula is as follows:

$$\eta\_{ij}(t) = \frac{Q\_2}{h(n) + \mathcal{g}(n) + cost(bend)} \tag{6}$$

$$cost \, (bend) = \boldsymbol{\varphi} \cdot turn + \boldsymbol{\psi} \cdot thita$$

where Q2 is a constant more than 1. cost(bend) is a bending suppression operator. turn is the number of turns from node n − 1(previous node) to node n + 1 (next node). thita is the angle between the line segment of node n − 1 to node n (current node) and the line segment of node n to node n + 1. ϕ is the coefficient converting turning times into grid length. ψ is the coefficient converting angle into grid length.

#### Solve the Deadlock Problem

When the robot environment is more complex (especially the ants go into the environment of concave obstacles), due to the presence of the taboo table, the ants may fall into a deadlock state without the next grid to move. As shown in **Figure 2**, when the ant travels from the grid T to the grid S, the next optional grid is C. At this time, the ant is trapped in a deadlock state and it cannot move to its surrounding grid.

For the deadlock problem, Wang and Yu (2011) adopted the early death strategy, which deleted the ants trapped in a deadlock state from the ant colony and did not update the global pheromone. However, when more of the ants are trapped in the deadlock state, the number of ants that can reach the goal is significantly reduced, which results in a decrease in the diversity of solutions and is not conducive to the search of optimal path for ants. In this paper, the improvement measure is that the ants adopt retraction mechanism when they fall into the deadlock state. As shown in **Figure 2**, the ant, which has walked into the grid, is trapped in the deadlock state, and the improved strategy allows the ant to roll back one step and updates the taboo table information. If the ant is still trapped into a deadlock state, the ant will continue to rollback untill grid T. At this moment, the ant escapes the deadlock area. Since the deadlock

edge may be the part of global optimal path, no pheromone punishment is carried out on the deadlock edge. The retraction mechanism cannot prevent ants from entering a deadlock state, but it lets the deadlocked ants return back to the previous grid until there is a feasible grid around the ants, so the ACO with the retraction mechanism has higher efficiency and fewer iterations. The ACO with the retraction mechanism and without the retraction mechanism is compared in section The Retraction Mechanism Results Analysis below.

#### Max–Min Ant System

As the traditional ant colony algorithm may cause premature convergence and precocious phenomenon, it needs to improve algorithm to solve these problem. The MAX-MIN Ant System (MMAS) (Stützle and Hoos, 2000) can solve these problems well.

(1) Pheromone trail updating. After each iteration trial, the pheromone is submitted into update history in traditional ant colony algorithm. While in the MMAS, only the path pheromone of the optimal network is updated after the iteration is completed. Accordingly, the modified pheromone trail update rule is stated by:

$$\tau\_{\vec{ij}}(t+1) = (1 - \rho)\tau\_{\vec{ij}}(t) + \Delta\tau\_{\vec{ij}}^{best} \tag{7}$$

$$\Delta\tau\_{\vec{ij}}^k(t) = \frac{Q\_1}{L^{best}} + \frac{Q\_3}{C\_{turn}^{best}}$$

$$C\_{turn}^{best} = \omega\_1 \text{Cals}\left(l\right) + \omega\_2 \text{Turns}\left(l\right)$$

$$\begin{aligned} \omega\_1 &= \frac{V\_{robot}}{W\_{robot}}\\ \omega\_2 &= V\_{robot} \times t\_a \end{aligned}$$

where Q3 is a constant more than 1.L best denotes to the shortest path currently found by the algorithm. Cals(l) represents the sum of all the angles of turning on the best optimized path. Turns(l) is the sum of the turns on the best optimized path. w1 and w2

represent different weight coefficient and are set by analyzing the robot's structure and kinematics (Wu et al., 2013; Li et al., 2017). The w1 and w2 can convert turning angle and turning times into grid length, respectively. Vrobot represents the constant speed of a mobile robot. Wrobot represents the angular speed of a mobile robot as it turns. ta represents the time of acceleration and deceleration as the mobile robot turns once.

(2) Pheromone trail limits. In order to avoid the situation that the traditional ant colony algorithm may falls into local optimal solution and loses the further search space ability by pheromone accumulation, the pheromone trail of the MMAS is limited in the upper limits and lower limits - Taumin,Taumax, . The formula is:

$$\text{Tau} = \begin{cases} \text{Tau}\_{min}, & \text{Tau} \le \text{Tau}\_{min} \\ \text{Tau}, & \text{Tau}\_{min} < \text{Tau} \le \text{Tau}\_{max} \\ \text{Tau}\_{max}, & \text{Tau} > \text{Tau}\_{min} \end{cases} \tag{8}$$

#### Aco Procedure

To sum up, specific steps of mobile robot path planning based on the improved ant colony algorithm are as follows:

Step 1: The working environment is modeled by the grid method, and the starting point start and the target point goal of the mobile robot are given.

Step 2: Initialize the ant system. Set the number m of ants, parameter α which determines the relative influence of the pheromone trail, parameter β which determines the heuristic value, the global pheromone volatilization coefficient ρ, pheromone intensity Q1 and other related parameters.

Step 3: Update taboo table. Place the ant k (k = 1, 2, · · · , m) on the current node and add the current node to the corresponding taboo table.

Step 4: Process deadlock. According to the taboo table, it will judge whether ants are trapped in a deadlock state. If the ants are in a deadlock state, the retraction mechanism will be adopted and the deadlock node will be added to the taboo table. Conversely, it will judge whether the ants reach the target point. If the ants reach the target point, it will turn to Step 6, otherwise it will turn to Step 5.

Step 5: Select the next grid. It will calculate the heuristic function according to formula (6), and calculate the probability function according to formula (2). Finally, it will use the roulette method to select the next feasible grid. If the ants reach the target grid, it will turn to Step 6, otherwise it will turn to Step 3.

Step 6: If the ants reach the target node, it will repeat Step 3 until each ant completes the search target during its iteration process and then turn to Step 7.

Step 7: Update pheromone. After each iteration, if the number of iterations satisfies inequality N ≤ Nmax, it will update the path pheromone and determine whether it meets the convergence conditions. If it meets the convergence conditions, it will withdraw. If it does not meet, it will turn to Step 3. If the number of iterations satisfies inequality N > Nmax, it will be not counted further. The final result is output as long as the end condition is satisfied.

The implementation process of improved ant colony algorithm is as in **Table 1**.

## RESULTS

In order to verify the effectiveness of the improved ant colony algorithm, this paper uses MATLAB software to simulate. It is more convincing to use comparative method to carry out experiments under the same experimental conditions. In the simulation, the main parameter values of the ACO should be determined firstly. The main parameters include number of ants, stimulating factor of pheromone concentration, stimulating factor of visibility and pheromone evaporation coefficient. Through parameter analysis method (Wu et al., 2010), the relationship between each parameter and simulation results (path length, number of iterations) can be obtained. According to the relationship between each parameter and simulation results (Shi et al., 2014), we can get the value of the main parameters in the ACO. In the simulation, value of each parameter in the ACO is as in **Table 2**:

## Comparative Analysis of Path Planning Algorithms

The experiment was divided into four parts according to four types of maps(the common map, the tunnel map, the trough map and the baffle map) and three algorithms (the traditional ant colony algorithm, the algorithm (Zhao et al., 2016) and

TABLE 1 | Description of ACO algorithm for solving path planning.

#### Algorithm A\*MMAS Begin

create grid environment initialize the ant colony system

#### Repeat

for each ant k do

if grid *i* ∈ *allowk* then

if grid *i* ∈ *taboodeadlock* then

fallback

end if

according to formula (2) and (6) select next grid j

#### Update taboo end if

Update pheromone on each iteration by improved MMAS method according to formula (7) and (8)

Until algorithm convergence

Return best grid serial number

END

TABLE 2 | Values of the main parameters in the ACO.

the improved ant colony algorithm proposed in this paper) are simulated on each map in turn**.** The convergence speed, shortest path length and bending suppression effect of those algorithms are compared.

As shown in **Figure 3**, the optimized path length of the improved ant colony algorithm is 29.2133 and the number of bending times is 6. The improved ant colony algorithm is basically as same as the path planning effect of the ant colony algorithm (Zhao et al., 2016) on the path length, but it is 25% lower on bending times than the ant colony algorithm (Zhao et al., 2016). Compared with the traditional ant colony algorithm, it is 73% reduction in the number of bending times.

As shown in **Figure 4**, the optimized length of the improved ant colony algorithm is 37.3849, and the number of bending times is 7. In the shortest path length, the improved ant colony algorithm is basically as same as the algorithm (Zhao et al., 2016). In the number of bending times, it is 50% decrease than the traditional ant colony algorithm and is 22% decrease than the algorithm (Zhao et al., 2016).

As shown in **Figure 5**, the optimized path length of the improved ant colony algorithm is 51.1128. In **Figure 6**, the optimized path length of the improved ant colony algorithm is 50.7280. But in **Figures 5**, **6**, both the traditional ant colony algorithm and the algorithm (Zhao et al., 2016) can't search the global optimized path. Even as the scale of the problem expands and the environment map becomes more and more complex, the improved algorithm can still perform very well.

The results of the three algorithms that run 100 times in same map environments are shown in **Table 3**. Compared with the traditional ant colony algorithm and the algorithm (Zhao et al., 2016), the improved algorithm has a good performance on the efficiency. At the same time, it has a good adaptability in a complicated area. The improved algorithm proposed in this paper can be used not only in the path planning of mobile robots, but also in the path planning of robot manipulators (Yang et al., 2017, 2018).

## The Retraction Mechanism Results Analysis

In order to show the function of retraction mechanism, the ACO with the retraction mechanism and ACO without the retraction mechanism are tested on the trough map and the baffle map, respectively.

As shown in **Figures 7A**, **8A**, ACO with the retraction mechanism has higher efficiency and fewer iteration than ACO

TABLE 3 | Test results for three algorithms under different maps.

➀*: The traditional ant colony algorithm.*

➁*: The algorithm [20].* ➂*: the improved ant colony algorithm.*

without the retraction mechanism. When ants fall into deadlock state, the retraction mechanism is used to replace the early death strategy, which avoids a large number of ant deaths in one iteration. Therefore, each ant can obtain a path by using the retraction mechanism, which increases the diversity of results and is beneficial to find the optimal path. As shown in **Figures 7B**, **8B**, the number of ant retracted in the initial stage of the algorithm is higher than in the middle and later stage of the algorithm and the retraction mechanism can effectively suppress the decline of the number of ants.

### DISCUSSION

This paper makes a valuable contribution to the improvement of ant colony algorithm in complicated maps for the mobile robot, especially the improvement on convergence speed, shortest path length and bending suppression effect. The estimated function of improved A∗ algorithm is used as the heuristic function to improve search efficiency and smoothness of path. By employing the retraction mechanism and the improved MAX–MIN Ant System method, the problem of ant deadlock is solved and the global search ability of the algorithm is improved.

Three algorithms are researched on path planning in the common map, tunnel map, trough map and baffle map, respectively. Compared with the traditional ant colony algorithm and the algorithm (Zhao et al., 2016), the improved ant colony algorithm is better in the convergence rate and the bending suppression effect. Compared with the traditional ant colony algorithm, the improved ant colony algorithm has more than 65% reduction in number of iterations and 41% decrease in bending suppression. In addition, the improved ant colony algorithm is 54% lower than the algorithm (Zhao et al., 2016) in number of iterations. To sum up, this paper proves the effectiveness, rapidity and adaptability of the improved ant colony algorithm in the complex map environment.

## AUTHOR CONTRIBUTIONS

XD and DG proposed the innovation and designed the experiment in this study. ZZ and SL performed the simulation experiment and analyzed the experiment results. XD checked the

results. SL wrote the manuscript and DG provided writing advice of manuscript.

#### FUNDING

This work was supported by the National Natural Science Foundation of China (61603076) (51305066), Fundamental Research Funds for the Central Universities (ZYGX2016J116),

#### REFERENCES

and Science and Technology Support Program of Sichuan Province (2016GZ0198).

## ACKNOWLEDGMENTS

We acknowledge the support of the School of Mechatronics Engineering and Center of Robot in University of Electronic Science and Technology of China.

algorithm for multi-robot path planning. Swarm Evol. Comput. 28, 14-28. doi: 10.1016/j.swevo.2015.10.011

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Dai, Long, Zhang and Gong. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Neural Network Based Uncertainty Prediction for Autonomous Vehicle Application

Feihu Zhang1 \*, Clara Marina Martinez 2 , Daniel Clarke3 , Dongpu Cao4 and Alois Knoll 5

*1 School of Marine Science and Technology, Northwestern Polytechnical University, Xi'an, China, 2 Porsche Engineering Services GmbH, Bietigheim-Bissingen, Germany, 3 Cogsense Technologies Limited, London, United Kingdom, 4 Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, Canada, 5 Department of Informatics, Technical University of Munich, Munich, Germany*

This paper proposes a framework for uncertainty prediction in complex fusion networks, where signals become available sporadically. Assuming there is no information of the sensor characteristics available, a surrogated model of the sensor uncertainty is yielded directly from data through artificial neural networks. The strategy developed is applied to autonomous vehicle localization through odometry sensors (speed and orientation), so as to determine the location uncertainty in the trajectory. The results obtained allow for fusion of autonomous vehicle location measurements, and effective correction of the accumulated odometry error in most scenarios. The neural networks applicability and generalization capacity are proven, evidencing the suitability of the presented methodology for uncertainty estimation in non-linear and intractable processes.

#### Edited by:

*Changhong Fu, Tongji University, China*

#### Reviewed by:

*Liang Lu, Polytechnic University of Madrid, Spain Chao Zhang, Shanghai Jiao Tong University, China Hui Xie, Western Sydney University, Australia*

#### \*Correspondence: *Feihu Zhang<EMAIL_ADDRESS>
Received: *08 October 2018* Accepted: *18 March 2019* Published: *10 May 2019*

#### Citation:

*Zhang F, Martinez CM, Clarke D, Cao D and Knoll A (2019) Neural Network Based Uncertainty Prediction for Autonomous Vehicle Application. Front. Neurorobot. 13:12. doi: 10.3389/fnbot.2019.00012* Keywords: neural network, autonomous driving, uncertainty prediction, localization, odometry

## 1. INTRODUCTION

Mobility has become a serious challenge in a society with a gradually aging population and a perpetually increasing traffic. This situation is motivating the automotive industry and governments to invest heavily in highly automatized vehicles toward full autonomy. Nevertheless, the progress in autonomous vehicle integration is being hindered from an engineering perspective, due to limits in vehicle sensing and infrastructure modernization requirements (Ma et al., 2018; Taeihagh and Lim, 2019). Moreover, considerable advances in electronics and control theory, safety and robust autonomous driving can only be achieved in conditions that vehicles are fully aware of the driving scenario (Han et al., 2012; Li et al., 2014).

High quality measurements can be obtained using expensive sensors (Elfring et al., 2016), as installed in the well-known Google Car. This vehicle includes an advanced laser range finder, between other sensors, able to process the environment real time (Poczter and Jankovic, 2014). Nonetheless, these advanced devices are generally associated with elevated cost, and therefore are not feasible for serial production vehicles. An alternative solution is to compensate measurements quality with higher redundancy by installing larger number of low cost devices based on different technologies. As a consequence, the features perception becomes a complex problems where heterogeneous signals need to be registered, transformed into a common level and conveniently combined to guarantee safety (Jiang et al., 2011). This process is known as data fusion and usually involves noisy measurements and highly non-linear transformations.

Data fusion can be executed in either centralized or decentralized architectures. Whilst the first involves a common processor, and decentralized architectures consist of networks where each sensor has its own processing unit (Grime and Durrant-Whyte, 1994; Durrant-Whyte et al., 2001; Garcia-Ligero et al., 2012). On the one hand, centralized architectures need to be re-designed when changes in the sensing units take place, which implies costly and time-consuming development. On the other hand, decentralized solutions are particularly convenient in networks where sensors can be dynamically added and removed from the networks as a result of being sporadically available. Nevertheless, although a plurality of measurements might become accessible in decentralized architectures, fusion requires knowledge of the uncertainty associated to them. Furthermore, measurements are not used directly, but information extracted from them hereby referred to as features, which generally involves nonlinear transformations. Consequently, the conversion of sensor noise into feature noise is a complex task that usually involves arduous mathematical derivations and can be intractable in many applications.

Several attempts to uncertainty prediction for inertial measurement units (IMU) have been presented in the literature. These include methodologies to fuse odometric measurements with global positioning system measurements, geographical information systems and laser scanners, between others. Vision systems are used in Park et al. (2012) and the disparity between the image space and the Cartesian space is used to derive the uncertainty mathematical model. Vision based controlled is used by Fu et al. (2018) in a system based on an onboard camera and an IMU. The authors use a non-singleton fuzzy logic controller able to handle high uncertainties. The Kalman filter has been also widely used to deal with noisy measurement and models. The parameters estimation was performed using methods such as random walk, Gaussian-Markov and autoregressive processes (El-Diasty and Pagiatakis, 2008). Extended Kalman filters have been proposed by other authors such as Bry et al. (2012) and Fabrizi et al. (2000), where the noise assumption is taken using Gaussian white noise.

This paper presents a solution to facilitate data fusion in decentralized architectures. The proposed paper enhances our previous system for feature extraction (Martinez et al., 2017), where either the source of information, or the sensor noise is unknown. These networks require an appropriate estimation of the signals uncertainty so as to properly fuse them into an "improved measurement," rather than worsen the fusion output. The uncertainty allows evaluating the quality of the signals and provides a combined result of higher accuracy where the information retained is maximized.

Despite its importance, in the literature revised sensor noise is either assumed to be known, or fitted with simple Gaussian distributions. Hereby, a methodology directly for uncertainty prediction from raw data is proposed based on Artificial Neural Networks (ANN), assuming no information is prior available about the sensor characteristics. The applicability of this data-driven strategy extends to highly non-linear and even intractable feature transformation, avoiding tedious mathematical derivations. Proof of this is supported by its implementation for autonomous vehicle location through odometry data, obtaining satisfactory results in varied scenarios.

## 2. PROBLEM DEFINITION

Autonomous driving highly depends on the sensor measurement, uncertainty and fusion. Nonetheless, sensor models are not generality provided by sensor manufacturers. This lack of data significantly increases challenges in decentralized architectures, where new sensors can be "plugged & played" within the ad-hoc network.

## 2.1. Motivation

The operational limits of the sensor technology condition safety in autonomous driving, as a consequence of their strong dependence on the measurements quality (Zheng and McDonald, 2003; Michalke et al., 2011). The GPS precision limits are a well-known example of the noise effect in systems performance, experienced daily by the general public through of the shelf navigation devices (Schrader et al., 2012). This involves the fusion of pseudo-range GPS signals of vehicles, used to minimize the error produced by uncontrolled sources like satellite clock bias, atmospheric delay, and acquisition noise. Nevertheless, despite the complexity of the error origin, previous studies model noise using Gaussian distributions (Liu et al., 2013).

The main barriers for sensor fusion in application, such as vehicle localization, are found in the uncertainty of sensor technology integrated in each vehicle. This inevitably affects the uncertainty characteristics, and the different nature of the signals to fuse, involving highly non-linear feature transformations (Xu et al., 2014). Furthermore, incorrect uncertainty estimation could reduce the fusion accuracy and produce a security hazard by deteriorating the system performance. Regardless of its importance, most research in this area has limited the error prediction to single vehicle model-based approaches usually using Gaussian distributions, developed either theoretically or empirically. Therefore, more accurate uncertainty estimation would be of great benefit for these applications, and would solve issues that hinder the implementation of the autonomous technology nowadays.

## 2.2. Problem Statement

Odometry measurement for vehicle location is subjected to sensor noise applied to velocity d and orientation θ. This uncertainty is extended to the features x and y coordinates, which are calculated from the noisy signals through geometrical transformations (Choi and Huhtala, 2016). Consequently, the vehicle location error is accordingly described by non-linear mathematical equations and accumulates along the path with every sampling time. The absence of information about the features uncertainty, x and y covariance, prevents from fusing odometry data with additional measurements that might become available along the trajectory. By means for this, the estimation of the location uncertainty is of great interest to efficiently correct the accumulated error (Zhang et al., 2013). Hereby a feature noise estimator is obtained from data with independence of the sensor characteristics, and complexity of the feature transformation. This solution allows for sensor noise prediction, avoids the use of complex mathematical formulations and facilitates sensor fusion under any use case.

#### 2.3. Data-Driven Modeling

Feature transformations are often difficult to derive in mathematical terms, and calculations that are usually time consuming and occasionally intractable. Nevertheless, vehicle trajectory data collection under real-life conditions is generally possible using limited resources. These evidences support the use of data-driven algorithms, methods that can efficiently manage big data and yield insightful conclusions from unknown complex processes (McAfee et al., 2012; Hou and Wang, 2013). Various algorithms can be used to derive the so-called surrogate models without requiring actual understanding of the relationship between inputs and outputs. These models are compact, normally inexpensive to evaluate compared to their homologous strictly mathematically derived. Furthermore, they are mathematically tractable and can estimate the process with high-fidelity at least locally to the training set (Gorissen et al., 2010; Koziel et al., 2011). Some examples of surrogate modeling techniques include: polynomial regressions, kernel methods, kriging, support vector machine, Radial Basis Functions (RBF), and Neural Networks (NN) (Jin et al., 2001; Razavi et al., 2012). Each method has different characteristics in terms of operation, complexity, design flexibility and fidelity capability. For instance, support vectors perform particularly well with high dimensional spaces when only scarce training data is available (Forrester and Keane, 2009). Highly non-linear and complex process are better captured using RBF, kriging, and NN, which require determining a specific number of parameters by trial and error. From a high level analysis, the structural limits of RBF are relaxed with kriging, which assumes the model response has stochastic behavior and fits it with a statistical basis. In the kriging method the basis function variance is considered a parameter, providing larger flexibility and resultant increase in training time.

Artificial Neural Networks allow modeling the relationship between inputs and outputs from data. This characteristic, applied to sensor noise, is expected to be able to find the underlying relation between measurement and noise associated to them. Furthermore, NN accept multiple inputs, which can be used to determine additional features affecting the noise and their correlation. With these precedents, NN offer an exceptional framework for implementing and testing the suitability of models generated from data applied to noise magnitude prognosis of sensor measurements. Hereby, NN are selected within the above strategies to exploit their potential for sensor noise estimation, and explore their high level of flexibility owing to their substantial number of defining parameters: network structure, neuron function, number of hidden layers, and number of neurons per layer.

## 2.4. Design for Surrogate Model Development

In terms of number of hidden layers, the criterion applied focuses on the trade-off between accuracy and generalization capabilities. Sensor fusion algorithms will benefit from a guidance to assess the extent of the error covariance of new sensor measurements. This information will allow the system to identify the degree of information present in the new measurement and perform the data fusion accordingly, ensuring the output maximizes the information content. It is therefore acceptable to obtain a guidance value of this error covariance and not highly precise results, reason why the network structure selected for sensor noise estimation is formed by a single hidden layer. This simplified structure might prevent from learning accurate noise behavior as observed in deep learning, but would also facilitate training and avoid noise fitting when applied to noisy sensor signals. Generalization capability is hereby prioritize against results accuracy with the selection of the single hidden layer structure.

The network layers, named input, hidden and output, can be connected through either feedforward (FF) configuration or using a feedback (FB) connection. Layers in FFNNs only receive information from forward layers, whilst in FBNNs any neurons can connect with each other. Consequently, signals in FBNN are repeatedly transformed and lean toward steady state or vibration state. By introducing feedback delays, this structure is also able to capture the relationship between past inputs and current output, influence that is completely ignored in the FF configuration. In the following, both FF and FB configurations are examined as candidates of NN structure so as to determine the most suitable configuration with the support of the training and test results.

Once the network structure is defined, the size of the layers need to be determined. Input and output layers are constrained by the input and output signals selected set, but the hidden layer is a prior a free parameter, closely related with the process complexity. By reason of lack of known mathematical formulation of this particular case, this number has to be determined by trial and error. The optimal size criteria should consider a trade-off between complexity, accuracy, and generalization capability of the neural network candidates. Excessively complex networks not only raise training time, but also increase the risk of over fitting, which would return high accurate results over the training set and poor generalization capacity on new data (Hagan et al., 2014). The growing method is used in this application in order to prevent for over fitting by establishing an initial network with relative small size, and increasing it gradually with special attention over both the training and testing accuracy.

## 3. TRAINING DATA GENERATION AND ANALYSIS

To encourage acceptable performance under all possible scenarios, the amount and variability of the training data should ideally account for any conceivable use case. The data selected for training proceeds from six different trajectories that combine disparate direction, length, orientation, and speed as depicted in **Figure 1**.

## 3.1. Training Data Generation

The trajectories contain highly precise vehicle locations in Cartesian coordinates, and yaw signals, regarded as Ground

Truth (GT). The GT signals need to be processed to generate realistic data complying with a real case scenario. Training data is generated assuming the vehicle velocity and orientation, identified herewith with the symbols d and θ respectively, are collected with sensors characterized by white noise. For the purpose of the following investigations, the standard deviation values 0.1 and 0.001 are respectively selected for speed and orientation measurements. These values are based on experience and are considered representative of general noise measure in odometry sensors.

• Calculate d and θ GT, dGT and θGT, from xGT and yGT.

• Add white noise artificially to dGT and θGT, through a Monte Carlo (MC) simulation with 1000 iterations. These results in d and θ measured (M), dM and θM, and emulates sensor noisy data acquisition.

• Use the inverse equations to calculate xM and yM from dM and θM, which in essence is the signal to feature transformation.

• Use the 1000 MC noisy versions of the trajectories to calculate the location error standard deviation (std) in x and y and the location covariance (covxy).

**Figure 2** illustrates the detailed process in a flow diagram describing the steps and signals obtained from ground truth to measured feature data. As included in the respective steps for geometrical transformations, the signals d and θ could be obtained along the sampling steps in a cumulative manner as detailed in following equations (Zhang et al., 2013):

$$\mathbf{x}\_n = \sum\_{i=1}^n d\_i \cdot \sin\left(\sum\_{j=1}^i \theta\_j\right) \tag{1}$$

$$y\_n = \sum\_{i=1}^n d\_i \cdot \cos\left(\sum\_{j=1}^i \theta\_j\right) \tag{2}$$

where n refers to the current time step. By combining Equations (1) and (2), the variables of interest can be obtained for every sampling time:

$$\theta\_{i+1} = \arctan\left( (\mathbf{x}\_{i+1} - \mathbf{x}\_i) / \left( \mathbf{y}\_{i+1} - \mathbf{y}\_i \right) \right) - \sum\_{j=1}^{i} \theta\_j \tag{3}$$

$$d\_{i+1} = (\varkappa\_{i+1} - \varkappa\_i) \left/ \sin \left(\sum\_{j=1}^{i+1} \theta\_j \right) \right. \tag{4}$$

Noise can be artificially added to the GT results of Equations (3) and (4) by randomly generating numbers with the previously designated standard deviation. The results are regarded as sensor measurements and can be used to obtain the measured features following Equations (1) and (2), implemented as:

$$\varkappa\_{i+1} = \varkappa\_i + d\_{i+1} \cdot \sin\left(\sum\_{j=1}^{i+1} \theta\_j\right) \tag{5}$$

$$\gamma\_{i+1} = \gamma\_i + d\_{i+1} \cdot \cos\left(\sum\_{j=1}^{i+1} \theta\_j\right) \tag{6}$$

The ground truth original data and measured features can be compared to determine the uncertainty over the vehicle location at every point of the trajectories. This is defined by the standard deviation of the feature estimation error:

$$
\sigma\_{\mathfrak{x}} = \sqrt{\frac{1}{N} \sum\_{i=1}^{N} (e x\_i - \mu\_{\mathfrak{x}})} \tag{7}
$$

where N corresponds to the number of MC iterations of the same trajectory. The estimation error and the mean estimation error can be calculated as follows:

$$e\mathbf{x} = \mathbf{x}\_{GT} - \mathbf{x}\_{M} \tag{8}$$

$$
\mu\_{\mathfrak{x}} = \frac{1}{N} \sum\_{i=1}^{N} e \mathbf{x}\_i \tag{9}
$$

Similarly, the equations can be applied to y coordinate to obtain ey,µy, σy. Finally, the covariance of the errors in x and y is obtained by:

$$\text{cov}\_{\mathbf{x}\mathbf{y}} = \frac{1}{N} \sum\_{i=1}^{N} \left( \mathbf{x}\_i - \boldsymbol{\mu}\_{\mathbf{x}} \right) \left( \mathbf{y}\_i - \boldsymbol{\mu}\_{\mathbf{y}} \right) \tag{10}$$

The uncertainty is defined as σx, σy, and covxy. Mean error in x and y bias the location measurement, but are not considered as estimation targets in this particular study. The previous transformation provides information of the uncertainty of the vehicle location, when measured through noisy velocity and orientation sensors subjected to a specific level of white noise. This measurement allows evaluating the quality of the current location through odometry, and therefore to which extent this measure should contribute into a sensor fusion framework when compared to other sources.

σx obtained for each of the trajectories is illustrated in **Figures 3**, **4**, by assuming the vehicle is perfectly located at the initial point. **Figure 3** represents σx growth along the entire trajectories until the end point of the longest one, whilst **Figure 4** illustrates a zoom in the σx to better visualize the uncertainty accumulated in shorter paths. σx always presents and increasing trend due to the cumulative characteristics of the error in the vehicle location. Nonetheless, this tendency of accumulation differs between trajectories, which suggests that the shape of the trajectory and the characteristics of

the displacement affect the uncertainty growth. The curves are therefore dissimilar between trajectories and presumably influenced by variables such as 1x, 1y, and 1yaw. Analogous behavior is observed when analyzing σy and covxy, which agrees with the previous assumption.

The growth of the combined uncertainty in all directions σx, σy, and covxy is illustrated in **Figures 5**, **6**, which is represented via ellipses that increase in area as the error accumulates. A first visual examination allows identifying how larger increments in a specific

direction affect to the growth of the uncertainty differently, which also allows drawing a priori hypothesis of the variables closely related. In the following, the study concentrates on σx, although the results and conclusions are expected to be dimension agnostic and applicable to both σ y and cov xy.

#### 3.2. Training Data Analysis

As aforementioned, one of the remaining design parameters of the NN is the number of inputs. Ideally, the input variables should contain the maximum number influencing factors over the target to estimate, but its scope should be constraint to prevent from excessive training time and network complexity and overfitting. The optimal input selection should aim to gather maximum relevant information for the prediction and minimum non-relevant data. Non-useful information would increase the model complexity, and might introduce misleading data that deteriorate the generalization capability.

Inputs can be selected using common sense in easily interpretable applications, albeit there are alternative correlation analysis able to evaluate numerically their level of dependence with respect to the target (Sudheer and Ramasastri, 2002). In addition to a prior evaluation of inputs and outputs, NNs can be themselves used for signals selection. In simple network structures, the importance of each signal can be identified by looking into the magnitude of the weights that connect them to the successive layers. A formal analysis of the signals weights is included in Giordano et al. (2014), where a criteria for input selection is derived mathematically and tested.

As a result of the noisy characteristics of the variables used in this particular application, complex signal evaluation is not considered of interest. Instead, the procedure follows inputs selection, by consisting the combination of a correlation analysis with the training and testing results interpretation of various input candidates. First, the signals linear correlation is studied calculating the Pearson correlation coefficient with respect to the output. The high sampling rate used compared to the input candidates' variability, allows assuming no time lag exists between inputs and outputs, simplifying the evaluation.

Although immediate effect of inputs over the outputs is impracticable, the study of the signals' variability with respect to delayed ones suggests this assumption is acceptable (Maier and Dandy, 1997). The output of the correlation provides useful information to select several input candidates, which are later tested at a second stage to conclude into the most suitable option. The best candidate is assessed in terms of training and testing results in an iterative process.

#### 3.3. Inputs vs. Output Correlation

The signals available to use as inputs are: x, y, yaw, θ, d. The use of values such as real x and y directly impairs generalization, as the network would learn from training trajectories characterized by specific absolute location points. Moreover, the incremental tendency of the uncertainty suggests additive behavior happening in every trajectory with independence from the initial and relative vehicle location. This reasoning supports the use of signals increments between sampling steps, rather than absolute values with respect to a pre-defined reference system.

Another hypothesis that can be reasonably stated is the effect of the direction of the displacement over the uncertainty growth; that is, whether variables increments or absolute variables increments are suitable for the input set. This conjuncture would be resolved when determining the effect of the inputs increments sign on the uncertainty accumulation. It is sensible to assume that the features uncertainty is affected by the actual magnitude of the displacement with independence on the direction; the uncertainty should not be affected by the reference system. Consequently, it could be deduced that the sign omission would avoid needless information to be fed into the NN, and therefore would encourage the generalization capacity.

All previous hypotheses are considered to determine the signal candidates to evaluate in the correlation analysis. **Table 1** contains the Pearson correlation coefficients between input candidates and output in all training trajectories. The formula used is the base line Pearson equation, where r, cov, and σ represent respectively correlation coefficient, covariance and variance of the designated signals (Lee Rodgers and Nicewander, 1988).

$$r\_{\text{xy}} = \text{cov}(\nu ar\_1, \nu ar\_2) / \sigma\_{\text{var1}} \sigma\_{\text{var2}} \tag{11}$$

**Table 1** shows high correlation between σx and the signals yaw, x, and y. d appears to have secondary importance, although it proves to be relevant in training trajectories 3 and 4, when compared to 1 and 5 for instance. These differences are associated with the mean value and mean absolute value of the speed, observed to be higher in trajectories 3 and 4. In contrast, θ correlation seems to be negligible.

The correlation coefficients change substantially when analyzing variables increments. The weight of 1x and 1y weights reduce, with respect to the original variables and 1yaw becomes practically independent to the output. 1y shows stronger relationship with 1σx when compared to 1x. In contrast, 1yaw are only tangible in trajectories characterized by substantial direction changes, as happens in trajectories 4 and 5. 1d also loses relevance when compared to the absolute variable analysis, and θ influence is barely affected and kept negligible. When focusing on absolute values of the increments, the correlation results produce similar values compared to relative increments, supporting a priori hypothesis over the uncertainty independence with respect to the sign of the movement. A part from the magnitude of the correlation coefficients, the sign can be also interpreted. The broad variety of values and sign within trajectories prevents from selecting a single preferred combination of training signals, reason why several candidates are selected to further pinpoint the suitability.

As a final remark, it is worth highlighting that 1y presents larger correlation with σx than 1x, and vice versa. That is, in trajectories with more movements in x direction, σy grows quicker than σx and similarly, in trajectories with larger displacement in y direction, σx grows quicker than σy. This is observed in all trajectories with exception of trajectory 4, where both uncertainties are similar probably due to the followed direction in repeated circles. An explanation of this phenomenon might be found, in the relative amplitude of the actual displacement every sampling and the error magnitude. Whilst the error might be negligible after a large displacement, it could be of the same order of magnitude of short movements, causing larger distortion in the vehicle location. Consequently, large growth of σx could be associated to low 1x instead of being related to 1y as it was initially deduced from the results of the analysis.

## 3.4. Delayed Signals Correlation

Inputs to output correlation analysis is complemented with the signals delay study, also evaluated using the Pearson coefficient. The aim of this test is to determine the possible relationship between old inputs and current outputs; that is, the influence of past changes in the vehicle movement and location on the accumulation of the current uncertainty in the vehicle positioning. The first correlation test of delayed signals analyses the relationship between delayed inputs and current output. The steps used are 0, 1, and 2 sampling times. Next, in order to draw a holistic understanding of the signals interrelation, a second correlation test between current input signals and the same ones delayed 1 and 2 sampling steps is also analyzed.

The results of the first test show similar correlation between input and outputs with independence of the delay implemented. Nonetheless, the second test also shows strong correlation between inputs and the respective delayed inputs. Although from the first results it could be considered that the output depends on past input signals, the second analysis discredits this assumption as they could also be due to the high similarity between current and past inputs. Consequently, no conclusive assumptions can be derived from this correlation test.

The results from the second correlation test effectively show that the inputs and delayed version of the inputs are practically identical, and consequently show similar correlation with the output. As previously states, this similarity might be due to the small sampling step implemented with respect to the input signals variability in time. Further investigations should be conducted to arise conclusive answers to the previous hypothesis. Accordingly, additional study with respect to the delay effect of the feedback NN states is considered during training.

## 4. TRAINING SETS CANDIDATES

Hereby, a training set is defined as the union of a specific combination of input signals, obtained from a selected number of training trajectories in an enclosed array used as training data. That is, the training set is defined by the signals used between the candidates previously analyzed in the correlation analysis and the trajectories from which signals are extracted. The training sets can contain data proceeding exclusively from a single trajectory or from the combination of more than one. Furthermore, the same trajectory can be repeated in each set in more than one occasion by implementing different noisy version from the 1000 MC simulations, practice that intents to encourage the response robustness to the presence of noise in the inputs.

## 4.1. Training Sets

The input training sets are designed in terms of number of signals, trajectory characteristics and amount of trajectories used, and always contain noisy data so as to simulate with

TABLE 2 | Input sets candidates proposed for training and testing including: signals selected, data used, and hypothesis to verify/reject in the training results.

maximum fidelity real case studies. **Table 2** includes the training set candidates carefully designed to determine: the most suitable combination of inputs, optimal network structure and size and data variability requirements.

The second column in **Table 2** specifies the input signals used in each set, where abs and 1 indicates absolute value and signals increment, respectively. The amount of data used in each training set is detailed in the third column, alluding to the variability of trajectories used and amount of MC noisy versions of each trajectory. For instance, set 1 considers all trajectories repeated three times each; including therefore three MC noisy versions of each. The use of larger number of trajectories or specific ones is thoroughly defined, so as to reflect changes in the generalization capability with respect to training data variability. Consequently, by repeating noisy version of the same trajectory, the data variability should be much less than noisy versions of different trajectories.

The incremental variables specified in **Table 2** are calculated, with respect to the constant sampling rate in training sets. As an attempt to encourage generalization, alternative incremental inputs are proposed by using random dynamic sampling within the boundary of 1 to 9 sampling steps. Nonetheless, the networks trained with this data were not able to estimate the target variable, reason why they are neither included in **Table 2** nor in the test results. The failure to capture the process could be excused in the data variability and complexity introduced through variable sampling. The networks trained with this data were presumably required to emulate a behavior more complicated than the one described with constant sampling. Consequently, it might be the case that the amount of training data and network size used were not suitable to capture efficiently the underlying process. **Figure 7** illustrates a flow diagram that clarifies the design process and characteristics that define a training set. The input selection is partitioned in three stages: selection of the key signals combination, format of the signals preferred (real value, increments, or absolute increments) and trajectories used to extract the data.

## 4.2. Training Candidates

Sets 1 and 2, similarly to 3 and 4, compare the effect of the data variability on the generalization capacity by implementing identical input variables, but using data from different trajectories. Sets 2 and 3 determine the effect of the inputs sign again with respect to generalization, and intend to provide numerical support to proof the independence between uncertainty accumulation and movement direction. Sets 5 to 8 implement identical data, but use different input candidates so as to obtain the optimal signals combination.

#### 5. NN DESIGN AND TRAINING

The NN candidates are compared in terms of: network size, structural complexity and results output quality. The accuracy of the estimation is evaluated using various error measurements applied to both network output, 1σx, and target variable σx.

FIGURE 7 | Training sets generation from signals proceeding from the available trajectories including signals selection (from correlation analysis), signal processing (true value, increment, or absolute increment), and training trajectories used (data variability).

TABLE 3 | Training results comparison and analysis in terms of structural and data complexity, training time and performance, and accuracy indexes as convey for error evaluation.

#### 5.1. Error Measurements

The results are evaluated by using Root-Mean-Square (RMS) error and relative error measures between the network output and reference data, GT variables. The error measurement include: RMS of σx and relative error of the accumulated uncertainty obtained at the end of the trajectory, σx(end). The first assesses the actual performance of the network given the fact that the target variable is the increment of the uncertainty, 1σx, and not the cumulative one, σx. The second, σx RMS, evaluates the variable of interest and determines the possible predicted error accumulation and the actual one. Finally, the relative error of the final cumulative uncertainty analyzes how well the NN would perform in case a new sensor becomes available at the end of the vehicle path and fusion is required.

These indicators are calculated for each of the trajectories separately, so that it is possible to compare the performance in both, data used for training and data not seen before. It is also worth mentioning that the data used for training in all cases consist of 70% of the total amount that defined the training set, whilst the rest is used for testing and validation during the training process. None of the sets or NN configurations converge during the training process, when using the LR architecture as a consequence of the noisy characteristics of the data used. Nonetheless, this behavior is not necessarily detrimental due to the fact that the generalization capability is preferred to the estimation accuracy of a specific trajectory; it is of importance to avoid noise fitting. Consequently, the networks are trained up to a certain performance value or number of iterations, epochs, and early stopping is used before the training gradient stabilizes. The output to estimate is the relative increment of the error in x std in all cases, 1σx.

#### 5.2. Training Algorithms

Two algorithms are used for training, Levenberg-Marquardt (LM) and Scaled Conjugate Gradient (SCG). These aim to compensate from deficiencies in terms of robustness, and convergence time of the well-known Error Backpropagation (EBP) and Gauss-Newton algorithms (Moller, 1993; Yu and Wilamowski, 2011). Both differ in the selection of the step size and direction during convergence. Ideally, longer steps should be implemented at early stages and gradually smaller ones should be considered to encourage the result finesse in later stages. Moreover, the error shape might also change, affecting simultaneously to the optimal step direction. SCG implements

optimized step size and direction, whilst LM alternates EBP and Gauss-Newton methods depending on the error shape. LM combines the advantages of both strategies taking advantage of the speed convergence of Gauss-Newton with quadratic error, and the robustness of EBP convergence behavior under conditions of non-advantageous for Gauss-Newton.

The training results usually benefit from LM when compared to SCG throughout the test cases. Furthermore, LM presents low µ values, variable that determines the alternation between methods, but it converges neither into Gauss-Newton nor into the steepest descent method.

## 5.3. Training Results: Input Signals Selection

**Table 3** summarizes the results obtained after training specific network structures, second column, with the training sets enumerated in the first column. Training set 2 is used to compare FF and LR networks. In all cases, the training with identical set and structure is repeated in more than one occasion, typically up to six times. This practice is recommended due to the possible effect that the random weights initialization can cause over the end solution, which can potentially be trapped into local optima. The results included in **Table 3** are taken from the best network obtained after training several candidates. These figures are compared between equal amounts of iterations.

Sets 1 and 2 are omitted for brevity, as the conclusion coincides with the analysis of sets 3 and 4. Set 2 is used to implement FF and FB configurations as included in the first two rows of **Table 3**. The noise filtering capability of FBNN improves the estimation accuracy notably, reason why the LR structure is concluded as the most suitable and successively used in the subsequent training sets.

FIGURE 10 | (Top) x and y coordinates of trajectory 1; (Bottom-Left) 1σ*x* in trajectory 1 with respect to the sensor sampling; (Bottom-Right) 1σ*x* in trajectory 1 with respect to the sensor sampling.

Six networks trained with set 2 are compared to six networks trained set 3, to determine the effect of sign in the generalization capacity. The best candidate from set 3 shows that the error is 30% lower compared to the best candidate provided by set 2. The results evidence the benefits of reducing non-relevant data in the input set, and corroborate the independence of the location error accumulation with respect to the movement direction.

Sets 3 and 4 evaluate the relationship between generalization and variability of the training sets. The candidates are trained for similar number of epochs using the same structure and size. Networks trained with set 4 are expected to have improved generalization capacity, when compared to the ones trained with set 3, oppositely to the results detailed in rows four and five. These results are not conclusive due to the deficient amount of training time allowed for set 4 network candidates, but show evident differences between networks complexity when trained with each set. Larger data variability would presumably imply also higher network complexity, and therefore longer training time and number of epochs.

Sets 5 to 8 implement the same data variability, but use different combination of input signals. The training is programmed for similar number of epochs in all cases, and therefore the results are expected to benefit simpler training sets. By comparing sets 5 and 7 it can be deduced that the yaw is preferred to θ, and therefore set 6 does not benefit from the additional information. Nevertheless, the result of set 8 do Zhang et al. Neural Network Based Uncertainty Prediction

not corroborate this hypothesis, reason why extra training is programmed with sets 9 and 10. These last sets incorporate maximum data variability, and are trained for a larger number of iterations up to a satisfactory performance. As expected from the preliminary results obtained in sets 5, 6, and 7, the combination of inputs used in sets 5 and 9, 1x, 1y, and 1yaw, is superior to the other candidates. These results were already anticipated by looking into the relationship between coordinates x and y and odometry signals d and θ, which geometrically dependent through (1) and (2). Consequently, it is reasonable to assume that the combination of any of the previous pairs would not provide extra information to the training set. In contrast, yaw proceeds from a three dimensional displacement incorporating new data that seems to the valuable for uncertainty prognosis.

## 5.4. Delay Effect on the Training Results

The effect of the feedback delay over the estimation accuracy is studied in the LR configuration by implementing: 1 sample delay, 2 samples delay and the combination of both. The results obtained do not vary substantially with incremental delays, which supports the hypothesis of independence between delayed inputs and current outputs. Nevertheless, the output precision benefit in all cases from the noise filtering effect of feedback structures, reason why 1 step delayed is selected.

**Figure 8** presents the structure used to find the optimal hidden layer size, where n corresponds to the number of neurons in the hidden layer.

#### 5.5. Hidden Layer Optimal Size

Further training with set 9 using 20, 30, 40, and 50 neurons was programmed to determine the optimal hidden layer size. Networks with 20 and 30 neurons presented unacceptable error measures as they were not able to capture the process complexity. Fifty neurons networks were able to accurately estimate the cumulative uncertainty in most of the trajectories, but presented inconsistent behavior in some cases. The estimation results obtained with 40 and 50 neurons networks are visually compared in **Figures 9**, **10**. These illustrations are formed by three graphs, the trajectory shape at the top level and the uncertainty estimation at the low level, including the uncertainty increment of the left and the accumulation on the right.

**Figure 9** illustrates trajectory 0, in which the 50-neurons network returns improved results when compared to the 40 neurons networks. Higher number of neurons are able to filter the noisy inputs more effectively, as illustrated in the left graph, and seem to follow better the uncertainty increment, almost matching the cumulative value at the end of the trajectory. The estimation results of the 40-neurons network also match the increments in uncertainty and the shape of the accumulated error, but it is not able to effectively filter the noise. It could be deduced from the previous results that the more the noise is filtered, the better the estimation accuracy obtained. Nevertheless, this is not the case observed when analyzing trajectory 1 as illustrated in **Figure 10**. Although again 50 neurons networks filter the noise in the uncertainty increments in the left graph, the tendency of the cumulative uncertainty diverges from the target variable causing inconsistent behavior. TABLE 4 | Comparison of training sets 9 and 11 in terms of set characteristics and testing results in trajectories not used for training.

Oppositely, 40-neurons networks are able to both filter the noise and follow the cumulative uncertainty tendency, returning reasonably accurate results at the end point of the trajectory.

Although 50-neurons network are able to return very accurate results in most of the training trajectories, they show inconsistent behavior at times, which notably diverge from the target. As previously stated, the generalization capability primes in front of the estimation accuracy in the specific application of sensor fusion. Consequently, 40-neurons networks are considered to be the best candidate to model the uncertainty increment, and are therefore considered as reference size in the following tests.

These results agree with the so-called Ockham's Razor principle, which prefers simpler networks structures able to provide acceptable level of accuracy, rather than complex and more accurate ones. The 50-neurons network is able to capture higher non-linear process than the smaller versions. This extra modeling capacity could be either trained to fit the process more accurately, or to capture other processes such as inputs noise, excusing the divergent results depicted in **Figure 10**. Nonetheless, it is not a suitable network size for the characteristics of the available training data.

## 6. TEST RESULTS

Given the fact that the NN has seen 70% of the data used for testing, although different version of the noisy trajectories are used, these still share similar characteristics which might

prevent from yielding final conclusions. Furthermore, NNs can be considered as blackbox models whose robustness cannot be tested with conventional methods. In order to further analyze the candidate performance, 28 new trajectories combining straight lines, sharp turning and winding routes along urban areas are used for testing.

#### 6.1. Set Candidate 9

Likewise to the methodology followed to process the training trajectories, the test trajectories are converted into noisy features emulating data collection through noisy sensors. Again 1000 MC noisy versions are generated to model the uncertainty accumulation along the path. The estimation performance of the 40-neurons network is tested in all 1000 noisy version of each of the 28 trajectories so as to obtain the average error: RMS of 1σx, RMS of σx, and end error of the cumulative uncertainty. Although the estimation results are in average satisfactory, the network candidate is not able to fit error accumulation with appropriate accuracy in all test cases. Due to the characteristics of blackbox model of the NN, it is difficult to predict in which case scenarios the network will be able to capture the uncertainty growth.

**Figures 11**–**13** illustrate the results obtained in test trajectories 4, 24, and 12, respectively. These contain three graphs including the trajectory coordinates, the cumulative uncertainty and the yaw signal respectively from top to bottom. The middle graph illustrates in red the targeted cumulative uncertainty and 1000 estimation outputs of the network in blue. Although the cumulated uncertainty is well-captured in trajectories 24 and

12, this is not the case of trajectory 4. These results can be explained through the values of the yaw signal in this test scenarios. The first 200 sampling steps are characterized by close-to-zero yaw, followed by a large increment in yaw and a close-to-constant value until the end of the route. When looking into the tendency of the accumulated uncertainty, the shape seems to match the growth only in the last 200 sampling steps. This behavior is observed in some of the testing trajectories, suggesting a consistent response. Furthermore, when assessing the training trajectories, no scenario with large yaw changes is observed. As a consequence of this deficient training data, the network is not able to capture the uncertainty when these conditions take place in the testing trajectories.

In contrast to testing trajectory 4, the trajectories 24 and 12 do not present zero yaw values and maintain it mostly constant along the whole path. As a consequence, the network is able to accurately fit the uncertainty accumulation, with accuracy acceptable to allow for sensor fusion at any point of the route. It can be concluded from the previous that the generalization capacity of the network could be potentially improved provided that the training set includes the test cases missing.

## 6.2. Set Candidate 11

The deficiencies of training set 9 data variability are corrected in candidate 11. This implements the same input signals, absolute value of 1x, 1y, and 1yaw, but includes a larger amount of trajectories and therefore a larger number of case scenarios that include possible changes in the yaw signal, not previously captured. Whilst set 9 only considers data from the training trajectories 0 to 5, set 11 also includes some of the trajectories previously used for testing; test trajectories 1, 2, 3, 4, 9, 14, 22, and 24. **Table 4** contains the details of the sets characteristics, data used, network structure, training and testing results. This table includes the results corresponding to the best network trained with sets 9 and 11. Again, new networks are trained with identical structure and data, and the best is selected to avoid deceiving results caused by random weights initialization. The results included in **Table 4** evaluate the networks performance in all testing trajectories, including the ones also used for training in set 11.

In average, the prediction accuracy of the increment in the uncertainty, 1σx, is identical in both cases when analyzing the average accuracy in all test trajectories. This result suggest that the 40 neurons LRNN has reached a performance limit with set 9 and does not admit the further complexity provided in set 11. Moreover, set 9 presents better accuracy when analyzing the average RMS error in σx and worst results when comparing the end error.

When looking into the trajectories used to train set 11, as expected the results provided by the network trained with set 9 are worse. In this case set 11 has the advantage of having implemented 70% of those trajectories during training. Nevertheless, the accuracy of the network trained with set 11 is worse when looking exclusively to the trajectories not used in any of the sets. This result could be explained considering the loss of generalization capability, when the training data complexity overcomes the non-linear capacity of the network structure.

The networks results are visually compared in **Figures 14**, **15**, where test trajectories 6 and 12 are illustrated. None of these trajectories have been used to train any of the networks and therefore, the results can be interpreted as pure testing. The uncertainty estimation in the 1000 MC using the network trained with set 9 is represented in green, whilst the respective results of the network trained with set 11 are illustrated in blue. The target curve is illustrated in red in both cases. **Figure 14** shows how set 11 outperforms set 9 prediction results, whilst **Figure 15** illustrates the opposite case. Although the results of both networks are rather similar in terms of accuracy, set 11 prediction in the 1000 MC noisy versions of each trajectory are less spread than the equivalent ones from set 9. The larger variability of data used in set 11 seems to have the effect to improve the prediction robustness to noise, and therefore reduce the variability of the prediction when noisy versions of the same trajectory are used. It can be deduced that the consistency of the results is improved when implementing sets with larger variability due to improved noise robustness.

After identifying the most suitable set for training, the optimized number of neurons was investigated. The previous results suggested that the optimal number of neurons is close to 40, probably situated between 40 and 50 neurons. Therefore, further tests were performed using 38, 42, 45, and 47 neurons. The training was maintained until stabilization of the networks performance and allowing generally higher number of epochs for larger sizes. The results verify the a priori hypothesis and situate the best candidates within 38 and 42 neurons. In particular the 42-neurons candidate improves the RMS error in 14 and 18% the 40 and 45-neurons candidates respectively. The cumulative RMS and end errors are also noticeable improved. The RMS error is improved by 0.0296 and 0.0184 and the end error by 0.161 and 0.457, when compared to the 40 and 45-neurons networks respectively. It can be therefore concluded that the optimal network size is 42-neurons hidden layer.

## 7. CONCLUSIONS

This paper proposes a strategy for feature uncertainty estimation directly from data without prior knowledge of the sensors characteristics. NNs learning capability of nonlinear processes is tested in the particular application of vehicle location through odometry measurements. Both input set and network structure design are based on training and testing results obtained with various neural network candidates. The final results confirm NNs as suitable surrogate modeling technique robust to changes in the testing data, inputs noise and variable case scenarios, provided that the training data captures enough data variability and the network size and structure complexity is able to resemble the process non-linear characteristics.

## AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

#### REFERENCES

#### ACKNOWLEDGMENTS

This work was supported by the National Natural Science Foundation of China (NSFC) under Grants 61703335, the Fundamental Research Funds for the Central Universities under Grants G2017KY0302, and the 111 Project under Grant B18041.

Struct. Multidiscipl. Optim. 23, 1–13. doi: 10.1007/s00158-001-0 160-4

**Conflict of Interest Statement:** CM was employed by company Porsche Engineering Services GmbH, DC was employed by company Cogsense Technologies Limited. All other authors declare no competing interests.

Copyright © 2019 Zhang, Martinez, Clarke, Cao and Knoll. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# SVM-Based Classification of sEMG Signals for Upper-Limb Self-Rehabilitation Training

Siqi Cai 1 , Yan Chen1 , Shuangyuan Huang1 , Yan Wu2 , Haiqing Zheng3 , Xin Li 3 and Longhan Xie1 \*

*1 Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, Guangzhou, China, 2 A* ∗*STAR Institute for Infocomm Research, Singapore, Singapore, 3 The Third Affiliated Hospital, Sun Yat-sen University, Guangzhou, China*

Robot-assisted rehabilitation is a growing field that can provide an intensity, quality, and quantity of treatment that exceed therapist-mediated rehabilitation. Several control algorithms have been implemented in rehabilitation robots to develop a patient-cooperative strategy with the capacity to understand the intention of the user and provide suitable rehabilitation training. In this paper, we present an upper-limb motion pattern recognition method using surface electromyography (sEMG) signals with a support vector machine (SVM) to control a rehabilitation robot, ReRobot, which was built to conduct upper-limb rehabilitation training for post-stroke patients. For poststroke rehabilitation training using the ReRobot, the upper-limb motion of the patient's healthy side is first recognized by detecting and processing the sEMG signals; then, the ReRobot assists the impaired arm in conducting mirror rehabilitation therapy. To train and test the SVM model, five healthy subjects participated in the experiments and performed five standard upper-limb motions, including shoulder flexion, abduction, internal rotation, external rotation, and elbow joint flexion. Good accuracy was demonstrated in experimental results from the five healthy subjects. By recognizing the model motion of the healthy side, the rehabilitation robot can provide mirror therapy to the affected side. This method can be used as a control strategy of upper-limb rehabilitation robots for self-rehabilitation training with stroke patients.

#### Edited by:

*Changhong Fu, Tongji University, China*

#### Reviewed by:

*Zhaohui Xia, Rensselaer Polytechnic Institute, United States Dongdong Zheng, National University of Singapore, Singapore Qinyuan Ren, Zhejiang University, China*

#### \*Correspondence: *Longhan Xie*

<EMAIL_ADDRESS>
Received: *12 February 2019* Accepted: *09 May 2019* Published: *04 June 2019*

#### Citation:

*Cai S, Chen Y, Huang S, Wu Y, Zheng H, Li X and Xie L (2019) SVM-Based Classification of sEMG Signals for Upper-Limb Self-Rehabilitation Training. Front. Neurorobot. 13:31. doi: 10.3389/fnbot.2019.00031* Keywords: surface electromyography, support vector machine, rehabilitation robot, upper limb, motion pattern recognition

#### INTRODUCTION

Stroke is the leading cause of adult disability around the world (Burton et al., 2017), with upper-limb motor impairments being the main factor influencing the quality of life in stroke survivors (Stinear et al., 2017). Repetitive motor training on movement has a notable curative effect on the restoration of arm function in stroke patients, and the patients' degree of recovery is positively influenced by treatment intensity (Steven et al., 2006; Gittler and Davis, 2018). Conventionally, stroke patients usually rehabilitate with the assistance of therapists. However, the involvement of therapists is challenging because rehabilitation training is a time-consuming and labor-intensive process. Many stroke survivors experience upper-limb impairment with few rehabilitation opportunities due to a lack of rehabilitation therapists. Robotassisted therapy devices, which can provide the affected arm with high intensity and repetitive treatment, have been increasingly used in rehabilitation training and can potentially enhance upper-limb functional recovery in stroke survivors (Yoo and Kim, 2015; Veerbeek et al., 2017).

Various rehabilitation robotic devices have been developed for upper-limb training in stroke patients. Among them, MIT-Manus (Krebs et al., 1999) was one of the first systems to be developed and can provide stroke survivors with plane movements. Furthermore, MIME (Lum et al., 2006), GENTLE/s (Coote et al., 2008), T-WREX (Domien et al., 2011), and NEREBOT (Stefano et al., 2014) were proposed to permit threedimensional exercise training for patients with impaired arms.

Different control strategies have been developed and applied to rehabilitation robots for the recovery of the affected arm. Motion parameters of the patient's arm are one of the major inputs in the rehabilitation robot's control system. Many types of mechanical inputs, such as switches (e.g., Aubin et al., 2013; Artz et al., 2015), force sensors (e.g., Diftler et al., 2014), and computer vision (e.g., Taati et al., 2012), have been used as feedback in the controllers of rehabilitation robots. The surface electromyography (sEMG) signal, which is composed of the action potentials from groups of muscle fibers, is one of the major sources of information about neural control and can reflect the degree of activity of the muscles (Yang et al., 2016). During the rehabilitation training of the upper limb, sEMG signals can be captured, interpreted, and used as input for the control algorithms of rehabilitation robots (Rosen et al., 2001; Kiguchi and Hayashi, 2012; Peternel et al., 2016). Considering that rich motor control information and the user's intention can be detected from sEMG signals, the sEMG-based control scheme is one of the most appropriately suited approaches for upper-limb rehabilitation robots (Singh et al., 2014).

However, the sEMG signal is affected by many factors and is not stable, which can lead to low accuracy in recognizing patient motion intentions. Some studies have shown that machine learning techniques can be employed for classifying different tasks and improving the robustness and accuracy of the identification and classification of arm movements through the exploitation of sEMG signals (Lucas et al., 2008; Young et al., 2013; Suberbiola et al., 2015). The SVM algorithm is a well-established technique to learn how to classify new data starting from a collection of classified events and has been widely applied in machine learning problems (Vapnik, 1995; Suykens et al., 2015) and sEMG processing (Song et al., 2007) because of its simplicity and robustness. With the determination of a few additional tuning parameters, SVM solutions are characterized by a convex quadratic optimization problem (Platt, 1999). Considering that the availability and quality of sEMG signals can vary from patient to patient, it is difficult to obtain a large number of training samples. SVM is suitable for solving learning tasks where the number of attributes is large relative to the number of training examples (Suykens et al., 2002).

Aimed at developing a control strategy for upper-limb rehabilitation robots with the capacity to understand the intention of the patients and provide the corresponding rehabilitation training, this paper proposed an sEMG-based control framework based on SVM classifiers for intention identification of the upper limb. The control strategy was applied to the upper-limb rehabilitation robot, ReRobot, to perform the rehabilitative exercise training. The motion of the patient's healthy side is first recognized from the measured and processed sEMG signals, and then the ReRobot assists the affected side in conducting the corresponding rehabilitation therapy. Based on the developed sEMG-based control strategy, self-rehabilitation training in stroke patients can be conducted.

## METHODS

#### Data Collection

Many stroke patients have trouble moving their upper limbs on the affected side, and they must receive much rehabilitation training to recover motion ability (Merletti et al., 1999). Stroke patients often show abnormal shoulder motor ability, so shoulder rehabilitation actions, including shoulder forward flexion, shoulder level adduction, and shoulder level abduction (Chen and Zhou, 2015), should be carried out.

The coordinates were defined where the coronal axis of the patient is the X-axis, the sagittal axis is the Y-axis, the vertical

axis is the Z-axis, and the acromioclavicular joint is the origin of the coordinates. Five motions were used to test the performance of this model, including shoulder flexion, abduction, internal rotation, external rotation, and elbow joint flexion, as shown in **Figure 1**.

Five healthy subjects (N = 5, age 25 ± 4 years, body mass 70 ± 5 kg, height 174 ± 6 cm, all male and all right-handed) participated in the experiments. All subjects gave their informed consent before participation. The experimental procedures were conducted in accordance with the Declaration of Helsinki and approved by the Ethic Board of Medical School, South China University of Technology. Each subject performed five repetitions in accordance with five standard motions. Since stroke patients are mostly elderly people with lower motion abilities on their healthy side compared to young adults, the designed motions were imitated as movements of elderly stroke patients on their healthy side.

For each test, the test subject did not carry weight, and the movement lasted for 1–2 s. After the end of each movement, the subject took at least 1 min to rest to prevent muscle fatigue. There are many muscles involved in the movement of the shoulder and elbow joints. Muscles play two roles in the movement process: proximal stability and distal activity. In this paper, eight superficial muscles involved in the distal shoulder and elbow movements were selected as monitoring objects. To ensure safety, patients controlled the emergency stop of the equipment according to their own comfort level. In this paper, the patient's clenched fist is used as the emergency stop action, and the flexor radialis is used as the detection channel for the emergency stop action. Therefore, a total of nine muscles were selected as test objects. The sEMG signals of nine muscles in the upper limb were acquired, including the middle deltoid, anterior deltoid, pectoralis major, biceps brachii, brachioradial muscle, ulnar flexor carpal, trapezius, posterior deltoid, and triceps brachii. The first eight muscles were used to evaluate how the muscle works while the signal changes. The ulnar flexor carpal played a role in the subject's self-initiation of the safety protection mechanism. When the subject felt uncomfortable during the test, he or she could stop the robot by clenching his or her fist and activating the ulnar flexor.

**Figure 2** shows the experimental setup. A 16-channel sEMG acquisition instrument with 1-kHz sampling frequency in each channel was used. Each channel was related to a three-channel differential electrode. After the muscle was disinfected by alcohol, the electrodes were placed along the direction of the muscle abdomen with an interval of 2 cm. **Figure 3** shows the raw sEMG signals of the eight muscles without preprocessing from one of the five healthy individuals.

## Data Processing

#### Preprocessing

sEMG signals are easily disturbed by the external environment in the acquisition process. Motion artifacts, baseline offset, and power frequency interference may all lead to distortion of the sEMG signals, which leads to poor classification accuracy (Chen et al., 2015). Data preprocessing methods of baseline correction, 20–500 Hz bandpass filter, power frequency filter (50 Hz notch), full-wave rectification, and amplitude normalization were carried out to improve the signal-to-noise ratio (SNR) of the sEMG signal. All filters used in this paper are fourth-order Butterworth filters.

#### Signal Segmentation

Although the intensity of the sEMG signal detected in each channel is different in the different movements, the signals show good synchronization (Zhang and Zhou, 2012): if the related muscles did not contract, then the sEMG signal showed a stable low-amplitude signal before the test; in contrast, the signal changed dramatically in the course of executing the action. The characteristic of this type of signal was that the signal could be segmented by a sample entropy algorithm (Liu and Zhou, 2013). The sample entropy algorithm is an efficient and time-consuming algorithm that can avoid the signal deviation caused by self-matching. Therefore, we adopted the sample entropy algorithm for data segmentation.