Robust On-Line ADP-based Solution of a Class of Hierarchical Nonlinear Differential Game
Mohammad reza Satouri Hamed Kebriaei Abolhassan Razminia Mohammad javad Yazdanpanah
eess.SY cs.SY 91A65, 49N70, 68T05,
















































Robust On-Line ADP-based Solution of a Class of Hierarchical Nonlinear Differential Game
    Mohammad Reza Satouri, 
             Hamed Kebriaei, Senior Member, IEEE,
        Abolhassan Razminia 
        and Mohammad Javad Yazdanpanah
        
M. R. Satouri and H. Kebriaei are with the School of Electrical and Computer Engineering, College of Engineering, University of Tehran, P.O. Box 4395-515, Tehran, Iran (e-mail: satouri.mr@ut.ac.ir ; kebriaei@ut.ac.ir ).


A. Razminia is with the Dynamical Systems & Control (DSC) Research Lab.,
Department of Electrical Engineering, School of Engineering, Persian Gulf University, P.O. Box 75169, Boushehr, Iran (e-mail: razminia@pgu.ac.ir).  


M. J. Yazdanpanah is with the School of Electrical and Computer Engineering, College of Engineering, University of Tehran, P.O. Box 4395-515, Tehran, Iran, and also with the Control & Intelligent Processing Center of Excellence (e-mail: yazdan@ut.ac.ir).
     ^⋆  Laboratoire d'Informatique et des Systèmes, Université de Toulon, Aix Marseille Univ, CNRS UMR 7020, LIS, Marseille, France

boizot@univ-tln.fr; 

^†  Department of Mathematics, Lehigh University, Bethlehem, PA, USA. 

lus219@lehigh.edu; 

 

This research has been partially supported by the ANR SRGI (reference ANR-15-CE40-0018). 


This is a post-peer-review, pre-copyedit version of an article published in Journal of Mathematical Imaging and Vision (JIMIV). The final authenticated version is available online at: <http://dx.doi.org/10.1007/s10851-020-00990-5>

   
      
   

 
 

=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================






In this paper a hierarchical one-leader-multi-followers game for a class of continuous-time nonlinear systems with disturbance is investigated by a novel policy iteration reinforcement learning technique in which, the game model consists both of the zero-sum and nonzero-sum games, simultaneously. An adaptive dynamic programming (ADP) method is developed to achieve optimal control strategy under worst case of disturbance. This algorithm reduces the number of neural networks which are used for estimation for about thirty percent. Proposed algorithm uses neural networks to estimate value functions, control policies and disturbances. Convergence analysis of the estimations is investigated using Lyapunov theory and exploiting properties of the Nemytskii operator. Finally the simulation results will show effectiveness of developed ADP method. 




Adaptive dynamic programming (ADP); Optimal control; Nonlinear systems; Hierarchical game; Nemytskii operator; Neural networks; Reinforcement learning.       






theoremTheorem
definition[theorem]Definition
remark[theorem]Remark
lemma[theorem]Lemma














§ INTRODUCTION
















Nowadays, the optimal control theory has penetrated in various disciplines from aerospace engineering to resource management <cit.>. Based on its merits and importance, the scholars have developed different approaches to study the dynamic optimization problems <cit.>. One of the main trends in optimal control systems has been provided by Bellman, namely the optimality principle, and now extended to more complex versions. The adaptive dynamic programming (ADP) is one of these extensions whose essential root is the Hamilton-Jacobi-Bellman (HJB) equation <cit.>.





Inspiring by the reinforcement learning (RL) techniques, the ADP resolves  some drawbacks of the classical dynamic programming (DP) like curse of dimensionality <cit.>. As a matter of fact, the kernel of the operation of ADP is  estimating the cost functional by a function approximating mechanism and then solving the DP problem forward in time. Based on its efficient performance, the ADP is widely used in numerous optimal control problems such as power system control <cit.>, battery management <cit.> and autopilot systems <cit.>.





In a multi-agent system, when the control strategy of an agent affects the cost function of other players (e.g. through system's dynamic) one can model the problem as a dynamic game rather than a single optimal control problem. Usually, in a dynamic game with non-quadratic cost, nonlinear or unknown dynamics, the closed form of the solution which is known as the equilibrium point of the game, cannot be found and therefore, the ADP method can be utilized as a promising tool to estimate the solution.





Dynamic games can be classified based on the sum of the cost functions into zero-sum (ZS) and nonzero-sum (NZS) games <cit.>.

In ZS games, two major approaches have been proposed in <cit.> where the underlying system is linear: the on-line algorithm based on the integral RL in which  some partial information of the dynamical system is needed, and the off-line algorithm based on the policy iteration (PI) wherein full information of the system is utilized. The extension of the approach for an unknown nonlinear dynamical system has been studied in <cit.>. In <cit.>, an ADP method has been proposed for input constraint ZS dynamic game. For the linear systems with partially measurable states in presence of various matched uncertainties, a robust ADP method for the ZS game has been explored in
<cit.>.
An on-line iterative ADP algorithm has been proposed in <cit.> with known disturbance matrix and unknown state and control matrices.





In contrast to the ZS games, where the goals of the players are fully opposed and the equilibrium point is known as the saddle point, in NZS games the equilibrium strategy of the game is defined as a strategy profile of the players in which, no player can lower its cost function by changing its control strategy, while the other players do not change their strategy. This equilibrium strategy is also known as Nash equilibrium point. In <cit.>, a PI-ADP approach for a NZS game with nonlinear discrete-time dynamic has been proposed and both on-line and off-line algorithms has been investigated. An on-line method for multi-player NZS games which requires complete knowledge of system dynamics has been presented in <cit.>. In <cit.>, an off-policy IRL method for a continuous-time multiplayer NZS game with unknown dynamics has been explored. Utilizing the actor-critic-identifier (ACI), without full information of the system dynamics is another important approach in NZS games. This technique is adopted in <cit.> to approximately solve a set of coupled HJB equations of the proposed N player NZS game.




All of the mentioned research works which have used the ADP method to solve dynamic games, only considered the case that the players decide simultaneously. However, the problem becomes more complicated when the players decide in hierarchical scheme. In dynamic Nash games where the strategy of each player affects other players' costs and strategies, indirectly through the state of the system and the players have access to the state measurement, the Nash optimal strategy of the game can be obtained using ADP to solve HJB equations of the players, simultaneously.

 However, in a hierarchical (e.g. one-leader-multi-followers) game, since the strategies of the followers depend both on the strategy of the leader and the state of the system, finding the Stackelberg-Nash equilibrium point of the game and the proof of convergence of the ADP algorithm to the equilibrium point of the game become more complicated.
 
 
 
 
The basic type of hierarchical games is the leader-follower or Stackelberg game which was introduced by H. von Stackelberg in 1934 <cit.>. In this game, the leader decides first at the upper level (level 1), and in the lower level (level 2), the follower decides upon the decision of the leader is revealed <cit.>. The leader-follower game has widespread applications in many engineering fields like smart grids and cognitive radio network <cit.>.




Dynamic Stackelberg games are well studied in the case of linear dynamic and quadratic cost <cit.>. However, studies on dynamic Stackelberg games with nonlinear dynamic and non quadratic cost are fairly scarce.
In <cit.> the authors proposed a heuristic iterative algorithm for solving dynamic Stackelberg game, however the convergence of the proposed algorithm was not proven.
In <cit.> discrete time dynamic Stackelberg games with general form of dynamic and cost is studied and dynamic programming approach is used to derive the Bellman equation and find the optimal solution. However, for continuous time systems the Bellman optimality principle leads to the well known HJB equation. In non-LQ continuous time problems like our case, the HJB equation cannot be solved analytically and ADP is a promising approach which can be utilized to obtain the value function and optimal strategy.




The ADP method has been utilized to solve non-LQ dynamic Nash-games in the literature like <cit.>, nevertheless, to the best of authors' knowledge, the ADP has not been explored to solve nonlinear-non-quadratic Stackelberg games up to now. In this paper, an on-line ADP method is proposed to obtain the equilibrium point of a group of dynamic continuous time hierarchical nonlinear non-quadratic games. Further, the proposed algorithm is robust with respect to uncertainty which is considered as an exogenous disturbance. In the upper level (level 1) of the game, the leader acts first and plays a Stackelberg game with the followers. At the lower level (level 2), the followers play an N-player non-cooperative Nash game and decide simultaneously after the leader. All the players also play a ZS game with the disturbance.




Because of the non-simultaneous decision making in our proposed game, the strategy of the leader appears in
the Nash strategy of the followers, since the best response of each follower is a function of both the current state and the strategy of the leader. Such dependency imposes more difficulties when we want to calculate the equilibrium strategies of the players and also to prove the convergence of the ADP algorithm to the equilibrium point of the game. To overcome this issue, we have used Nemytskii operator and neural networks.
 In the proposed ADP algorithm, neural networks are used in actor-critic scheme. Update rules for critic and actor networks are obtained using gradient descent and adaptive control, respectively. Convergence analysis of the ADP method to equilibrium point of the game is performed using Lyapunov theory and exploiting the properties of Nemytskii operator.  In addition, the number of neural networks used in actor-critic method is reduced with respect to similar works <cit.> which facilitates the training procedure and speeds up the convergence.
 
 
 

The contribution of this paper can be summarized as follows:


  * Using an ADP-based method to solve a class of differential hierarchical games with nonlinear dynamic and non-quadratic cost in presence of disturbance for the first time.

  * Convergence analysis of ADP algorithm to the Stackelberg-Nash-Saddle equilibrium point of the proposed game model by guaranteeing the uniform ultimate boundedness (UUB) condition.

  * Using Nemytskii operator in the ADP-based methods for the first time to overcome the complexity of computing the leader's optimal strategy.






The organization of the paper is as follows: In Section 2 the problem is formulated and the proposed game model is discussed. Some preliminaries are reviewed in Section 3.  Section 4 devotes to the main results in which the final solution is obtained in three serial steps. A numerical simulation in Section 5, verifies the obtained theoretical results. Concluding remarks are presented in Section 6.
















§ SYSTEM MODEL AND PROBLEM FORMULATION

 


The general scheme and information structure of the proposed hierarchical game in this paper is adopted from <cit.> where the authors studied discrete time linear-quadratic hierarchical dynamic games. The game consists of N+2 players: one leader, N followers and a disturbance. The players decide in a dynamic environment. The sate dynamic equation of the system is known to all players and also they have closed loop information about the state of the system. The strategy of all players appear in the state dynamic equation of the system and therefore, the objective function of all players are coupled indirectly through the state of the system. The disturbance is considered as a virtual player who wants to provide the worst case condition for the objective function of all players. In the lower level (level 2) of the proposed hierarchy, the followers play a dynamic Nash game with each other and also play a zero-sum game with the disturbance. The followers decide simultaneously and they don't have information about other followers' costs and strategies, while they know the leader's strategy. At this level, each follower independently updates its value function and  best response strategy  to the other followers' strategies and disturbance. In the higher level (level 1) of the game, the leader uses the obtained strategies of the followers as the reaction of the followers to play a ZS game with disturbance and find its min-max optimal strategy. Since there is a Stackelberg, a Nash and (N+1) ZS games in this model, the equilibrium point of the whole game is called Stackelberg-Nash-saddle equilibrium. A schematic diagram of the proposed game structure is shown in Fig. <ref>.

Consider a dynamical system with one leader and N followers playing over a state space whose evolution dictated by  the following differential equation:

    𝐱̇(t)=𝐟(𝐱)+∑_j=1^N g_j(𝐱)𝐮_j +p(𝐱)ν +h(𝐱)ω

where 𝐱(t) ∈𝒳⊂ℝ^n, 𝐮_j(t) ∈𝒰_j⊂ℝ^m_j, ν(t) ∈𝒩⊂ℝ^α , and ω(t) ∈𝒲⊂ℝ^w, are state vector, controls or actions of followers,  control or action of leader, and disturbance, respectively. Moreover, 𝐟: ℝ^n→ℝ^n is a Lipschitz vector function with 𝐟(0)=0, and g_j: ℝ^n→ℝ^n× m_j, j=1,...N, h: ℝ^n →ℝ^n×α, p: ℝ^n→ℝ^n× w are some continuous matrices. In addition, similar to <cit.>, it is assumed that ‖ g_i(𝐱)‖ < b_i for 1≤ i≤ N , ‖ h(𝐱) ‖ <b_h and ‖ p(𝐱) ‖ <b_p. The leader's decision is based on followers' rationality. All players make decisions in such way to minimize their cost functionals in the worst case scenario generated by the disturbance with attenuation level γ as follows:

    V_i   =∫_t^∞( Q_i(𝐱(τ))+∑_j=1^N 𝐮_j^T(τ)R_ij𝐮_j(τ) 
            +ν^T(τ)S_iν(τ) -γ^2ω^T(τ)ω(τ) ) dτ
       = ∫_t^∞r_i(𝐱(τ), 𝐮_1(τ), 𝐮_2(τ), … , 𝐮_N(τ), ν(τ) , ω(τ)) dτ

where 1≤ i≤ N are followers and i=N+1 is the leader. Functions Q_i(𝐱(t))>0 are generally nonlinear and satisfy Q_i(𝐱)≥ϑ_i𝐱^T𝐱 for certain positive constants ϑ_i, γ is a scalar, and S_i>0, R_ii>0, R_ij≥ 0 are known and symmetric matrices. In the subsequent, the parameter t is dropped for the sake of simplicity.




Based on the given points, the goal is to find a set of state feedback control policies which minimize cost functions in the worst case strategy of disturbance player. In the subsection 3.1 we solve the game in the lower level (level 2) for followers and then the obtained solution is used in subsection 3.2 to solve the leader's problem in the upper level (level 1) and complete the solution of the proposed bi-level game. Then in the subsection 3.3, the update rules for critic and actor networks are introduced and the convergence of the proposed algorithm is proven.







§ PRELIMINARY DEFINITIONS




An N-tuple of strategies {γ^1*,γ^2*,...,γ^N*} with γ^i*∈Γ^i, where Γ^i is ith player strategy set, i∈ℤ^+, is said to constitute a Nash equilibrium solution for N-person nonzero-sum finite game in extensive form, if the following N inequalities are satisfied for all γ^i∈Γ^i, i∈ℤ^+ :

    J^i(γ^1*,...,γ^N*)≤ J^i(γ^1*,...,γ^(i-1)*,γ^i,γ^(i+1)*,...,γ^N*)

The N-tuple of quantities { J^1*,J^2*,...,J^N*} is known as a Nash equilibrium outcome of the nonzero-sum finite game in extensive form.








In a Stackelberg game a player (leader) has the ability to enforce his strategy on the other player (follower). In this game leader acts first and then follower reacts to leader's decision. A strategy γ^1*∈Γ^1 is called a Stackelberg equilibrium strategy for the leader, if

    J^1*   =max_γ^2∈ R^2(γ^1*) J^1(γ^1*, γ^2)
       =min_γ^1∈Γ^1max_γ^2∈ R^2(γ^1)J^1(γ^1, γ^2)

where Γ^1 , J^1(γ^1, γ^2) and Γ^2 , J^2(γ^1, γ^2) are pure strategy spaces and costs of leader and follower, respectively. Also the set R^2(γ^1) ⊂Γ^2 defined for each γ^1∈Γ^1 by

    R^2(γ^1)={ξ∈Γ^2 : J^2(γ^1, ξ)≤ J^2(γ^1, γ^2),  ∀γ^2∈Γ^2}

is the optimal response set of follower to the strategy γ^1∈Γ^1 of leader.


Consider a game with two players who decide in a dynamic environment as follows

    ẋ(t)=f(t, x(t), u^1(t), u^2(t))

and the cost functionals of the players ∀ t ∈ [t_0,T] defined as:

    L^i(u^1, u^2)=∫_t_0^Tg^i(t, x(t), u^1(t), u^2(t))dt+q^i(x(T))

 where [t_0, T] denotes the prescribed duration of the game. If the players decide simultaneously, then the game is called differential Nash game or simply differential game and if they decide in leader-follower scheme, the game is called differential Stackelberg game. 



















Consider a set Ψ which is a measure or metric space and let X and Y  be two Hausdorff topological spaces. The Nemytskii operator is defined by:

    N_f(𝐮)(𝐳)≜ f(𝐳,𝐮(𝐳))   ∀𝐳∈Ψ

in which  𝐟:Ψ× X→ Y.





As can be seen from the definition, the Nemytskii operator assigns the Y-valued 𝐳⟼ f(𝐳,𝐮(𝐳)) to each function 𝐮: Ψ→ X. Another concept is the uniformly ultimately boundedness (UUB) property of a signal.





The solutions of the nonlinear dynamical system 𝐱̇(t)=𝐟(t,𝐱) with 𝐱(t_0)=𝐱_0, are uniformly ultimately bounded with ultimate bound b if there exists a positive constant c , independent of t_0, and for every a∈ (0,c), there is a T(a,b)≥ 0, such that:

    ‖𝐱(t_0)‖≤ a   ⇒  ‖𝐱(t)‖≤ b,  ∀ t≥ t_0+T .








§ MAIN RESULTS









According to cost functionals (<ref>), the optimal value functions for all the players in presence of disturbance are obtained using min-max optimization as follows:

    V_i^⋆(𝐱, 𝐮_∙, ν, ω_i) =min_𝐮_imax_ω_i∫_t^∞( Q_i(𝐱) 
    
    +∑_j=1^N𝐮_j^TR_ij𝐮_j+ν^TS_iν -γ^2ω_i^Tω_i) dτ


    V_N+1^⋆(𝐱, 𝐮_∙, ν , ω_N+1) =min_νmax_ω_N+1∫_t^∞( Q_N+1(𝐱)
          +∑_j=1^N𝐮_j^TR_(N+1)j𝐮_j+ν^TS_N+1ν -γ^2ω_N+1^Tω_N+1) dτ

where 𝐮_∙={𝐮_1, 𝐮_2, … , 𝐮_N} and 1≤ i≤ N.
For the optimal set 𝐮_∙^*,ν^*,ω^* the  HJB equation holds as follows:

    ℋ_i(𝐱,∇_𝐱 V_i^*, 𝐮_∙^*, ν^* , ω^* ) +∂ V_i/∂ t =0

where

    ℋ_i(𝐱,∇_𝐱 V_i, 𝐮_∙, ν , ω )=r_i(𝐱, 𝐮_∙, ν , ω )+(∇_𝐱 V_i)^T( 𝐟(𝐱) 
            +∑_j=1^N g_j(𝐱)𝐮_j +h(𝐱)ω+p(𝐱)ν)






In proceeding the aim is to find the optimal policies of followers and leader and then solving the whole game. Also from now on, the subscript 𝐱 in ∇_𝐱 is dropped for the sake of simplicity.




 §.§ Level 2: Followers-Disturbance Game













According to (<ref>) the cost functionals are convex and concave with respect to controls and disturbance, respectively. The minimum and maximum of the Hamiltonian can be directly computed by the stationary condition as follows:


    ∂ℋ_i/∂𝐮_i   =0   ⇒  𝐮_i^⋆(𝐱)=-1/2R_ii^-1g_i^T(𝐱)∇ V_i
    ∂ℋ_i/∂ω_i   =0   ⇒  ω_i^⋆(𝐱)=1/2γ^-2h^T(𝐱)∇ V_i

Substituting (<ref>) and (<ref>) into (<ref>), N coupled Hamilton-Jacobi equations can be obtained:

    Q_i(𝐱)+1/4∑_j=1^N (∇ V_j)^Tg_j(𝐱)R_jj^-1R_ijR_jj^-1g_j^T(𝐱)∇ V_j
        -γ^-2/4(∇ V_i)^Th(𝐱)h^T(𝐱)(∇ V_i)+ ν^TS_iν
        +(∇ V_i)^T( 𝐟(𝐱)+∑_j=1^N g_j(𝐱)𝐮_j+p(𝐱)ν +h(x)ω)=0

Since neither V_i's nor optimal control strategies can be computed in a closed form, a conventional way is to approximate the value functions and control strategies with general function approximators (GFAs). A common tool that is employed in implementing the GFA, is neural networks.
Therefore, the solutions of (<ref>) on a compact set Ω⊆ℝ^n which contains the origin are approximated by NNs as follows:

    V_i(𝐱,ν)=𝐖_v_i^Tϕ_v_i(𝐱,ν)+ϵ_v_i(𝐱,ν),    1≤ i≤ N,

where ϕ_v_i(.,.)∈ℝ^κ and ϵ_v_i are NN activation functions and NN approximation errors, respectively. In <cit.> it has been shown that always there exists a number of hidden-layer neurons which makes the controls admissible and ϵ_v_i's tend to zero as the number of hidden-layer neurons tends to infinity. It should be mentioned that because of hierarchical nature of the game,  V_is , 1≤ i≤ N are obtained as functions of 𝐱 and ν. After calculating ν for the leader, the value functions would obtain in terms of 𝐱 and then, it is be substituted in control policies of the followers.



In proceeding the new variables:

    C_j^i   =∇ϕ_v_jg_j(𝐱)R_jj^-1R_ijR_jj^-1g_j^T(𝐱)∇ϕ_v_j^T
    
    E_i   =γ^-2∇ϕ_v_ih(𝐱)h^T(𝐱)∇ϕ_v_i^T
    
    D_i^j   =∇ϕ_v_ig_j(𝐱)R_jj^-1g_j^T(𝐱)∇ϕ_v_j^T

are introduced for simplicity. Substituting the value function approximations from (<ref>) into (<ref>) yields:

    1/4∑_j=1^N𝐖_v_j^TC_j^i𝐖_v_j-1/4𝐖_v_i^TE_i𝐖_v_i+Q_i(𝐱)+ν^TS_iν
          +𝐖_v_i^T( ∇ϕ_v_i𝐟(𝐱)-1/2∑_j=1^N D_i^j𝐖_v_j +∇ϕ_v_ip(𝐱)ν
          +1/2 E_i𝐖_v_i)=ϵ_HJ_i(𝐱,ν)

where ϵ_HJ_i(𝐱,ν) is the error incurred by replacing the actual value functions with the approximated ones (<ref>). Now, according to (<ref>), the equations (<ref>) and (<ref>) can be rewritten as:

    𝐮_i(𝐱,ν)   =-1/2R_ii^-1g_i^T(𝐱)∇ϕ_v_i^T(𝐱,ν)𝐖_v_i ,    1≤ i≤ N 
    ω_i(𝐱,ν)   =1/2γ^-2h^T(𝐱)∇ϕ_v_i^T(𝐱,ν)𝐖_v_i ,    1≤ i≤ N

Since the ideal weights 𝐖_v_i's are unknown, V_is and the control policies and disturbances should be defined in the form of critic and action neural networks, respectively as follows for 1≤ i≤ N:

    V̂_i(𝐱,ν)   =𝐖̂_v_i^Tϕ_v_i(𝐱,ν), 
    𝐮̂_i(𝐱,ν)   =-1/2R_ii^-1g_i^T(𝐱)∇ϕ_v_i^T(𝐱,ν)𝐖̂_a_i, 
    ω̂_i(𝐱,ν)   =1/2γ^-2h^T(𝐱)∇ϕ_v_i^T(𝐱,ν)𝐖̂_a_i

Substituting (<ref>) and (<ref>) in (<ref>), the following expression is obtained:

    1/4∑_j=1^N𝐖̂_a_j^T C_j^i𝐖̂_a_j-1/4𝐖̂_a_i^T E_i𝐖̂_a_i+Q_i(𝐱)+ν̂^TS_iν̂
          +𝐖̂_v_i^T( ∇ϕ_v_i𝐟(𝐱)-∑_j=1^N1/2D_i^j𝐖̂_a_j+∇ϕ_v_ip(𝐱)ν̂
          +1/2E_i𝐖̂_a_i)=e_i(𝐱,ν)

where e_i(𝐱,ν) is the error incurred by replacing ideal weights of NNs with their approximate values.







 §.§ Level 1: Leader-Disturbance Game






Similar to the previous subsection, the value function and critic NN of the leader can be approximated as follows:

    V_N+1(𝐱)   =𝐖_v_N+1^Tϕ_v_N+1+ϵ_v_N+1(𝐱)  
    V̂_N+1(𝐱)   =𝐖̂_v_N+1^Tϕ_v_N+1(𝐱)

After applying stationary condition of ω to the leader's Hamiltonian function similar to (<ref>), one can get:

    ω̂_N+1(𝐱)=1/2γ^-2h^T(𝐱)∇ϕ_v_N+1^T(𝐱)𝐖̂_a_N+1







Substituting (<ref>) and (<ref>) into the leader's Hamiltonian, one can get:

    1/4∑_j=1^N𝐖_v_j^TC_j^N+1𝐖_v_j-1/4𝐖_v_N+1^TE_N+1𝐖_v_N+1
       +ν̌^TS_N+1ν̌+Q_N+1(𝐱)+𝐖_v_N+1^T(∇ϕ_v_N+1𝐟(𝐱)
       -1/2∑_j=1^N D_N+1^j𝐖_v_j+∇ϕ_v_N+1p(𝐱)ν̌
       +1/2E_N+1𝐖_v_N+1) =ϵ_HJ_N+1(𝐱,ν̌)

and

    1/4∑_j=1^N𝐖̂_a_j^TC_j^N+1𝐖̂_a_j -1/4𝐖̂_a_N+1^TE_N+1𝐖̂_a_N+1
       + ν̂^TS_N+1ν̂+Q_N+1(𝐱)+𝐖̂_v_N+1^T(∇ϕ_v_N+1𝐟(𝐱)
       -1/2∑_j=1^N D_N+1^j𝐖̂_a_j+∇ϕ_v_N+1p(𝐱)ν̂
       +1/2E_N+1𝐖̂_a_N+1)=e_N+1(𝐱,ν̂)

where ν̌ and ν̂ are functions of exact and approximated NNs weights, respectively. In proceeding new variables

    B_j^N+1   =g_j(𝐱)R_jj^-1R_(N+1)jR_jj^-1g_j^T(𝐱)
    Π̌   =p^T(𝐱)∇ϕ_v_N+1^T𝐖_v_N+1
    Π̂   =p^T(𝐱)∇ϕ_v_N+1^T𝐖̂_v_N+1
    Ǎ_j^N+1   =𝐖_v_j𝐖_v_N+1^T∇ϕ_v_N+1g_j(𝐱)R_jj^-1g_j^T(𝐱)
    Â_j^N+1   =𝐖̂_a_j𝐖̂_v_N+1^T∇ϕ_v_N+1 g_j(𝐱)R_jj^-1g_j^T(𝐱)

are introduced for simplicity.
Now stationary condition of ν̌ is applied to (<ref>):







    ∂ℋ_N+1/∂ν̌   =Π̌+[1/2∑_j=1^N( ∂∇ϕ_v_j(𝐱,ν̌)/∂ν̌)^T𝐖_v_j𝐖_v_j^T
       ×∇ϕ_v_j(𝐱,ν̌)B_j^N+1-Ǎ_j^N+1] +2S_N+1ν̌=0

Thus






    ν̌   =-1/2 S_N+1^-1[ -1/2∑_j=1^N( ∂∇ϕ_v_j(𝐱,ν̌)/∂ν̌)^T(Ǎ_j^N+1
          -𝐖_v_j𝐖_v_j^T∇ϕ_v_j(𝐱,ν̌)B_j^N+1)+Π̌]

similar to (<ref>):

    ν̂   =-1/2 S_N+1^-1[ -1/2∑_j=1^N( ∂∇ϕ_v_j(𝐱,ν̂)/∂ν̂)^T(Â_j^N+1
          -𝐖̂_a_j𝐖̂_a_j^T∇ϕ_v_j(𝐱,ν̂)B_j^N+1)+Π̂]

To obtain the optimal strategy of the leader, we need to solve (<ref>) for ν̂. However, (<ref>) cannot be solved analytically with respect to ν̂. Therefore, to obtain the solution iteratively, we rewrite (<ref>) as an iterative equation in which, the
fixed point of the iterative equation is the optimal strategy of the leader:



    ν̂_k+1   =-1/2 S_N+1^-1[ -1/2∑_j=1^N( ∂∇ϕ_v_j(𝐱,ν̂_k)/∂ν̂_k)^T(Â_j^N+1
          - 𝐖_v_j𝐖_v_j^T∇ϕ_v_j(𝐱,ν̂_k)B_j^N+1)+Π̂]

To prove existence and uniqueness of the fixed point of iterative equation (<ref>), the Nemytskii operator is utilized and(<ref>) is rewritten as follows:

    ν̂(𝐱)=Ξ(𝐱,ν̂(𝐱))+ξ(𝐱)

where

    ξ(𝐱)=-1/2 S_N+1^-1p^T(𝐱)∇ϕ_v_N+1^T𝐖̂_v_N+1,

where ξ(𝐱)∈ W^1,β(Ω), 1≤β≤∞ and W^1,β(Ω) is a Sobolev space and Ξ: Ω×ℝ^α→ℝ^α is a Caratheodory function such that the Nemytskii operator N_Ξ is continuous and bounded <cit.>. In the proceeding lemma the condition in which the whole game has a unique Stackelberg-Nash-saddle equilibrium is introduced.

Equation (<ref>) has a unique solution ν̂∈ W^1,β(Ω) if Lipschitz coefficient ‖Ξ‖_Lip is less than one.


Suppose that there exist solutions ν̂_1 and ν̂_2, so from (<ref>) it is concluded that Ξ(𝐱,ν̌_1)-Ξ(𝐱,ν̂_2)=ν̂_1-ν̂_2. Thus

    Ξ(𝐱,ν̂_1)-Ξ(𝐱,ν̂_2)=ν̂_1-ν̂_2≤‖Ξ‖_Lip( ν̂_1-ν̂_2)

Now if the Lipschitz coefficient ‖Ξ‖_Lip is less than one, the assumption of existing more than one solution is incorrect.




According to the condition in Lemma <ref>, the Nemytskii operator N_Ξ has a unique fixed point and then  ν̂ can be calculated iteratively. The following NN is used for approximating ν̂:











    ν̂=Ŵ_ν^Tϕ_ν

where Ŵ_ν∈ℝ^μ×α and ϕ_ν∈ℝ^μ. Then the gradient descent is used for updating Ŵ_ν:






    Ŵ_ν^Tϕ_ν+1/2 S_N+1^-1[ -1/2∑_j=1^N( ∂∇ϕ_v_j(𝐱,ν̂)/∂ν̂)^T(Â_j^N+1
       -𝐖̂_a_j𝐖̂_a_j^T∇ϕ_v_j(𝐱,ν̂)B_j^N+1)+Π̂]=e_ν

and

    Ŵ̇_ν=-ϱ∂ e_ν^Te_ν/∂Ŵ_ν

Where ϱ is the learning rate. The proof of convergence of the above gradient based update rule is given in Theorem <ref>.




Now the results for both leader and followers are used to solve the hierarchical game in the next subsection.








 §.§ The Main Hierarchical Game








The errors, e_is, in (<ref>) and (<ref>), are used for updating critic NNs' weights with gradient descent approach:

    E=∑_i=1^N+1e_i^Te_i  ⇒  𝐖̇̂̇_v_i=-τ_i∂ E/∂𝐖̂_v_i


The following facts can be concluded:




  * The NN approximation errors and their gradients are bounded:

    ‖ϵ_v_i‖ <b_ϵ_i,  ‖∇_𝐱ϵ_v_i‖ <b_ϵ_x_i


  * The NN activation functions and their gradients are bounded:

    ‖ϕ_v_i(𝐱,ν)‖ <b_ϕ_i,  ‖∇_𝐱ϕ_v_i(𝐱,ν)‖ <b_ϕ_i_x
  
    ‖∇_ν∇_𝐱ϕ_v_i(𝐱,ν)‖ <b_ϕ_i_xν


  * The ideal NN weights are bounded:

    ‖𝐖_v_i‖ < W_i_max,   ‖ W_ν‖ < W_ν_max


In the following theorem update rules and ultimate boundedness of errors are analyzed.
 
Consider the dynamical system (<ref>) with critic NNs (<ref>) and control inputs (<ref>) and (<ref>). The tuning laws for the critic NNs are ( for 1≤ i ≤ N+1 ):

    𝐖̇̂̇_v_i   =-τ_iμ_i( η_i^T𝐖̂_v_i+1/4∑_j=1^N𝐖̂_a_j^T C_j^i𝐖̂_a_j+ν̂S_iν̂
          -1/4𝐖̂_a_i^TE_i𝐖̂_a_i+Q_i(𝐱))

where η_i=∇ϕ_v_i(𝐟(𝐱)+∑_j=1^Ng_j(𝐱)𝐮̂_j+p(𝐱)ν̂+h(𝐱)ω̂ ), ρ_i=η_i^Tη_i+1 , η̅_i=η_i/ρ_i and η̅_i/ρ_i=μ_i .
The tuning laws for the leader NN and actor NNs are

    Ŵ̇_ν=-ϱ{∑_j=1^N( ∂Ŵ_ν^Tϕ_ν/∂Ŵ_ν)( ∂^2∇ϕ_v_j/∂ν̂^2)^T[-1/4Â_j^N+1
           ×Π̂^TS_N+1^-2+1/8∑_k=1^NÂ_j^N+1(Â_k^N+1)^T(∂∇ϕ_v_k/∂ν̂)S_N+1^-2
            -1/2Â_j^N+1ϕ_ν^TŴ_νS_N+1^-1+1/4𝐖̂_a_j𝐖̂_a_j^T∇ϕ_v_jB_j^N+1
           ×Π̂^TS_N+1^-2+1/8∑_k=1^N𝐖̂_a_j𝐖̂_a_j^T∇ϕ_v_jB_j^N+1
           × B_k^N+1∇ϕ_v_k^T𝐖̂_a_k𝐖̂_a_k^T( ∂∇ϕ_v_k/∂ν̂)S_N+1^-2
            -1/8∑_k=1^NÂ_j^N+1B_k^N+1∇ϕ_v_k^T𝐖̂_a_k𝐖̂_a_k^T( ∂∇ϕ_v_k/∂ν̂)S_N+1^-2
            +1/2𝐖̂_a_j𝐖̂_a_j^T∇ϕ_v_j B_j^N+1ϕ_ν^TŴ_νS_N+1^-1-1/8∑_k=1^N𝐖̂_a_k
           ×𝐖̂_a_k^T∇ϕ_v_kB_k^N+1(Â_j^N+1)^T( ∂∇ϕ_v_j/∂ν̂)S_N+1^-2] 
    
            +∑_j=1^N( ∂Ŵ_ν^Tϕ_ν/∂Ŵ_ν)( ∂∇ϕ_v_j/∂ν̂)^T[ 1/8∑_k=1^N𝐖̂_a_j𝐖̂_a_j^T
           ×( ∂∇ϕ_v_j/∂ν̂)S_N+1^-2( ∂∇ϕ_v_k/∂ν̂)^T𝐖̂_a_k𝐖̂_a_k^T∇ϕ_v_k
           × B_j^N+1B_k^N+1+1/4𝐖̂_a_j𝐖̂_a_j^T( ∂∇ϕ_v_j/∂ν̂)S_N+1^-2Π̂[3] 
            +1/2𝐖̂_a_j𝐖̂_a_j^T( ∂∇ϕ_v_j/∂ν̂)S_N+1^-1Ŵ_ν^Tϕ_νB_j^N+1
            -1/8∑_k=1^N𝐖̂_a_k𝐖̂_a_k^T( ∂∇ϕ_v_k/∂ν̂)S_N+1^-2( ∂∇ϕ_v_j/∂ν̂)^T
           ×Â_j^N+1B_k^N+1]+2ϕ_νϕ_ν^TŴ_ν +Π̂^TS_N+1^-1
    
            -1/2∑_j=1^Nϕ_ν(Â_j^N+1)^T( ∂∇ϕ_v_j/∂ν̂)S_N+1^-1
            +1/2∑_j=1^Nϕ_νB_j^N+1∇ϕ_v_j^T𝐖̂_a_j𝐖̂_a_j^T( ∂∇ϕ_v_j/∂ν̂)S_N+1^-1}

and

    𝐖̇̂̇_a_i   =-θ_i[ (F_2^i𝐖̂_a_i-F_1^iη̅_i^T𝐖̂_v_i) -1/4∑_k=1^N+1C_i^k𝐖̂_a_i
         ×μ_k^T𝐖̂_v_k +1/4E_i𝐖̂_a_i𝐖̂_v_i],    1≤ i≤ N


    𝐖̇̂̇_a_N+1   =-θ_N+1[ (F_2^N+1𝐖̂_a_N+1-F_1^N+1η̅_N+1^T𝐖̂_v_N+1)
          +1/4E_N+1𝐖̂_a_N+1μ_N+1^T𝐖̂_v_N+1]

respectively.
 F_1^is and F_2^is are tuning parameters that can be selected as given in the proof of theorem in Appendix <ref>. In such condition there exists κ^* so that for κ>κ^* the closed loop system states, the critic and actor NN errors are UUB. Moreover, 𝐮̂_is, ω̂_is and ν̂ converge to the approximate Stackelberg-Nash-saddle equilibrium point of the game.



See Appendix <ref>.


 Terms ( ∂Ŵ_ν^Tϕ_ν/∂Ŵ_ν) and ϕ_ν are not equal, because the first term is a gradient of a vector-valued function in ℝ^α with respect to a matrix in ℝ^μ×α and hence is in ℝ^μ×α×α while the second term is a vector in ℝ^μ. In some especial cases these two terms become equal; e.g., α =1 .





§ SIMULATION




In this section the proposed algorithm is implemented on a hierarchical one-leader-two-followers game with nonlinear dynamic. Disturbance is considered as a uniform noise bounded in [-0.5 , 0.5] . following is the system dynamics:

    𝐱̇=𝐟(𝐱)+𝐠_1(𝐱)u_1+𝐠_2(𝐱)u_2+𝐩(𝐱)ν +𝐡(𝐱)ω

where 𝐟(𝐱)=[f_1(𝐱)    f_2(𝐱)]^T and

    f_1(𝐱)   = x_2 
    
    f_2(𝐱)   =-x_2+0.5x_1+0.25x_2(cos (2x_1)+2)^2 
            +0.25x_2(sin (4x_1^2)+2)^2


    𝐠_1(𝐱)=[            0; cos (2x_1)+2 ] ,    𝐠_1(𝐱)=[              0; sin (4x_1^2)+2 ]

and

    𝐩(𝐱)=[             1; cos (x_1^2)+4 ] ,     𝐡(𝐱)=[   sin (5x_1)+0.1; cos (2x_1^2)+0.2 ]

The parameters of performance index functions are Q_1(x)=2x_1^2+x_2^2, Q_2(x)=x_1^2 +4x_2^2, Q_3(x)=x_1^4+2x_2^2, S_1=4, S_2=2, S_3=20 (since in real situations the effect of leader is more than the effect of followers, S_3 is more than S_2 and S_1), R_11=4, R_12=1, R_21=1, R_22=2, R_31=1, R_32=1 and  the disturbance attenuation γ^2 =0.6. The initial state x_1(0)=x_2(0)=0.01. Initial values of NN weights, 𝐖̂_v_is, 𝐖̂_a_is and 𝐖̂_ν for 1≤ i≤ 3, are randomly selected in the interval (0, 1). Activation functions are ϕ_v_1=ϕ_v_2=[x_1^2, x_2^2, x_1x_2, x_1ν, x_2ν ]^T, ϕ_v_3=[x_1^2, x_2^2, x_1^4, x_2^4, x_1^2x_2^2]^T and ϕ_ν=[x_1, x_2, x_1^2, x_2^2, x_1x_2]^T. Learning rates are τ_1=τ_2=0.7 , τ_3=0.6 , θ_1=θ_2=θ_3=0.7 and ϱ =0.6. Tuning parameters F_2^1=F_2^2=F_2^3=200 and F_1^1=F_1^2=F_1^3=100× [1, 1, 1, 1, 1]^T .


Also a white Gaussian probing noise is added for satisfying persistence of exitation condition. After convergence at about 80 seconds one could get 𝐖_v_1=𝐖_a_1=[2.752, 0.3777, -0.0334, -0.1469, 0.7363]^T, 𝐖_v_2=𝐖_a_2=[1.0876, 0.1607, 0.1180, 0.4029, 0.9177]^T, 𝐖_v_3=𝐖_a_3=[1.0929, -0.2286, 2.8509, 0.0107, 0.3939]^T and 𝐖_ν=[-0.4446, -0.0240, 0.0643, 0.5727, 0.5827, 0.2225]^T .System states are presented in Fig. <ref>.


















§ CONCLUSION



In this paper an on-line ADP-based method is developed for solving a class of hierarchical one-leader-multi-followers nonlinear differential games. The game discussed here was made up of both zero-sum and nonzero-sum games. In the proposed algorithm the value functions approximated with NNs and the approximations improved by gradient descent. Also the actor NN weights were updated using adaptive rules. Using Lyapunov theory, it was showed that the NN weights and system states are UUB. It was assumed that the game had one equilibrium and the condition which satisfied this assumption was derived using Nemytskii operator.



As a future work of this study, the effect of model parameter uncertainties  on the convergence of the on-line ADP can be investigated.



















[Proof of theorem 7]



The candidate Lyapunov function is defined as follows

    L(t)    =∑_i=1^N+1V_i(𝐱)+∑_i=1^N+11/2𝐖̃_v_i^Tτ_i^-1𝐖̃_v_i
            +∑_i=1^N+11/2𝐖̃_a_i^Tθ_i^-1𝐖̃_a_i

where 𝐖_v_i-𝐖̂_v_i=𝐖̃_v_i and
𝐖_v_i-𝐖̂_a_i=𝐖̃_a_i.
Time derivative of (<ref>) is:

    L̇(t) =∑_i=1^N+1V̇_i(𝐱) +∑_i=1^N+1𝐖̃_v_i^Tτ_i^-1𝐖̇̃̇_v_i +∑_i=1^N+1𝐖̃_a_i^Tθ_i^-1𝐖̇̃̇_a_i

Since the time derivative of value functions V̇_i = (∇ V_i)^T𝐱̇=(𝐖_v_i^T∇ϕ_v_i+∇ϵ_i^T)𝐱̇ , one can get:

    V̇_i   =𝐖_v_i^T( ∇ϕ_v_i𝐟(𝐱)-1/2∑_j=1^ND_i^j𝐖̂_a_j+1/2E_i𝐖̂_a_i
          +∇ϕ_v_ip(𝐱)ν̂) ±1/2∑_j=1^N𝐖_v_i^TD_i^j𝐖_a_j
           ±1/2𝐖_v_i^TE_i𝐖_a_i+ϵ̇_i[3] 
       =𝐖_v_i^Tσ_i+1/2𝐖_v_i^T∑_j=1^ND_i^j𝐖̃_a_j-1/2𝐖_v_i^TE_i𝐖̃_a_i
            +𝐖_v_i^T∇ϕ_v_ip(𝐱)(ν̂-ν̌)+ϵ̇_i

where

    ϵ̇_i   =∇ϵ_i^T( 𝐟(𝐱)-1/2∑_j=1^Ng_j(𝐱)R_jj^-1g_j^T(𝐱)∇ϕ_v_j^T𝐖̂_a_j
            +γ^-2/2h(𝐱)h^T(𝐱)∇ϕ_v_i^T𝐖̂_a_i+p(𝐱)ν̂) 
    
    
    σ_i   =∇ϕ_v_i( 𝐟(𝐱)-1/2∑_j=1^N g_j(𝐱)R_jj^-1g_j^T(𝐱)∇ϕ_v_j^T𝐖_v_j
            +p(𝐱)ν̌+γ^-2/2h(𝐱)h^T(𝐱)∇ϕ_v_i^T𝐖_v_i)

From (<ref>) it is concluded that:

    𝐖_v_i^Tσ_i   =-Q_i(𝐱)-1/4∑_j=1^N𝐖_v_j^TC_j^i𝐖_v_j[3]
            +1/4𝐖_v_i^TE_i𝐖_v_i -ν̌^TS_iν̌+ϵ_HJ_i

Now the second part of (<ref>) is calculated according to (<ref>) and gradient descent method.

    𝐖̃_v_i^Tτ_i^-1𝐖̇̃̇_v_i   = 𝐖̃_v_i^Tη_i/(η_i^Tη_i+1)^2( η_i^T𝐖̂_v_i+Q_i(𝐱)
            +∑_j=1^N𝐮̂_j^TR_ij𝐮̂_j+ ν̂S_iν̂-γ^2ω̂^Tω̂)

Now the terms in (<ref>) which are in parentheses can be rewritten as

    Q_i(𝐱)+ 1/4∑_j=1^N( 𝐖_v_j-𝐖̃_a_j)^TC_j^i(𝐖_v_j-𝐖̃_a_j) 
        -1/4(𝐖_v_i-𝐖̃_a_i)^TE_i(𝐖_v_i-𝐖̃_a_i)
       +(𝐖_v_i-𝐖̃_v_i)^Tη_i+ν̂^TS_iν̂

According to (<ref>) and (<ref>), (<ref>) becomes

    -𝐖̃_v_i^Tη_i+ϵ_HJ_i+1/4∑_j=1^N𝐖̃_a_jC_j^i𝐖̃_a_j+ϕ_ν^TŴ_νS_i
           ×Ŵ_ν^Tϕ_ν -1/4𝐖̃_a_i^TE_i𝐖̃_a_i-ϕ_ν^TW_νS_iW_ν^Tϕ_ν[3] 
             -1/2∑_j=1 
     j≠ i^N𝐖_v_j^TC_j^i𝐖̃_a_j
            +𝐖_v_i^T(- ∇ϕ_v_ip(𝐱)W̃_ν^Tϕ_ν+1/2∑_j=1 
     j≠ i^ND_j^i𝐖̃_a_j)

Thus

    𝐖̃_v_i^Tτ_i^-1𝐖̇̃̇_v_i=𝐖̃_v_i^Tη_i/(η_i^Tη_i+1)^2( -𝐖̃_v_i^Tη_i
            +1/4∑_j=1^N𝐖̃_a_j^TC_j^i𝐖̃_a_j-1/2∑_j=1 
     j≠ i^N𝐖_v_j^TC_j^i𝐖̃_a_j
            +𝐖_v_i^T(- ∇ϕ_v_ip(𝐱)W̃_ν^Tϕ_νg_j(𝐱)R_jj^-1
            +1/2∑_j=1 
     j≠ i^ND_i^j𝐖̃_a_j) -2ϕ_ν^TW̃_νS_iW_ν^Tϕ_ν[3] 
            +ϵ_HJ_i-γ^-2/4𝐖̃_a_i^TE_i𝐖̃_a_i+ ϕ_ν^TW̃_νS_iW̃_ν^Tϕ_ν)

Now according to (<ref>), (<ref>), (<ref>), and (<ref>), (<ref>) is rewritten as:

    L̇(𝐱) = ∑_i=1^N+1( -Q_i(𝐱)+ϵ_HJ_i-1/4∑_j=1^N𝐖_v_j^TC_j^i𝐖_v_j
            +1/4∑_j=1^N𝐖̃_a_j^TC_j^i𝐖̃_a_jμ_i^T𝐖̃_v_i
            +1/4𝐖_v_i^TE_i𝐖_v_i- 𝐖̃_v_i^Tη̅_iη̅_i^T𝐖̃_v_i -1/2∑_j=1 
     j≠ i^N𝐖̃_a_j^T[3] 
           × C_j^i𝐖_v_jμ_i^T𝐖̃_v_i+𝐖̃_a_i^Tθ_i^-1𝐖̇̃̇_a_i
            +1/2∑_j=1 
     j≠ i^N𝐖̃_a_j^TD_i^j𝐖_v_iμ_i^T𝐖̃_v_i+1/2𝐖_v_i^T∑_j=1^ND_i^j
           ×𝐖̃_a_j -1/2𝐖_v_i^TE_i𝐖̃_a_i+𝐖̃_v_i^Tμ_iϵ_HJ_i
            -𝐖_v_i^T∇ϕ_v_i p(𝐱)W̃_ν^Tϕ_ν+ ϕ_ν^TW̃_νS_iW̃_ν^Tϕ_ν
           ×μ_i^T𝐖̃_v_i -𝐖̃_v_i^Tμ_i𝐖_v_i^T∇ϕ_v_i p(𝐱)W̃_ν^Tϕ_ν
            -ϕ_ν^TW_νS_iW_ν^Tϕ_ν- 2ϕ_ν^TW̃_νS_iW_ν^Tϕ_ν
           ×μ_i^T𝐖̃_v_i -1/4𝐖̃_a_i^TE_i𝐖̃_a_iμ_i^T𝐖̃_v_i
            +∇ϵ_i^T( 𝐟(𝐱)-1/2∑_j=1^Ng_j(𝐱)R_jj^-1g_j^T(𝐱)∇ϕ_v_j^T𝐖̂_a_j
            +γ^-2/2h(𝐱)h^T(𝐱)∇ϕ_v_i^T𝐖̂_a_i+p(𝐱)ν̂) )

Expanding some terms in (<ref>) yields:

    L̇(𝐱)    = ∑_i=1^N+1( -Q_i(𝐱)+ϵ_HJ_i-1/4∑_j=1^N𝐖_v_j^TC_j^i𝐖_v_j
            -1/4∑_j=1^N𝐖̃_a_j^TC_j^i𝐖_v_jμ_i^T𝐖_v_i+∇ϵ_i^T𝐟(𝐱) 
            +1/4∑_j=1^N𝐖̃_a_j^TC_j^i𝐖_v_jμ_i^T𝐖̃_v_i-𝐖̃_v_i^Tη̅_iη̅_i^T𝐖̃_v_i
            -1/2∑_j=1 
     j≠ i^N𝐖̃_a_j^TC_j^i𝐖_v_jμ_i^T𝐖̃_v_i
            +1/2∑_j=1 
     j≠ i^N𝐖̃_a_j^TD_i^j𝐖_v_iμ_i^T𝐖̃_v_i+1/2𝐖_v_i^T∑_j=1^ND_i^j
           ×𝐖̃_a_j -1/2𝐖_v_i^TE_i𝐖̃_a_i+𝐖̃_v_i^Tμ_iϵ_HJ_i
            -𝐖_v_i^T∇ϕ_v_i p(𝐱)W̃_ν^Tϕ_ν+ ϕ_ν^TW̃_νS_iW̃_ν^Tϕ_ν
           ×μ_i^T𝐖̃_v_i-𝐖̃_v_i^Tμ_i𝐖_v_i^T∇ϕ_v_i p(𝐱)W̃_ν^Tϕ_ν
            +1/4𝐖̃_a_i^TE_i𝐖_v_iμ_i^T𝐖_v_i -ϕ_ν^TW_νS_iW_ν^Tϕ_ν
            +1/4∑_j=1^N𝐖̃_a_j^TC_j^i𝐖̃_a_jμ_i^T𝐖_v_i-1/4𝐖̃_a_i^TE_i𝐖̃_a_i
           ×μ_i^T𝐖_v_i -2ϕ_ν^TW̃_νS_iW_ν^Tϕ_νμ_i^T𝐖̃_v_i
            +1/4𝐖_v_i^TE_i𝐖_v_i -1/4𝐖̃_a_i^TE_i𝐖_v_iμ_i^T𝐖̃_v_i
            +∇ϵ_i^T( -1/2∑_j=1^Ng_j(𝐱)R_jj^-1g_j^T(𝐱)∇ϕ_v_j^T𝐖_v_j
            +γ^-2/2h(𝐱)h^T(𝐱)∇ϕ_v_i^T𝐖_v_i+p(𝐱)ν̌) 
            -∇ϵ_i^T( -1/2∑_j=1^Ng_j(𝐱)R_jj^-1g_j^T(𝐱)∇ϕ_v_j^T𝐖̃_a_j[3] 
            +γ^-2/2h(𝐱)h^T(𝐱)∇ϕ_v_i^T𝐖̃_a_i+p(𝐱)ν̃))  
            -∑_i=1^N𝐖̃_a_i^T[ θ_i^-1𝐖̇̂̇_a_i-1/4∑_k=1^NC_i^k
           ×𝐖̂_a_iμ_k^T𝐖̂_v_k +1/4E_i𝐖̂_a_iμ_i^T𝐖̂_v_i] 
             -𝐖̃_a_N+1^T[θ_N+1^-1𝐖̇̂̇_a_N+1
            +1/4E_N+1𝐖̂_a_N+1μ_N+1^T𝐖̂_v_N+1]

where ν̌ -ν̂=ν̃. So update rules for action networks are defined as:

    𝐖̇̂̇_a_i   =-θ_i[ (F_2^i𝐖̂_a_i-F_1^iη̅_i^T𝐖̂_v_i) -1/4∑_k=1^N C_i^k𝐖̂_a_i
           ×μ_k^T𝐖̂_v_k +1/4E_i𝐖̂_a_iμ_i^T𝐖̂_v_i]    1≤ i≤ N


    𝐖̇̂̇_a_N+1   =-θ_N+1[ (F_2^N+1𝐖̂_a_N+1-F_1^N+1η̅_N+1^T𝐖̂_v_N+1)
            +1/4E_N+1𝐖̂_a_N+1μ_N+1^T𝐖̂_v_N+1]

where F_2^is and F_1^is are tuning parameters and they should be selected as mentioned in proceed of proof. With these tuning laws the following terms added to L̇(𝐱):

    ∑_i=1^N+1𝐖̃_a_i^T( F_2^i𝐖_v_i-F_2^i𝐖̃_a_i -F_1^iη̅_i^T𝐖_v_i+F_1^iη̅_i^T𝐖̃_v_i)

Now (<ref>) becomes

    L̇(𝐱) = ∑_i=1^N+1( -Q_i(𝐱)+ϵ_HJ_i-1/4∑_j=1^N𝐖_v_j^TC_j^i𝐖_v_j
            +1/4∑_j=1^N𝐖̃_a_j^TC_j^i𝐖̃_a_jμ_i^T𝐖_v_i
            +1/4𝐖_v_i^TE_i(𝐖_v_i-2𝐖̃_a_i) - 𝐖̃_v_i^Tη̅_iη̅_i^T𝐖̃_v_i
            -1/2∑_j=1 
     j≠ i^N𝐖̃_a_j^T(C_j^i-D_i^j)𝐖_v_jμ_i^T𝐖̃_v_i+∇ϵ_i^T𝐟(𝐱) 
            -𝐖_v_i^T∇ϕ_v_i p(𝐱)W̃_ν^Tϕ_ν+ ϕ_ν^TW̃_νS_i(W̃_ν-2W_ν)^T
           ×ϕ_νμ_i^T𝐖̃_v_i-𝐖̃_v_i^Tμ_i𝐖_v_i^T∇ϕ_v_i p(𝐱)W̃_ν^Tϕ_ν
            +1/4𝐖̃_a_i^TE_i(𝐖_v_i-𝐖̃_a_i)μ_i^T𝐖_v_i-ϕ_ν^TW_νS_i
           × W_ν^Tϕ_ν+1/2𝐖_v_i^T∑_j=1^ND_i^j𝐖̃_a_j
            +1/4∑_j=1^N𝐖̃_a_j^TC_j^i𝐖_v_jμ_i^T(𝐖̃_v_i-𝐖_v_i)
            -1/4𝐖̃_a_i^TE_i𝐖_v_iμ_i^T𝐖̃_v_i+𝐖̃_v_i^Tμ_iϵ_HJ_i
            +𝐖̃_a_i^T( F_2^i𝐖_v_i-F_2^i𝐖̃_a_i-F_1^iη̅_i^T𝐖_v_i[3] 
            +F_1^iη̅_i^T𝐖̃_v_i)+∇ϵ_i^T(+p(𝐱)(W_ν-W̃_ν)^Tϕ_ν
            -1/2∑_j=1^Ng_j(𝐱)R_jj^-1g_j^T(𝐱)∇ϕ_v_j^T(𝐖_v_j-𝐖̃_a_j)
            +γ^-2/2h(𝐱)h^T(𝐱)∇ϕ_v_i^T(𝐖_v_i-𝐖̃_a_i)) )


A new parameter is defined which only has ideal weights:

    Υ   =∑_i=1^N+1( ϵ_HJ_i-1/4∑_j=1^N𝐖_v_j^TC_j^i𝐖_v_j+1/4𝐖_v_i^TE_i𝐖_v_i
            -ϕ_ν^TW_νS_iW_ν^Tϕ_ν+∇ϵ_i^Tp(𝐱)W_ν^Tϕ_ν[3] 
            +∇ϵ_i^T(-1/2∑_j=1^Ng_j(𝐱)R_jj^-1g_j^T(𝐱)∇ϕ_v_j^T𝐖_v_j
            +γ^-2/2h(𝐱)h^T(𝐱)∇ϕ_v_i^T𝐖_v_i)

According to the assumptions, Υ has an upper bound, Υ_max.
Since ϵ_HJ_is are NN approximations errors of value functions, κ (number of hidden layers in NNs) can be selected such that to decrease these errors to a desired constant:

    ∀ϵ_i>0,  ∃κ^*  s.t.   sup_x∈Ω‖ϵ_HJ_i‖ <ϵ_i    for   κ >κ^*

Now by assuming κ >κ^*  and defining Z̃_1=𝐱, Z̃_2=[ 𝐖̃_v_1^Tη̅_1, … ,  𝐖̃_v_N+1^Tη̅_N+1]^T, Z̃_3=[ 𝐖̃_a_1^T, … , 𝐖̃_a_N+1^T]^T, Z̃_4= W̃_ν^Tϕ_ν, and Z̃=[Z̃_1^T, Z̃_2^T, Z̃_3^T, Z̃_4^T]^T, (<ref>) can be rewritten as

    L̇(𝐱)    = Υ +Z̃_2^T(Z̃_4^T⊗ I_N+1)SZ̃_4 -Z̃^TMZ̃
            +Z̃^T[    λ_1;    λ_2;      ⋮; λ_2N+4 ]+∇ϵ_i^T𝐟(𝐱)

Where ⊗ is the Kronecker product, I_N+1 is the identity matrix and S=[S_1^T, S_2^T, … , S_N+1^T]^T. By assuming 2≤ a≤ N+2, N+3≤ b≤ 2N+2, i=a-1 and j=b-N-2, the blocks of symmetric matrix M and vector Λ are:

    m_11=∑_l=1^N+1ϑ_l
       m_aa=1 
       m_ba=1/8ρ_iC_j^i𝐖_v_j-1/4ρ_iD_i^j𝐖_v_i,      i≠ j 
       m_ba=-1/8ρ_jD_j^j𝐖_v_j+1/8E_j𝐖_v_j-1/2F_1^j,      i=j 
        m_bb=-1/4∑_l=1^N+1C_j^lμ_l^T𝐖_v_l
            +1/4E_jμ_j^T𝐖_v_j+F_2^j
       m_a(2N+4)=1/2ρ_i𝐖_v_i^T∇ϕ_v_ip(𝐱)+1/ρ_iϕ_ν^T𝐖_νS_i
       m_(2N+3)(2N+3)=1/4E_N+1μ_N+1^T+F_2^N+1[3]
        m_1a=m_1b=m_1(2N+3)=m_1(2N+4)=m_b(2N+3)
            =m_b(2N+4)=m_(2N+3)(2N+4)
            =m_(2N+4)(2N+4)=0


    m_a(2N+3)= {[ 1/8E_N+1𝐖_v_N+1-1/2F_1^N+1 ,                        a=N+2;                          0 ,                       a≠ N+2 ].

and

    λ_1   =0
    λ_a   =ϵ_HJ_i/ρ_i
    λ_b   =1/2∑_l=1^N+1(D_l^j)^T𝐖_v_l-1/2E_j𝐖_v_j
         -1/4∑_l=1^N+1C_j^lμ_l^T𝐖_v_l+F_2^j𝐖_v_j
         +∑_l=1^N+11/2∇ϕ_v_jg_j(𝐱)R_jj^-1R_ljR_jj^-1
        × g_j^T(𝐱)∇ϵ_l -γ^-2/2∇ϕ_v_jh(𝐱)h^T(𝐱)∇ϵ_j
         +1/4E_j𝐖_v_jμ_j^T𝐖_v_j-F_1^jη̅_j^T𝐖_v_j
    λ_2N+3   =-1/2E_N+1𝐖_v_N+1+F_2^N+1𝐖_v_N+1
         +1/4E_N+1𝐖_v_N+1μ_N+1^T𝐖_v_N+1[3] 
         -γ^-2/2∇ϕ_v_N+1h(𝐱)h^T(𝐱)∇ϵ_N+1
         -F_1^N+1η̅_N+1^T𝐖_v_N+1
    λ_2N+4   =-∑_l=1^N+1p^T(𝐱)∇ϕ^T_v_l𝐖_v_l-p^T(𝐱)∑_l=1^N+1∇ϵ_l

Since f(𝐱) is Lipschitz, one can get

    L̇(𝐱)    ≤Υ +Z̃_2^T(Z̃_4^T⊗ I_N+1)SZ̃_4 -Z̃^T MZ̃
            +Z̃^T[    λ_1;    λ_2;      ⋮; λ_2N+4 ]

and (<ref>) becomes λ_1=∑_l=1^N+1b_ϵ_x_lb_f.
Parameters F_2^i and F_1^i should be chosen such that matrix M becomes positive definite. Elements of vector Λ are bounded. Thus:

    L̇<-‖Z̃‖^2λ_min(M)+Λ_max‖Z̃‖ +Υ_max+‖Z̃_2‖‖Z̃_4^2‖‖ S‖

Expanding terms in (<ref>) yields:

    L̇<-λ_min(M)(‖Z̃_1‖^2+‖Z̃_2‖^2+‖Z̃_3‖^2+‖Z̃_4‖^2) 
          +Λ_max√(‖Z̃_1‖^2+‖Z̃_2‖^2+‖Z̃_3‖^2+‖Z̃_4‖^2)[3] 
          +‖Z̃_2‖‖Z̃_4^2‖‖ S‖ +Υ_max
         < -λ_min(M)(‖Z̃_1‖^2+‖Z̃_2‖^2+‖Z̃_3‖^2+‖Z̃_4‖^2) 
          +Λ_max(‖Z̃_1‖ +‖Z̃_2‖ +‖Z̃_3‖ +‖Z̃_4‖) 
          +‖Z̃_2‖‖Z̃_4‖^2‖ S‖ +Υ_max

For simplicity λ_min(M)=λ_m, Λ_max=Λ_m, Υ_max=Υ_m and ‖ S‖ =S_m. By defining ‖Z̃_2‖ =rsin(φ) and ‖Z̃_4‖ =rcos(φ) where 0≤φ≤π/2 , one can get:

    Λ_m(‖Z̃_2‖ +‖Z̃_4‖)+‖Z̃_2‖‖Z̃_4‖^2S_m
            +λ_m(M)(‖Z̃_2‖^2+‖Z̃_4‖^2)
       = 
            -λ_mr^2+Λ_m r( sin(φ)+cos(φ)) 
            +S_mr^3sin(φ)cos^2(φ) 
       ≤
            -λ_mr^2+√(2)rΛ_m + ( 0.385 S_m)r^3

Thus for

    r_1 < r < r_2

where

    r_1=   λ_m -√(λ_m^2-1.54√(2)S_mΛ_m)/0.77 S_m
    
    r_2=   λ_m +√(λ_m^2-1.54√(2)S_mΛ_m)/0.77 S_m

 relation (<ref>) becomes negative. Tuning parameters F_2^is and F_1^is should be selected such that λ_m becomes large enough to achieve two goals: making the term under radical sign positive and making the region in (<ref>) big enough to discard improper initializations of 𝐖̂_v_is and Ŵ_ν.
 Now if

    ‖Z̃_1‖ +‖Z̃_3‖ >Λ_m/λ_m+√(Υ_m/λ_m+Λ_m^2/2λ_m^2)

with condition (<ref>), L̇ becomes negative. So with proper initialization, 𝐱, 𝐖̂_v_is and 𝐖̂_a_is and hence 𝐖̂_v_i-𝐖̂_a_is become UUB. According to (<ref>) and using gradient descent for updating Ŵ_ν and ultimate boundedness of 𝐖̂_a_is and 𝐱, Ŵ_ν becomes UUB too.

































ieeetr















  






























































































