Autonomous Drifting Based on Maximal Safety Probability Learning
Hikaru Hoshino Jiaxing Li Arnav Menon John M. Dolan Yorie Nakahira



    The YMDB catalog: Young massive detached binaries for the determination of high-precision absolute stellar parametersFull version of Tables 2, 3 and 4 are only available in electronic form at the CDS via anonymous ftp to cdsarc.u-strasbg.fr (130.79.128.5) or via http://cdsweb.u-strasbg.fr/cgi-bin/qcat?J/A+A/.
    
    
    Pablo Martín-Ravelo 
    1,2
    Roberto Gamen 
    3,4
    Julia I. Arias 
    1
    André-Nicolas Chené 
    2
    Rodolfo H. Barbá In Memoriam (1962–2021)
    1
    
    Received: 20 June 2024 / Accepted: 05 July 2024
================================================================================================================================================================================================================================================================================================================================

empty
empty



§ ABSTRACT 

This paper proposes a novel learning-based framework for autonomous driving based on the concept of maximal safety probability.  
Efficient learning requires rewards that are informative of desirable/undesirable states, but such rewards are challenging to design manually due to the difficulty of differentiating better states among many safe states. On the other hand, learning policies that maximize safety probability does not require laborious reward shaping but is numerically challenging because the algorithms must optimize policies based on binary rewards sparse in time.
Here, we show that physics-informed reinforcement learning can efficiently learn this form of maximally safe policy. Unlike existing drift control methods, our approach does not require a specific reference trajectory or complex reward shaping, and can learn safe behaviors only from sparse binary rewards. This is enabled by the use of the physics loss that plays an analogous role to reward shaping. The effectiveness of the proposed approach is demonstrated through lane keeping in a normal cornering scenario and safe drifting in a high-speed racing scenario. 









§ INTRODUCTION


Driving in adverse conditions (e.g. high-speed racing or icy roads with low traction) is challenging for both human drivers and autonomous vehicles.
The vehicle operates near its handling limits in a highly nonlinear regime and has to cope with uncertainties due to unmodeled dynamics and noises in sensing and localization <cit.>. 
Furthermore, it needs to have a very low response time to adapt to the rapidly changing environment <cit.>. 
Deterministic worst-case frameworks including robust sliding-mode control <cit.> 
can often be efficiently computed but require full system models and small bounded uncertainties (errors). Techniques based on set invariance, such as control barrier functions and barrier certificates <cit.>, are applied to vehicle systems with analytical models when these functions can be designed. Techniques based on probabilistic invariance <cit.> can be used to generate safety certificates using samples of complex (black-box) systems in extreme environments <cit.> and occluded environments <cit.>.  
Model Predictive Control (MPC) techniques, including stochastic MPC <cit.> and chance-constrained MPC <cit.>, exploit future predictions to account for uncertainties. 
However, as the number of possible trajectories grows exponentially to the outlook time horizon, there are often stringent tradeoffs between outlook time horizon and computation burdens. 
Thus, it is still challenging to ensure safety in adverse and uncertain conditions with lightweight algorithms suitable for onboard computation. 






Meanwhile, within drifting control, various model-based/model-free techniques are considered. 
In <cit.>, a bicycle model is used to describe the phenomenon of vehicle drifting, and an unstable “drift equilibrium” has been found. 
Sustained drifting has been achieved by stabilizing a drift equilibrium using various control methods, such as LQR <cit.>, robust control <cit.>, and MPC <cit.>. 
Hierarchical control architectures are proposed in <cit.> 
for general path tracking. 
A drawback of these previous works is that they aim to stabilize a specific equilibrium or reference trajectories, and pre-computation of such a reference is required.  

As a data-driven drifting control, Probability Inference for Learning Control (PILCO) has been applied <cit.>, but the method is only considered in a single-task setting of minimizing tracking error of a particular drift equilibrium. 
Soft actor-critic algorithm was designed to go through sharp corners in a racing circuit in simulations <cit.>. 
Twin-Delayed Deep Deterministic (TD3) Policy Gradient  is developed in <cit.>, and tabular Q-learning is used in <cit.>. 
The transference of RL agents to the real world was studied by  <cit.>, using experiments of radio-controlled (RC) model cars. 
Drift parking task is considered in <cit.>. 
However, like other RL problems <cit.>, an appropriate design of reward function is required to generalize across different drift maneuvering tasks.
The reward functions in these works tend to be complex, and the distance from a target drift equilibrium or a reference trajectory is typically used for reward shaping. 


In this paper, we propose a novel approach to control vehicle drifting based on Physics-Informed Reinforcement Learning (PIRL). Specifically, PIRL is used to learn a control policy that maximizes the safety probability such that the vehicle can safely drift or slip in adverse conditions. Major technical challenges stem from the fact that the objective function associated with this problem takes multiplicative or maximum costs over time and that the reward is binary and sparse in time. To overcome the difficulty of learning multiplicative or maximum costs, we first present a transformation that converts this problem into RL with additive costs. To efficiently learn from sparse reward, we built upon our previous work  <cit.>, which imposes physics constraints on the loss function. This term in the loss function plays an analogous role in rewarding shaping and allows safe policies to be learned only from binary rewards that are sparse in time. The proposed framework only requires safe events (safe regions) to be specified without the need for reference trajectories or laborious reward shaping.  







 



The effectiveness of the proposed approach is demonstrated through lane keeping in a normal cornering scenario and safe drifting in a high-speed racing scenario.  




































 §.§ Notation


Let ℝ and ℝ_+be the set of real numbers and the set of nonnegative real numbers, respectively. 
Let  and _+ be the set of integers and the set of non-negative integers. 
For a set A, A^c stands for the complement of A, and ∂ A for the boundary of A. 
Let ⌊ x ⌋∈ be the greatest integer less than or equal to x∈.  
Let 1[ℰ] be an indicator function, which takes 1 when the condition ℰ holds and otherwise 0.
Let [ ℰ | X_0 = x ] represent the probability that the condition ℰ holds involving a stochastic process X = { X_t }_t∈_+ conditioned on X_0 = x.  
Given random variables X and Y, let 𝔼[X] be the expectation of X and 𝔼[X|Y=y] be the conditional expectation of X given Y=y. We use upper-case letters (e.g., Y) to denote random variables and lower-case letters (e.g., y) to denote their specific realizations. 
For a scalar function ϕ, 
∂_x ϕ stands for the gradient of ϕ with respect to x, and ∂_x^2 ϕ for the Hessian matrix of ϕ. 
Let tr(M) be the trace of the matrix M. 
For x ∈^n and A ⊂^n, dist(x, A) := inf_y ∈ A x - y.  









































§ MAXIMAL SAFETY PROBABILITY LEARNING


In this section, the problem formulation of estimating maximal safety probability is given in sec:problem_formulation, and the PIRL algorithm <cit.> is briefly reviewed in  sec:PIRL. 





 §.§ Problem Formulation
 


We assume that the vehicle dynamics can be represented as a control system with stochastic noise of w-dimensional Brownian motion { W_t }_t ∈_+ starting from W_0 = 0. 
The system state X_t ∈𝕏⊂ℝ^n is assumed to be observable and evolves according to the following stochastic differential equation (SDE): 

    dX_t =  f(X_t, U_t) dt + σ(X_t, U_t) dW_t,


where U_t ∈𝕌⊂ℝ^m is the control input. 
Throughout this paper, we assume sufficient regularity in the coefficients of the system (<ref>).
That is, the functions f and σ are chosen in a way such that the SDE (<ref>) admits a unique strong solution (see, e.g., Section IV.2 of <cit.>). 
The size of σ(X_t, U_t) is determined from the uncertainties in the disturbance, unmodeled dynamics, and prediction errors of the environmental variables. 


For numerical approximations of the solutions of the SDE and optimal control problems, we consider a discretization with respect to time with a constant step size Δ t under piecewise constant control processes. 
For 0=t_0 < t_1 < … < t_k < …, where t_k := kΔ t, k ∈_+, by defining the discrete-time state X_k := X_t_k with an abuse of notation, the discretized system can be given as

    X_k+1 = F^π(X_k, Δ W_k),

where Δ W_k := { W_t }_t ∈ [t_k, t_k+1), and F^π stands for the state transition map derived from (<ref>) under a Markov control policy π : [0, ∞) ×𝕏→𝕌. 





Safety of the system can be defined by using a safe set ⊂𝕏.  

For the discretized system (<ref>) and for a given control policy π, 
the safety probability ^π of 
the initial state X_0 = x for the outlook horizon τ∈ 
can be characterized as 
the probability that the state X_k stays within the safe set  for 
k ∈𝒩_τ := {0, …, N(τ)}, where N(τ) := ⌊τ / Δ t ⌋, i.e., 





    ^π(τ, x)
    := [ X_k ∈, ∀ k ∈𝒩_τ |  X_0 = x, π ].

Then, the objective of the learning is to estimate the maximal safety probability defined as 
   
    Ψ^∗(τ,x) := sup_π∈𝒫Ψ^π(τ,x),

where 𝒫 is the class of bounded and Borel measurable Markov control policies, and to learn the corresponding safe control policy π^∗ := _π∈𝒫Ψ^π.    








 §.§ Physics-informed RL (PIRL)
 











The training framework is illustrated in fig:PIRL_framework. 
PIRL integrates RL and Physics-Informed Neural Networks (PINNs) <cit.>
for efficient estimation of maximal safety probability. 
This paper uses a PIRL algorithm based on Deep Q-Network (DQN) <cit.>.
The overall structure follows from the standard DQN algorithm, and the  optimal action-value function will be estimated by using a function approximator. 
For this function approximator,  we use a PINN, which is a neural network trained by penalizing the discrepancy from a partial differential equation (PDE) condition that the safety probability should satisfy. 

To define an appropriate RL problem, we consider the augmented state space 𝒮 := ×𝕏⊂^n+1 and the augmented state S_k ∈𝒮, where we denote the first element of S_k by H_k and the other elements by X_k, i.e., 

    S_k =[ H_k, X_k^⊤]^⊤,

where H_k represents the remaining time before the outlook horizon τ is reached. 
Then, consider the stochastic dynamics starting from the initial state s := [τ, x^⊤]^⊤∈𝒮 given as follows
[With this augmentation, the variable X_k in S_k is no longer equivalent to that in (<ref>), because X_k transitions to itself when S_k ∈𝒮_abs. However, we use the same notation for simplicity. See <cit.> for precise derivation.]: 
for ∀ k ∈_+, 

    S_k+1 = 
      F̃^π(S_k, Δ W_k),      S_k ∉𝒮_abs,   
    
          S_k,      S_k ∈𝒮_abs,

with the function F̃^π given by 

    F̃^π(S_k, Δ W_k) := 
     [        H_k - Δ t; F^π( X_k, Δ W_k) ],

and the set of absorbing states 𝒮_abs given by 

    𝒮_abs := { [ τ, x^⊤]^⊤∈𝒮 | τ < 0    x ∉}.

Then, the following proposition states that the value function of our RL problem is equivalent to the safety probability. 



 
Consider the system (<ref>) starting from an initial state s = [τ, x^⊤]^⊤∈𝒮 
and the reward function r: 𝒮→ given by

    r(S_k) := 
          [ H_k ∈𝒢 ]

with 𝒢 := [0, Δ t). 
Then, for a given control policy π, the value function v^π defined by  

    v^π(s) := 𝔼[ ∑_k=0^N_f r( S_k )   |   S_0= s, π],

where N_f := inf{ j ∈_+  |  S_j ∈𝒮_abs},
takes a value in [0,1] and is equivalent to the safety probability ^π(τ, x), i.e., 

    v^π(s) = ^π(τ, x).



  The proof follows from Proposition 1 of <cit.> with a minor rewrite of the reward and value functions.   


Thus, the problem of safety probability estimation can be solved by an episodic RL problem with the action-value function q^π(s, u), defined as the value of taking an action u ∈𝕌 in state s and thereafter following the policy π:

    q^π(s, u) := 𝔼[ ∑_k=0^N_f r(S_k)  |  S_0 = s,  U_0 = u, π].







Furthermore, it is shown in <cit.> that the maximal safety probability can be characterized by the Hamilton-Jacobi-Bellman (HJB) equation of a stochastic optimal control problem.
While the HJB equation can have discontinuous viscosity solutions, the following theorem shows that a slightly conservative but arbitrarily precise approximation can be constructed by considering a set _ϵ smaller than : 












 
Consider the system (<ref>) derived from the SDE (<ref>) and the action-value function q^π_ϵ(s,u) given by 

    q^π_ϵ(s, u) := 𝔼[ ∑_k=0^N_f r_ϵ(S_k)  |  S_0 = s,  U_0 = u, π],

where  the function r_ϵ is given by 

    r_ϵ(S_k) := 
          [ H_k ∈𝒢 ]  l_ϵ( X_k ),

with _ϵ := { x ∈ | dist(x, ^c) ≥ϵ} and 

    l_ϵ (x) :=  max{  1- dist(x, _ϵ)ϵ,   0}.

Then, if the Assumption 1 in <cit.> holds, the optimal action-value function q^ϵ(s,u): = sup_ u ∈𝕌 q^π_ϵ(s,u) becomes a continuous viscosity solution of the following PDE in the limit of Δ t → 0: 

    ∂_s  q^∗_ϵ(s, u^∗) 
          
          f̃(s, u^∗) 
          
        
          +12tr[ σ̃(s,u^∗)σ̃(s,u^∗)^⊤∂_s^2 q^∗_ϵ(s, u^∗)  
          ] = 0,
 
where the function f̃ and σ̃ are given by  

    f̃(s, u) := 
        [      -1; f(x, u) ],
          σ̃(s, u) := 
        [       0; σ(x, u) ],

and u^∗ := _u ∈𝕌 q^∗(s, u). 
The boundary conditions are given by 

    q^∗_ϵ([0, x^⊤]^⊤, u^∗) = l_ϵ( x ),   ∀ x ∈𝕏, 
        q^∗_ϵ([τ, x^⊤]^⊤, u^∗) = 0,   ∀τ∈,  ∀ x ∈∂.
















































  When we assume further regularity conditions on the function l_ϵ(x) (i.e., differentiability of l_ϵ(x)), the PDE (<ref>) can be understood in the classical sense (see e.g., <cit.>). This means that the PDE condition can be imposed by the technique of PINN using automatic differentiation of neural networks. 








Based on the above, a standard DQN algorithm is integrated with a PINN by using a  modified loss function given by 

    L =     L_D + λ L_P
        + μ L_B,

where L_D is the data loss of the original DQN, L_P the loss term imposing the PDE (<ref>), and L_B the loss term for the boundary conditions (<ref>) and (<ref>). 
The parameters λ and μ are the weighting coefficients for the regularization loss terms L_P and L_B, respectively. 
Detailed descriptions of these loss terms are given in <cit.>. 





§ NUMERICAL EXPERIMENTS


This section discusses numerical results of autonomous driving based on maximal safety probability learning. 
After presenting training methodology in sec:setup, we provide results of normal cornering in sec:lane_keeping and high-speed drifting in sec:drifting. 
Our implementation is available at <https://github.com/hoshino06/pirl_itsc2024>.





 §.§ Training Methodology
 

The PIRL agent learns maximal safety probabilities by interacting with CARLA <cit.>, an open-source simulator providing high-fidelity vehicle physics simulations that are not explicitly described by control-oriented models such as a bicycle model. 
The input to the neural network as a function approximator Q for our action-value function is the state s ∈𝒮 defined in (<ref>), and the output is a vector of q values for all discrete actions as explained below. 
The state s consists of the vehicle state x and the outlook horizon τ:

    s = [τ, x^⊤]^⊤,

and x can be further decomposed into x_vehicle for the vehicle dynamics and x_road for the vehicle position relative to the road:

    x = 
     [  v_x,  β,   r_x_vehicle,   e,  ψ,  𝒯_x_road  ]^⊤∈ℝ^15,

where x_vehicle consists of v_x standing for the vehicle longitudinal velocity, β for the sideslip angle, defined as β := arctan(v_y/v_x) with the lateral velocity v_y, and r for the yaw rate. 
The variable x_road consists of e standing for the lateral error from the center line of the road, and ψ for the heading error, and 𝒯 that contains five (x, y) positions of reference points ahead of the vehicle placed on the center line of the lane.



The control action a is given by 

    a = [ δ,    d  ]^⊤,

where δ is the steering angle, and d is the throttle.
They are normalized to [-1, 1] and [0, 1], respectively. 
To implement our DQN-based algorithm, which admits a discrete action space, the control inputs are restricted to d ∈{ 0.6,  0.7,  0.8,  0.9,  1.0  } and δ∈{ -0.8,   -0.4,   0,   0.4,   0.8}. 
The reason for limiting d ≥ 0.6 is to prevent slow driving with trivially safe behaviors (the vehicle is safe if it never moves). 
The steering is limited to |δ| ≤ 0.8, since high-speed vehicles are prone to rollover if large steering angles are applied <cit.>. 



For better sample efficiency and generalization to unseen regions, PIRL can exploit the structure of a control-oriented model encoded in the form of a PDE in (<ref>) and the regularization loss term L_P in (<ref>). 
In this paper, we used a dynamic bicycle model with a 
Pacejka tire model. 
The parameters of the model that correspond to the vehicle dynamics in CARLA are assumed to be known in this paper, but PINN can also be used for the discovery of coefficients of PDEs from data <cit.>, or the entire tire model can be obtained from online learning <cit.>.   
Also, the term σ(X_t, U_t) and the convection term for the reference points 𝒯 were neglected in the simulations, which need to be constructed or learned from data for more precise estimation. 











 §.§ Normal Cornering
 


We first show preliminary results with a normal cornering task. 
The objective is to keep the vehicle within the lane while driving on a road. 
The safe set ⊂𝕏 is given by 

    = { x ∈𝕏 |  |e| ≤ E_max},

where E_max is the maximum distance from the center line of the lane, and it is assumed to be constant. 
We used a corner in a built-in map of the CARLA simulator (southwest part of Town2 and shown in fig:town2_south_west), and the bound of the lateral error is set to E_max = 1m. 

During the training, the initial vehicle speed was randomly selected within [5m/s, 15m/s], and the vehicle spawn point was randomized for each episode. 
Also, the outlook horizon τ was uniformly randomized between 0 and 5.0s. 
For the function approximator Q, we used a neural network with 3 hidden layers with 32 units per layer and the hyperbolic tangent () as the activation function. 
At the output layer, the sigmoid activation function was used. 








fig:town2_training_progress shows the training progress with the learning rate of 5×10^-4 and the regularization weight of 1 × 10^-4. 
The orange and blue lines show the episode reward and the q-value averaged over a moving window of 500 episodes.   
The solid curves represent the mean of 10 repeated experiments, and the shaded region shows their standard deviation. 
While there is a large variance during the transient phase of learning, it converges to similar values after about 20,000 episodes.









As defined in (<ref>), the safety probability depends on the initial state and the horizon τ. 

fig:SafeProb_e_psi shows the dependency of the safety probability on the lateral error e and the relative heading angle ψ, when the vehicle is placed on the straight part of the road. 
The vehicle state and the horizon are fixed to x_vehicle = [10m/s,  0,  0]^⊤ and τ = 5.0s. 
The learned safety probability is as high as approximately 1 near (e, ψ)=(0,0), and decreases near the boundaries of the road (|e|=1m). 
Also, it decreases when the heading angle is large.  
Similarly, fig:town2_safety_probabilityb shows that the safety probability decreases as the velocity v_x increases and is higher in the inner side of the curve (e < 0). 
Overall, the above result appropriately describes how the safety probability depends on the vehicle state and position in the lane. 
However, the values of the safety probability differed across agents, and further investigation is needed to ensure the accuracy. 
Nevertheless, it has been confirmed that the learned agents safely go through the corner when they are placed near the center of the road. 























 §.§ Safe Drifting
 

Here we present results for safe drifting based on maximal safety probability learning. 
A racing circuit for CARLA developed in <cit.> is used, and the training is performed at a specific corner as shown in fig:mapC_task. 
The road width is about 20m and E_max can be 10.0m, but the safe set was conservatively set as E_max = 8.0m during the training.



The initial vehicle speed was set to 30m/s, and the slip angle β and the yaw rate r were randomized in the range of 

    β∈ [ -25deg, -20deg ], 
        
        
        r ∈ [ 50deg/s, 70deg/s ].

The neural network has the same shape with that used above. 
















For this task, the training progress is strongly influenced by the learning rate as shown in fig:mapC_training_progress. 
As before, the solid curves represent the mean of the averaged rewards of 8 repeated experiments, and the shaded regions show their standard deviations. 


At the initial phase of training, the averaged reward (empirical safety probability) gradually decreases, but after a while it starts to increase.
The timing of this increase is faster for larger learning rates, but the final episode reward increases as the learning rate is reduced from 2e-4 to 5e-5. 
However, at the learning rate of 2e-5, the agents failed to learn safe policy and the reward dropped to zero except for 2 cases out of 8 experiments.
Also, while the learning rate of 5e-5 performed best among the four settings above, there were few cases where the reward dropped to zero, which were excluded from the plot in fig:mapC_training_progress. 
After these trials, we selected an agent trained with the learning rate 5e-5 at a check point of 90,000 episodes that behaved best when tested with closed-loop simulations. 













fig:mapC_closed_loopa shows the vehicle trajectories simulated using the learned agent. 
The cross mark (×) in the figure shows the initial position of the vehicle, and 20 trajectories with different initial conditions of β and r are illustrated. 


It can be confirmed that the learned policy is navigating through the corner without hitting the boundary of the track for different initial conditions. As shown in fig:mapC_closed_loopb,  the slip angle and yaw rate take large values, and the vehicle is safely drifting while cornering. 
These safe behaviors have been learned only from sparse zero and one rewards. 
It is interesting that such a drifting maneuver can be learned by maximizing the safety probability without providing a specific reference trajectories or laborious reward shaping. 
























§ CONCLUSIONS


This paper explored autonomous driving based on the learning of maximal safety probability by using Physics-informed Reinforcement Learning (PIRL). 
Safe drifting was achieved only from sparse binary rewards without providing a specific reference trajectory or laborious reward shaping.
Since this concept is related to forward invariance in the state space, the learned safety probability may be used to safeguard a nominal controller which does not necessarily consider the safety specifications. 
Future work of this paper includes verifying the accuracy of the learned safety probability, and testing the proposed framework in more diverse scenarios. 






§ ACKNOWLEDGMENT


The authors would like to thank Dvij Kalaria for providing vehicle model parameters for CARLA simulation. 

 
 


                                  
                                  
                                  
                                  
                                  









§ APPENDIX



 §.§ DQN based PIRL Algorithm


The PIRL algorithm is presented in Algorithm <ref>.   
After the initializations of the replay memory 𝒟, the function approximator Q, and its target function Q̂, 







the main loop starting at line 4 iterates M episodes, and the inner loop starting at line 6 iterates the time steps of each episode. 



Each episode starts with the initalization of the state s_0 = [h_0, x_0^⊤]^⊤ in line 5, which is sampled from the distribution P_D given by 

    P_D(s_0) = 
        
           1/|Ω_D|,    h_0 = τ_Dx_0 ∈Ω_D, 
    
           0,    otherwise,

where τ_D∈_+ is the time interval of the data acquired through the DQN algorithm, which can be smaller than τ. 
The set Ω_D⊂𝕏 is the domain of possible initial states, and |Ω_D| its its volume. 

At each time step k, through lines 7 to 10, a sample of the transition (s_k, a_k, r_k, s_k^' ) of the state s_k, the action a_k, the reward r_k, and the next state s_k^' is stored in the replay memory 𝒟.  
In lines 11 to 13, a random minibatch 𝒮_D of transitions is taken from 𝒟, and the set 𝒴_D of the target values is calculated using the target q-function Q̂, where the j-th element y_j of 𝒴_D is given by[The index j is independent of the time step k. Random sampling from different time steps improves the stability of learning process by reducing non-stationarity and correlation between updates <cit.>.] 

    y_j =  
        
          r_j,    for terminal s_j^', 
    
          r_j + max_aQ̂(s_j^', a; θ̂),    otherwise.





Then, the loss function L_D 

is given by 

    L_D(θ; 𝒮_D, 𝒴_D) = 1|𝒮_D|∑_j  (y_j - Q(s_j, a_j; θ))^2.













To calculate the loss term L_P, a random minibatch 𝒮_P =  { s_l } is taken at  15. 
Each element s_l=[h_l, x_l^⊤]^⊤ is sampled from the distribution P_P given by 

    P_P(s_l) = 
        
           1/(τ |Ω_P|),    h_l ∈ [0, τ] x_l ∈Ω_P, 
    
           0,    otherwise,

with Ω_P⊂ that specifies the domain where the PDE (<ref>) is imposed. 
In line 16, the set of greedy actions 𝒜_P = { a_l^∗} is calculated by a_l^∗ = max_a Q(s_l, a; θ). 
Then, the PDE loss L_P can be defined as 

    L_P(θ; 𝒮_P, 𝒜_P) =  1|𝒮_P|∑_l W_P(s_l, a_l^∗; θ)^2

with the residual function W_P(s_l, a_l^∗; θ) given by 

    W_P(s_l, a_l^∗; θ) := 
          
       ∂_s Q(s_l,a_l^∗; θ) 
       f̃(s_l, a_l^∗) 
         + 12tr[ σ̃(s_l, a_l^∗)σ̃(s_l, a_l^∗)^⊤
       ∂_s^2 Q(s_l,a_l^∗;θ) 
       ],
 

















For the boundary loss L_B, as stated in line 18, a minibatch 𝒮_B = { s_m } with s_m = [h_m, x_m^⊤]^⊤ is taken by using the 
distribution P_B(s) given by 

    P_B(s) = 
        
           1/(2|Ω_P|),    h_m=0 x_m∈Ω_P, 
    
           1/(2τ |Ω_B|),     h_m ∈ [0, τ ] x∈Ω_B,  
    
           0,    otherwise.

where 
Ω_B⊂∂ stands for the lateral boundary. 

The loss L_B can be defined as 


    L_B(θ; 𝒮_B, 𝒜_B) =  1|𝒮_B|∑_m W_B(s_m, a_m^∗; θ)^2,

with the set 𝒜_B = { a_m^∗} 
of greedy action 


and the residual W_B given by 

    W_B(s_m, a_m^∗; θ) = 
       Q(s_m, a_m^∗; θ) - l_ϵ(x_m).


Finally, at line 21, the parameter θ  is updated to minimize the total loss L based on a gradient descent step.  
The paramter θ̂ of the target function Q̂ used in (<ref>) is updated at the line 22 with a smoothing factor η∈ (0,1]. 








 §.§ Vehicle Model
 

A 3-DOF bicycle model is used following <cit.>. 

The model’s inputs are lateral force of the front wheel F_yf, longitudinal force F_xr and lateral force F_yr of the rear wheel. The governing equations for the model are given as follows:

    dv_xdt   = 1M(F_xr -F_yfsinδ) + r v_x β, 
    dβdt   = 1M v_x(F_yr+F_yfcosδ ) - r, 
    drdt   = 1I_z(L_f F_yfcosδ - L_rF_yr )

where M is the mass of vehicle, L_f and L_
r are the distance from C.G. to front and rear axle, 
and I_z is the vehicle moment of inertia around the z-axis.
When the vehicle enters the extreme driving condition of drifting, the tire force will become progressively saturated, and the linear tire model cannot be used to express the change of the lateral force accurately. 
Here we use a slightly modified version of the Fiala lateral tire force model given in <cit.>. 

To formulate a lane keeping problem, the vehicle dynamics model needs to be combined with the model of the vehicle position and heading angle. 
When they are described in a road coordinate, the road curvature can be viewed as a disturbance pushing the vehicle out of the lane. 
By defining the yaw rate error ψ between the vehicle and the road, its dynamics can be described as 

    dψd t    = r - v_x ρ(t)

where ρ is the curvature of the road. 
Then, 
the lateral error e can be described by the following equation:

    d e d t   =  v_y cosψ + v_x sinψ,

where v_y is the lateral velocity of the vehicle and given as v_y = v_x tanβ. 











IEEEtran













