Decision-Making for Autonomous Vehicles with Interaction-Aware
  Behavioral Prediction and Social-Attention Neural Network
Autonomous vehicles need to accomplish their tasks while interacting with
human drivers in traffic. It is thus crucial to equip autonomous vehicles with
artificial reasoning to better comprehend the intentions of the surrounding
traffic, thereby facilitating the accomplishments of the tasks. In this work,
we propose a behavioral model that encodes drivers' interacting intentions into
latent social-psychological parameters. Leveraging a Bayesian filter, we
develop a receding-horizon optimization-based controller for autonomous vehicle
decision-making which accounts for the uncertainties in the interacting
drivers' intentions. For online deployment, we design a neural network
architecture based on the attention mechanism which imitates the behavioral
model with online estimated parameter priors. We also propose a decision tree
search algorithm to solve the decision-making problem online. The proposed
behavioral model is then evaluated in terms of its capabilities for real-world
trajectory prediction. We further conduct extensive evaluations of the proposed
decision-making module, in forced highway merging scenarios, using both
simulated environments and real-world traffic datasets. The results demonstrate
that our algorithms can complete the forced merging tasks in various traffic
conditions while ensuring driving safety.
Xiao Li, Kaiwen Liu, H. Eric Tseng, Anouck Girard, Ilya Kolmanovsky
Decision-Making for Autonomous Vehicles with Interaction-Aware Behavioral Prediction and Social-Attention Neural Network

###### Abstract

Autonomous vehicles need to accomplish their tasks while interacting with human drivers in traffic. It is thus crucial to equip autonomous vehicles with artificial reasoning to better comprehend the intentions of the surrounding traffic, thereby facilitating the accomplishments of the tasks. In this work, we propose a behavioral model that encodes drivers' interacting intentions into latent social-psychological parameters. Leveraging a Bayesian filter, we develop a receding-horizon optimization-based controller for autonomous vehicle decision-making which accounts for the uncertainties in the interacting drivers' intentions. For online deployment, we design a neural network architecture based on the attention mechanism which imitates the behavioral model with online estimated parameter priors. We also propose a decision tree search algorithm to solve the decision-making problem online. The proposed behavioral model is then evaluated in terms of its capabilities for real-world trajectory prediction. We further conduct extensive evaluations of the proposed decision-making module, in forced highway merging scenarios, using both simulated environments and real-world traffic datasets. The results demonstrate that our algorithms can complete the forced merging tasks in various traffic conditions while ensuring driving safety.

 Autonomous Vehicles, Interaction-Aware Driving, Imitation Learning, Neural Networks, Traffic Modeling

## I Introduction

One of the challenges in autonomous driving is interpreting the driving intentions of other human drivers. The communication between on-road participants is typically non-verbal, and relies heavily on turn/brake signals, postures, eye contact, and subsequent behaviors. In uncontrolled traffic scenarios, e.g., roundabouts [1], unsignalized intersections [2], and highway ramps [3], drivers need to negotiate their order of proceeding. Fig. 1 illustrates a forced merging scenario at a highway entrance, where the ego vehicle in red attempts to merge into the highway before the end of the ramp. This merging action affects the vehicle behind in the lane being merged, and different social traits of its driver can result in different responses to the merging intent. A cooperative driver may choose a lane change to promote the merging process, while an egoistic driver may maintain a constant speed and disregard the merging vehicle. Therefore, understanding the latent intentions of other drivers can help the ego vehicle resolve conflicts and accomplish its task.

In this paper, we specifically focus on the highway forced merging scenario illustrated in Fig. 1 and the objective of transitioning the ego vehicle onto the highway from the ramp in a timely and safe manner. The difficulty of developing suitable automated driving algorithms for such scenarios is exacerbated by the fact that stopping on the ramp in non-congested highway traffic could be dangerous.

The forced merging has been addressed in the automated driving literature from multiple directions. In particular, learning-based methods have been investigated to synthesize controllers for such interactive scenarios. End-to-end planning methods [4] have been proposed to generate control inputs from Lidar point clouds [5] and RGB images [6]. Reinforcement Learning (RL) algorithms have also been considered to learn end-to-end driving policies [7, 8]. A comprehensive survey of RL methods for autonomous driving applications is presented in [9]. Meanwhile, Imitation Learning-based methods have been exploited to emulate expert driving behaviors [10, 11]. However, the end-to-end learning-based controllers lack interpretability and are limited in providing safety guarantees in unseen situations. To address these concerns, researchers have explored the integration of learning-based methods with planning and control techniques. Along these lines, Model Predictive Control (MPC) algorithms have been integrated with the Social-GAN [12] for trajectory prediction and planning [13]. Meanwhile, Inverse RL methods have also been explored to predict drivers' behavior for planning purposes [14, 15, 16]. However, the learning-based modules in these systems may have limited capability to generalize and transfer to unobserved scenarios or behaviors.

There also exists extensive literature on modeling the interactive behaviors between drivers using model-based ap

Fig. 1: Schematic diagram of the highway forced merging scenario: An on-ramp ego vehicle in red is merging onto the highway while interacting with the highway vehicles in grey.
proaches. Assuming drivers maximize their rewards [14], game-theoretic approaches have been proposed to model traffic interactions, such as level-\(k\) hierarchical reasoning framework [17], potential games [18], and Stackelberg games [15, 19]. In the setting of the level-\(k\) game-theoretic models, approaches to estimating drivers' reasoning levels have been proposed [20]. A novel Leader-Follower Game-theoretic Controller (LFGC) has been developed for decision-making in forced merging scenarios [21]. However, solving game-theoretic problems could be computationally demanding and has limited scalability to a larger number of interacting drivers or longer prediction horizons. To be able to account for the uncertainty in the interactions, probabilistic methods, leveraging either Bayesian filter [21, 22] or particle filter [23] with Partially Observable Markov Decision Process (POMDP) [21, 24], have also been implemented to encode and estimate the uncertain intent of other drivers as hidden variables.

In this paper, we consider the Social Value Orientation (SVO) from social psychology studies [25] to model drivers' interactions. The SVO quantifies subjects' tendencies toward social cooperation [26] and has been previously used to model drivers' cooperativeness during autonomous vehicle decision-making in [15, 27]. In addition, researchers have combined SVO-based rewards with RL to generate pro-social autonomous driving behaviors [28, 29] or synthesize realistic traffic simulation with SVO agents [30]. In our proposed behavioral model, we consider both social cooperativeness and the personal objectives of the interacting drivers. Leveraging a Bayesian filter, we propose a decision-making module that accounts for pairwise interactions with other drivers, and computes a reference trajectory for the ego vehicle under the uncertain cooperative intent of other drivers. The method proposed in this paper differs from the previous work [31] in the following aspects: 1) Instead of using an action space with a few coarse action primitives, we synthesize a state-dependent set of smooth and realistic trajectories as our action space. 2) We design a Social-Attention Neural Network (SANN) architecture to imitate the behavioral model that structurally incorporates the model-based priors and can be transferred to various traffic conditions. 3) We develop a decision-tree search algorithm for the ego vehicle's decision-making, which guarantees safety and scalability. 4) We conduct an extensive evaluation of the behavioral model in predicting real-world trajectory and demonstrate the decision-making module capabilities in forced merging scenarios on both simulations and real-world datasets, which is not done in [31].

The proposed algorithm has several distinguished features:

1. The behavioral model incorporates both the driver's social cooperativeness and personal driving objectives, which produces rich and interpretable behaviors.
2. The proposed decision-making module handles the uncertainties in the driving intent using a Bayesian filter and generates smooth and realistic reference trajectories for the downstream low-level vehicle controller.
3. Differing from pure learning-based methods, the designed SANN incorporates social-psychological model-based priors. It imitates the behavioral model and is transferable across different traffic conditions while providing better online computation efficiency.
4. The decision-making module utilizes an interaction-guided decision tree search algorithm, which ensures probabilistic safety and scales linearly with the number of interacting drivers and prediction horizons.
5. The behavioral model is evaluated in predicting real-world trajectories. This model demonstrates good quantitative accuracy in short-term prediction and provides qualitative long-term behavioral prediction.
6. The decision-making module is tested in the forced merging scenarios on a comprehensive set of environments without re-tuning the model hyperparameters. The proposed method can safely merge the ego vehicle into the real-world traffic dataset [32] faster than the human drivers, and into diverse Carla [33] simulated traffic with different traffic conditions.

This paper is organized as follows: In Sec. II, we describe the model preliminaries, including the vehicle kinematics model, the action space with the lane-change modeling, and the choice of model hyperparameters. In Sec. III, we present the behavioral model, which can be utilized to predict interacting drivers' trajectories given their latent driving intentions. In Sec. IV, we discuss the SANN architecture that imitates the behavioral model and is suitable for online deployment. In Sec. V, we introduce the decision-making module of our ego vehicle together with a decision tree search algorithm that incorporates the SANN and improves computation efficiency. In Sec. VI, we report the results of real-world trajectory prediction using the behavioral model for the forced merging test on a real-world dataset and in simulations. Finally, conclusions are given in Sec. VII.

## II System and Model Preliminaries

In this paper, we design a modularized algorithm architecture for decision-making and control of the autonomous (ego) vehicle in the forced merging scenario. In this framework (see Fig. 2), we develop a parameterized behavioral model for modeling the behavior of interacting drivers. Leveraging this model and observed traffic interactions, we can estimate the latent driving intentions of interacting drivers as model parameters. Thereby, we can predict their future trajectories in response to the action of the ego vehicle. Based on the observations, and the predictions, a high-level decision-making module optimizes a reference trajectory for the ego vehicle merging into the target highway lane while ensuring its safety

Fig. 2: Proposed algorithm architecture for autonomous vehicles decision-making and control.
in the traffic. A low-level controller controls the vehicle throttle and steering angle to track the reference trajectory.

This chapter first introduces the vehicle kinematics model in Sec. II-A. Then, we present a state-dependent action space (in Sec. II-B) of the vehicle, which is a set of trajectories synthesized from the kinematics model. We discuss the detailed lane change trajectory modeling in Sec. II-C together with model hyperparameter identification from a naturalistic dataset [32].

### _Vehicle Kinematics_

We use the following continuous-time bicycle model [34] to represent the vehicle kinematics,

\[\left[\begin{array}{c}\dot{x}\\ \dot{y}\\ \dot{\varphi}\\ \dot{v}\end{array}\right]=\left[\begin{array}{c}v\cos(\varphi+\beta)\\ v\sin(\varphi+\beta)\\ \frac{\mu}{l_{r}}\sin(\beta)\\ a\end{array}\right]+\tilde{w}, \tag{1}\] \[\beta=\arctan\left(\frac{l_{r}}{l_{r}+l_{f}}\tan\delta\right),\]

where \(x\), \(v\), and \(a\) are the longitudinal position, velocity, and acceleration of the vehicle center of gravity (CoG), respectively; \(y\) is the lateral position of the CoG; \(\varphi\) is the heading angle of the vehicle; \(\beta\) is the sideslip angle; \(\delta\) is the front wheel steering angle; \(l_{r}\) and \(l_{f}\) denote the distance from the vehicle CoG to the front and rear wheel axles, respectively; \(\tilde{w}\in\mathbb{R}^{4}\) is a disturbance representing unmodeled dynamics.

We assume that all the highway vehicles, together with the ego vehicle, follow this kinematics model. We then derive discrete-time kinematics from (1) assuming zero-order hold with the sampling period of \(\Delta T\) sec. This leads to the discrete-time kinematics model,

\[s_{k+1}^{i}=f(s_{k}^{i},u_{k}^{i})+\tilde{w}_{k}^{i},\;i=0,1,2,\ldots, \tag{2}\]

where the subscript \(k\) denotes the discrete time instance \(t_{k}=k\Delta T\) sec; the superscript \(i\) designates a specific vehicle, where we use \(i=0\) to label the ego vehicle and \(i=1,2\ldots\) for other interacting vehicles; \(s_{k}^{i}=[x_{k}^{i},y_{k}^{i},\varphi_{k}^{i},{v_{k}^{i}}]^{T}\) and \(u_{k}^{i}=[a_{k}^{i},\delta_{k}^{i}]^{T}\) are the state and control vectors of vehicle \(i\) at time instance \(t_{k}\). Then, we can use this discrete kinematics model to synthesize vehicle trajectories. Note that there are other vehicle kinematics and dynamics models that could potentially represent vehicle behaviors [34] more realistically. The model (2) was chosen as it provides adequate accuracy for the purpose of decision-making (planning) of the ego vehicle while it is simple and computationally efficient [35].

### _Trajectory Set as Action Space_

Given the initial vehicle state and the kinematics model (2) neglecting the disturbance \(\tilde{w}_{k}^{i}\) and using different control signal profiles, we can synthesize various trajectories with duration \(N\Delta T\) sec for trajectory prediction and planning. For the \(i\)th vehicle at time \(t_{k}\), we assume that vehicle's action space is \(\Gamma(s_{k}^{i})=\left\{\gamma_{m}(s_{k}^{i})\right\}_{m=k}^{M}\) where each individual element \(\gamma^{(m)}(s_{k}^{i})=\left\{s_{n}^{i}\right\}_{n=k}^{k+M+1}\) is a trajectory of time duration \(N\Delta T\) sec synthesized using a distinct control sequence \(\left\{u_{n}^{i}\right\}_{n=k,\ldots,k+N}\) via the kinematics model (2). We select \(225\) different control sequences such that the number of considered trajectories is finite, i.e., \(M\leq 225\) for all \(\Gamma(s_{k}^{i})\) and all initial state \(s_{k}^{i}\). As shown in Fig. 3, the action space \(\Gamma(s_{k}^{i})\) depends on the current vehicle state \(s_{k}^{i}\) for two reasons: With a fixed control sequence \(\left\{u_{n}^{i}\right\}_{n}\), the resulted trajectory from (2) varies with the initial condition \(s_{k}^{i}\); A safety filter is implemented for \(\Gamma(s_{k}^{i})\) such that all trajectories intersect with the road boundaries are removed, which is also dependent on \(s_{k}^{i}\). The chosen \(225\) control sequences generate trajectories that encompass a set of plausible driving behaviors (see Fig. 3) and suffice tasks of trajectory prediction and planning. Note that the trajectory set can be easily enlarged with more diverse control sequences if necessary.

Meanwhile, we assume a complete lane change takes \(T_{\text{lane}}=N_{\text{lane}}\Delta T\) sec to move \(w_{\text{lane}}\) meters from the center line of the current lane to that of adjacent lanes. We set \(N_{\text{lane}}<N\) to allow trajectory sets to contain complete lane change trajectories. As shown in Fig. 3a), the trajectory set comprises 109 regular driving trajectories for an on-ramp vehicle that is intended to merge. This trajectory set considers varieties of driver's actions such as lane keeping with longitudinal accelerations/decelerations, merging with constant longitudinal speed, merging with longitudinal acceleration/deceleration, accelerating/decelerating before or after merging, etc. Moreover, we also include the behavior of aborting lane change (see Fig. 3b). For a lane-changing vehicle, this is a regular "change-of-mind" behavior to avoid collision with nearby highway vehicles. For longitudinal behaviors, we also assume speed and acceleration/deceleration limits of \([v_{\min},v_{\max}]\) and \([a_{\min},a_{\max}]\) for all vehicle at all times. Namely, the trajectory sets and control sequences satisfy \(a_{n}^{i}\in[a_{\min},a_{\max}]\) and \(v_{n}^{i}\in[v_{\min},v_{\max}]\) for all \(s_{n}^{i}\in\gamma^{(m)}(s_{k}^{i}),n=k,\ldots,k+N+1\) and for all \(\gamma^{(m)}(s_{k}^{i})\in\Gamma(s_{k}^{i}),m=1,\ldots,M(s_{k}^{i})\). The speed limits are commonly known quantities on highways and the longitudinal acceleration/deceleration is typically limited by the vehicle's performance limits.

Fig. 3: Examples of trajectory set \(\Gamma(s_{k}^{i})\) with duration \(6\) sec: (a) A trajectory set of \(M=109\) encompasses behaviors of lane keeping, lane change, and coupled longitudinal and lateral behavior (e.g., lane change with longitudinal acceleration/deceleration). (b) A trajectory set of \(M=129\) contains actions of lane change abortion and re-merge after aborting the previous lane change. Other normal lane change trajectories are in semi-transparent lines. Likewise, the lane change abortion behaviors are also coupled with various longitudinal acceleration/deceleration profiles.
### _Trajectory Hyperparameters and Lane Change Behavior_

We use a naturalistic highway driving High-D [32] dataset to identify the model hyperparameters, i.e., \(v_{\min}\), \(v_{\max}\), \(a_{\min}\), \(a_{\max}\), \(w_{\text{lane}}\), and \(T_{\text{lane}}\). The statistics visualized in Fig. 4 are obtained from data of 110,500 vehicles driven over 44,500 kilometers. The minimum speed is set to \(v_{\min}=2\;\mathrm{m}/\mathrm{s}\) since most of the vehicles have speeds higher than that and the maximum speed limit of the dataset is \(v_{\max}=34\;\mathrm{m}/\mathrm{s}\). The majority of longitudinal accelerations and decelerations of High-D vehicles are within the range of \([a_{\min},a_{\max}]=[-6,6]\;\mathrm{m}/\mathrm{s}^{2}\). The lane width \(w_{\text{lane}}=3.5\;\mathrm{m}\) as in High-D dataset. We select \(T_{\text{lane}}=N_{\text{lane}}\Delta T=4\;\mathrm{sec}\) since most of the High-D vehicles take between \(4\) and \(6\;\mathrm{sec}\) to change lanes. We keep these hyperparameters fixed for the following discussion and experiments. Note that these parameters can be identified similarly to different values in other scenarios if necessary.

In terms of lane change behaviors, given an acceleration sequence \(\left\{a_{k}^{i}\right\}_{k}\), we can derive the steering profile \(\left\{\delta_{k}^{i}\right\}_{k}\) of a lane change trajectory from 5th order polynomials [36],

\[x(t|\{p_{j}\}) =p_{0}+p_{1}t+p_{2}t^{2}+p_{3}t^{3}+p_{4}t^{4}+p_{5}t^{5}, \tag{3}\] \[y(t|\{q_{j}\}) =q_{0}+q_{1}t+q_{2}t^{2}+q_{3}t^{3}+q_{4}t^{4}+q_{5}t^{5},\]

which represents a vehicle lane change between time \(t=0\) and \(t=T_{\text{lane}}\;\mathrm{sec}\). Such lane change trajectories are commonly used in vehicle trajectory planning and control [37, 38].

Suppose, without loss of generality, that the lane change starts and ends at \(t_{0}=0\) and \(t_{N_{\text{lane}}}=T_{\text{lane}}\;\mathrm{sec}\), respectively. The following procedure is utilized to determine the trajectory and steering profile during a lane change: At time \(t_{k}=k\Delta T\;\mathrm{sec}\), given vehicle state \(s_{k}^{i}=[x_{k}^{i},y_{k}^{i},\varphi_{k}^{i},v_{k}^{i}]^{T}\) and lateral target lane center \(y_{\text{target}}\), we first solve for the coefficients \(\{p_{k,j}\}\) and \(\{q_{k,j}\}\) in (3) from the following two sets of boundary conditions,

\[\left\{\begin{array}{ll}x(t_{k})=x_{k}^{i},\;\;\dot{x}(t_{k})=v_{k}^{i},& \;\ddot{x}(t_{k})=a_{k}^{i},\\ y(t_{k})=y_{k}^{i},&\;\dot{y}(t_{k})=\dot{y}_{k}^{i},&\;\ddot{y}(t_{k})=\ddot{ y}_{k}^{i},\end{array}\right. \tag{4}\]

\[\left\{\begin{array}{ll}&x(T_{\text{lane}})=x_{k}^{i}+v_{k}^{i}(T_{\text{ lane}}-t_{k})\\ &+\frac{1}{2}a_{k}^{i}(T_{\text{lane}}-t_{k})^{2},\\ \dot{x}(T_{\text{lane}})=v_{k}^{i}+a_{k}^{i}(T_{\text{lane}}-t_{k}),&\;\ddot{x }(T_{\text{lane}})=a_{k}^{i},\\ y(T_{\text{lane}})=y_{\text{target}},\;\dot{y}(T_{\text{lane}})=0,&\;\ddot{y}(T _{\text{lane}})=0,\end{array}\right.\]

where we assume initial/terminal conditions \(\dot{y}(0)=\dot{y}(T_{\text{lane}})=0\) and \(\ddot{y}(0)=\ddot{y}(T_{\text{lane}})=0\), i.e., zero lateral velocity and acceleration at the beginning and the end of a lane change. Recursively, initial conditions \(\dot{y}(t_{k})=\dot{y}_{k}^{i}\) and \(\ddot{y}(t_{k})=\ddot{y}_{k}^{i}\) at step \(k\) can be computed using (3) with the coefficients \(\{q_{k-1,j}\}\) at previous step \(k-1\); and we assume a constant longitudinal acceleration \(a_{k}^{i}\) throughout the lane change process. Then, we can compute \(s_{k+1}^{i}=[x_{k+1}^{i},y_{k+1}^{i},\varphi_{k+1}^{i},v_{k+1}^{i}]^{T}\) from (3) using the following equations,

\[x_{k+1}^{i}=x(t_{k+1}|\{q_{k,j}\}_{j}),y_{k+1}^{i}=y(t_{k+1}|\{q _{k,j}\}_{j}), \tag{5}\] \[\varphi_{k+1}^{i}=\arctan\left(\dot{y}(t_{k+1}|\{q_{k,j}\}_{j})/ \dot{x}(t_{k+1}|\{q_{k,j}\}_{j})\right),\] \[v_{k+1}^{i}=\dot{x}(t_{k+1}|\{q_{k,j}\}_{j}).\]

Repeating this procedure for \(k=0,1,\ldots,N_{\text{lane}}-1\), we can synthesize a smooth lane change trajectory \(\left\{s_{k}^{i}\right\}_{k=0,\ldots,N_{\text{lane}}}\) with corresponding acceleration sequences \(\left\{a_{k}^{i}\right\}_{k=0,\ldots,N_{\text{lane}}-1}\). Fig. 5 illustrates this approach to the lane change modeling. Given an acceleration sequence \(\left\{a_{k}^{i}\right\}_{k}\) (see Fig. 5b) from one of the \(225\) control sequences \(\left\{u_{k}^{i}\right\}_{k}\), we leverage (3) and produce a smooth lane change trajectory that qualitatively matches with a real-world (High-D) lane change trajectory. Meanwhile, the resulting steering angle profile \(\left\{\delta_{k}^{i}\right\}_{k}\) (see Fig. 5c) is similar to those from human driving [39, 40].

## III Social Behavior Modeling

In this section, we model the two components of drivers' driving incentives that motivate them to take action from the trajectory sets defined in Sec. II. The first component consists of each individual driver's objectives as a personal reward in Sec. III-A. The second component uses an SVO-based reward to incorporate the drivers' social cooperativeness (see Sec. III-B). In Sec. III-C, we integrate this reward model into the interacting vehicle's decision-making process.

### _Driver's Driving Objectives and Personal Rewards_

Similar to the previous work [31], we model the personal reward of the \(i\)th driver who interacts with an adjacent vehicle

Fig. 4: Histogram of vehicle driving statistics in the High-D dataset [32]: (a) Time duration for a complete lane change. (b) Longitudinal velocity. (c) Longitudinal acceleration/deceleration (y-axis in log scale).

Fig. 5: A lane change trajectory synthesized using a given acceleration sequence: (a) Synthesized trajectory using our algorithm compared with a real-world lane change trajectory in the High-D dataset. (b) Designed acceleration sequence \(\left\{a_{k}^{i}\right\}_{k}\). (c) Derived steering sequence \(\left\{\delta_{k}^{i}\right\}_{k}\) from (2).
\(j\) using the following formula,

\[\begin{split}& r\Big{(}\gamma_{n_{1}}^{n_{2}}(s_{k}^{i}),\gamma_{n_{ 1}}^{n_{2}}(s_{k}^{j})|w^{i}\Big{)}=\neg c\Big{(}\gamma_{n_{1}}^{n_{2}}(s_{k}^ {i}),\gamma_{n_{1}}^{n_{2}}(s_{k}^{j})\Big{)}.\\ &\Big{[}\begin{array}{c}h(s_{k+n_{2}}^{i},s_{k+n_{2}}^{j})\quad \tau(s_{k+n_{2}}^{i})\quad e\left(\gamma_{n_{1}}^{n_{2}}(s_{k}^{j})\right)\end{array} \Big{]}\cdot w^{i}\end{split}\;, \tag{6}\]

where \(s_{k}^{i},s_{k}^{j}\) are the current states of the vehicles \(i,j\); \(\gamma_{n_{1}}^{n_{2}}(s_{k}^{i})=\{s_{k+n}^{i_{2}}\}_{n=1}^{n_{2}}\circ\gamma (s_{k}^{i})\) is a segment of the trajectory \(\gamma(s_{k}^{i})\in\Gamma(s_{k}^{i})\), and \(0\leq n_{1}\leq n_{2}\leq N+1\); \(\gamma_{n_{1}}^{n_{2}}(s_{k}^{j})\) is defined likewise; \(\neg\) is the logical negative operator; \(w^{i}\in\mathbb{R}^{3}\) is a vector of weights so that the personal reward is a weighted summation of personal objectives \(h\), \(\tau\), and \(e\). Here, \(c\), \(h\), \(\tau\), and \(e\) are four functions that capture different aspects of drivers' driving objectives:

1. Collision avoidance \(c\in\{0,1\}\): \(c(\gamma_{n_{1}}^{n_{2}}(s_{k}^{i}),\gamma_{n_{1}}^{n_{2}}(s_{k}^{j}))=1\) implies vehicle \(i\) following trajectory \(\gamma_{n_{1}}^{n_{2}}(s_{k}^{i})\) collides with vehicle \(j\) which follows trajectory \(\gamma_{n_{1}}^{n_{2}}(s_{k}^{j})\), and \(c=0\) indicates that two vehicles' trajectories are free of collision with each other. This is used to penalize collisions between trajectories.
2. Safety consciousness \(h\in[0,1]\): If vehicle \(j\) is the leading vehicle of vehicle \(i\), \(h(s_{k+n_{2}}^{i},s_{k+n_{2}}^{i})\) computes a normalized Time-to-Collision (\(TTC\)) at the end of their corresponding trajectories \(\gamma_{n_{1}}^{n_{2}}(s_{k}^{i}),\gamma_{n_{1}}^{n_{2}}(s_{k}^{j})\). A larger \(h\) implies a larger TTC with the leading vehicle. The safety consciousness can encourage vehicles to keep an appropriate headway distance and be conscious of potential collisions.
3. Travelling time \(\tau\in[0,1]\): \(\tau(s_{k+n_{2}}^{i})\) measures the closeness between the vehicle's final state in the trajectory \(\gamma_{n_{1}}^{n_{2}}(s_{k}^{i})\) with its destination, where a larger value implies shorter distance to the goal. Including \(\tau\) in the reward reflects the objective of shortening the traveling time, e.g., merging to the highway as soon as possible for the on-ramp vehicles.
4. Control effort \(e\in[0,1]\): \(e\left(\gamma_{n_{1}}^{n_{2}}(s_{k}^{i})\right)\) takes a lower value if \(\gamma_{n_{1}}^{n_{2}}(s_{k}^{i})\) is a lane-changing trajectory segment or generated with longitudinal acceleration/deceleration. The control effort captures drivers' desire to keep the lane and constant speed to avoid both longitudinal and lateral maneuvers.

We refer the readers to our previous work [31] for more detailed descriptions of the four functions. Similar methods that model the driver's driving objectives have also been reported in [41, 42, 43, 21, 44]. The weight \(w^{i}\) is the latent model parameter in the reward function \(r(\cdot|w^{i})\). Different weights reflect distinct personal goals and, therefore, embed various driving behaviors. For example, a driver considering personal reward with \(w^{i}=[0,0,1]^{T}\) may keep the lane and drive at a constant speed, thereby maximizing the reward via minimizing the control effort. Another driver with weights \(w^{i}=[1,0,0]^{T}\) tries to maximize the headway distance and might change lanes to overtake a leading vehicle if there is one.

### _Social Value Orientation and Multi-modal Reward_

The personal reward function \(r(\cdot|w^{i})\) captures drivers' decision-making as maximizing their own gain in the traffic interaction. However, this model does not encode the behaviors of cooperation and competition. For example, a highway driver observing the merging intention of an on-ramp vehicle might slow down to yield. In social psychology studies [25, 26], the notion of Social Value Orientation (SVO) was proposed to model this cooperative/competitive behavior in experimental games, and it has more recently been applied to autonomous driving [15]. Taking inspiration from this work, we use the driver's SVO to incorporate the personal reward with the driver's tendency toward social cooperation.

To this end, we assume each vehicle \(i\) interacts with the adjacent vehicle \(j\in A(i)\), where \(A(i)\) contains indices of all the adjacent vehicles around vehicle \(i\). We model driver \(i\)'s intention using a multi-modal reward function of the form,

\[\begin{split}& R\Big{(}\gamma_{n_{1}}^{n_{2}}(s_{k}^{i}),\gamma_{n_{ 1}}^{n_{2}}(s_{k}^{-i})|\sigma^{i},w^{i}\Big{)}\\ &=\alpha(\sigma_{i})\cdot r\Big{(}\gamma_{n_{1}}^{n_{2}}(s_{k}^{ i}),\gamma_{n_{1}}^{n_{2}}(s_{k}^{j})|w^{i}\Big{)}\\ &+\beta(\sigma_{i})\cdot\mathbb{E}_{j\in A(i)}\left[r\Big{(} \gamma_{n_{1}}^{n_{2}}(s_{k}^{j}),\gamma_{n_{1}}^{n_{2}}(s_{k}^{i})|w^{j} \Big{)}\right],\end{split} \tag{7}\]

where \(\mathbf{s}_{k}^{-i}=[s_{k}^{0},s_{k}^{i},s_{k}^{i},\dots]\) is the aggregated state of all the adjacent vehicles of vehicle \(i\); \(\mathbf{\gamma}_{n_{1}}^{n_{2}}(\mathbf{s}_{k}^{-i})=[\gamma_{n_{1}}^{n_{2}}(s_{k}^{0}), \gamma_{n_{1}}^{n_{2}}(s_{k}^{1}),\gamma_{n_{1}}^{n_{2}}(s_{k}^{2}),\dots]\) concatenates one possible trajectory segment \(\gamma_{n_{1}}^{n_{2}}(s_{k}^{j})\) for each vehicle \(j\in A(i)\); The SVO \(\sigma^{i}\) is another latent model parameter. It takes one of the four values corresponding to four SVO categories and the values of \(\alpha(\sigma^{i})\) and \(\beta(\sigma^{i})\) are specified as follows

\[(\alpha,\beta)=\left\{\begin{array}{cl}(0,1)&\text{if $\sigma^{i}=$ ``altruistic'',}\\ (1/2,1/2)&\text{if $\sigma^{i}=$ ``prosocial'',}\\ (1,0)&\text{if $\sigma^{i}=$ ``egoistic'',}\\ (1/2,-1/2)&\text{if $\sigma^{i}=$ ``competitive''.}\end{array}\right. \tag{8}\]

In (7), \(\alpha(\sigma^{i})\) weighs the self-reward while \(\beta(\sigma^{i})\) weighs an averaged reward to the other vehicles. We also note that the weight \(w^{j}\) is an internal parameter of vehicle \(j\) and is a latent variable affecting the decision of vehicle \(i\). Similar to [31], we assume \(w^{j}=[1/3,1/3,1/3]\) in Eq. (7) for \(j\in A(i)\). The rationale behind this assumption is that an altruistic or prosocial (or competitive) driver of vehicle \(i\) is likely to cooperate (or compete) with other drivers in all three objectives if they do not know others' actual intentions.

Using this multi-modal reward, we can model each driver's intention to achieve their personal objectives (reflected in \(w^{i}\)) and, to a certain extent, cooperate with others (encoded in \(\sigma^{i}\)). For example, suppose two highway drivers with the same personal weights \(w^{i}=[0,0,1]^{T}\) encounter a merging on-ramp vehicle. A "egoistic" highway driver values the control effort heavily and, therefore is likely to keep the lane at a constant speed and ignore the merging vehicle. On the contrary, a "prosocial" highway driver might consider changing lanes or slowing down to promote on-ramp merging action such that the net reward in (7) is larger.

### _Driving Behavior Model_

Using the multi-modal reward, we can decode/infer drivers' intentions from their actions/trajectories, which can be represented by the model parameters \(w^{i},\sigma^{i}\). We formalize this process into a behavioral model,

\[\gamma^{*}(s_{k}^{i})=\operatorname*{argmax}_{\gamma(s_{k}^{i})\in\Gamma(s_{k}^{i })}\;Q\left(\mathbf{s}_{k}^{-i},\gamma(s_{k}^{i})|\sigma^{i},w^{i}\right), \tag{9}\]
where \(\gamma^{*}(s^{i}_{k})\) is the resulting reference trajectory for vehicle \(i\) and \(Q\) denotes the corresponding cumulative reward function. This cumulative reward admits the following form,

\[\begin{split}& Q\left(\mathbf{s}^{-i}_{k},\gamma(s^{i}_{k})|\sigma^{i}, w^{i}\right)=\underset{\mathbf{\gamma}(\mathbf{s}^{-i}_{k})\in\mathbf{\Gamma}(\mathbf{s}^{-i}_{k})}{ \mathbb{E}}\\ &\left[\sum_{n=0}^{\left\lfloor N^{\prime}/N^{\prime}\right\rfloor }\lambda^{n}R\Big{(}\gamma^{(n+1)N^{\prime}}_{\textsc{n}N^{\prime}}(s^{i}_{k} ),\mathbf{\gamma}^{(n+1)N^{\prime}}_{\textsc{n}N^{\prime}}(\mathbf{s}^{-i}_{k})|\sigma ^{i},w^{i}\Big{)}\right],\end{split} \tag{10}\]

where \(\lambda\in(0,1)\) is a discount factor; the summation is a cumulative reward of vehicle \(i\) over a \(N\Delta T\) sec look-ahead/prediction horizon, and this reward is obtained according to (7); \(N^{\prime}\Delta T\) (\(N^{\prime}<N\)) in second denotes the sampling period that the driver updates its decision; \(\lfloor x\rfloor\) denotes the largest integer lower bound of \(x\in\mathbb{R}\); the expectation averages the reward over all possible aggregated trajectories in the set,

\[\mathbf{\Gamma}(\mathbf{s}^{-i}_{k})=\left\{\mathbf{\gamma}(\mathbf{s}^{-i}_{k}):\gamma(s^{j}_ {k})\in\Gamma(s^{j}_{k}),j\in A(i)\right\}. \tag{11}\]

After obtaining the optimal \(\gamma^{*}(s^{i}_{k})\) using (9), we compute the control signal \(u^{i}_{n}=[a^{i}_{n},\delta^{i}_{n}]^{T}\) at each time step \(n=k,\dots,k+N^{\prime}\) to track this reference trajectory \(\gamma^{*}(s^{i}_{k})\) for one sampling period \(N^{\prime}\Delta T\) sec. Then, we update the reference trajectory using (9) after \(N^{\prime}\Delta T\) sec. Eventually, using the behavioral model (9), we formulate the decision-making process of a highway driver motivated by the reward (10) and a combination of social psychological model parameters \(\sigma^{i},w^{i}\), while the driving behaviors are formalized as a receding-horizon optimization-based trajectory-tracking controller with a horizon of \(\lfloor N/N^{\prime}\rfloor\). However, solving this problem online can be computationally demanding. We can use a neural network to learn the solutions of (9) offline from a dataset, thereby imitating this behavioral model for online deployment.

## IV Interaction-Aware Imitation Learning with Attention Mechanism

Based on (9), the decision-making process of vehicle \(i\) is deterministic given \(s^{i}_{k}\), \(\mathbf{s}^{-i}_{k}\). Instead of learning a one-hot encoding and to include stochasticity in the decision-making process, we prescribe a policy distribution from (10) using a softmax decision rule [45] according to,

\[\begin{split}&\pi\left(\gamma(s^{i}_{k})|\sigma^{i},w^{i},s^{i}_{ k},\mathbf{s}^{-i}_{k}\right)\\ &\propto\exp\left[Q\left(\mathbf{s}^{-i}_{k},\gamma(s^{i}_{k})|\sigma ^{i},w^{i}\right)\right].\end{split} \tag{12}\]

Note that \(\pi\) takes values in \(\mathbb{R}^{225}\), where we assign zero probabilities in \(\pi\) for the unsafe trajectories \(\gamma_{\text{ unsafe}}(s^{i}_{k})\notin\Gamma(s^{i}_{k})\) filtered out in Sec. II-B. Then, we can learn a neural network mapping \(\pi_{NN}\) for imitating the actual behavioral model \(\pi\) using minimization of a modified Kullback-Leibler divergence according to the loss function,

\[\begin{split}&\mathcal{L}\left(\pi,\pi_{NN}|\sigma^{i},w^{i},s^{i }_{k},\mathbf{s}^{-i}_{k}\right)\\ &=\sum\limits_{m=1}^{225}\Big{\{}\pi\left(\gamma^{(m)}(s^{i}_{k} )\right)\cdot\log\Big{[}\pi\left(\gamma^{(m)}(s^{i}_{k})\right)+\epsilon\Big{]} \Big{\}}\\ &-\pi\left(\gamma^{(m)}(s^{i}_{k})\right)\cdot\log\big{[}\pi_{NN} \left(\gamma^{(m)}(s^{i}_{k})\right)+\epsilon\big{]}\Big{\}},\end{split} \tag{13}\]

where a positive constant \(\epsilon\ll 1\) is chosen to avoid zero probability inside the logarithm for numerical stability, and for simplicity, we omit the terms \(\sigma^{i},w^{i},s^{i}_{k},\mathbf{s}^{-i}_{k}\) in the notation of \(\pi\), \(\pi_{NN}\). This loss function \(\mathcal{L}(\cdot)\geq 0\) measures the similarity between two discrete probability distributions where smaller loss implies more similar distributions.

We design a Social-Attention Neural Network (SANN) architecture (see Fig. 6) that comprises three components: The input normalization (Sec. IV-A) derives a set of normalized vectors from inputs \(\sigma^{i},w^{i},s^{i}_{k},\mathbf{s}^{-i}_{k}\) using the highway structural information. Then, we generate a set of feature vectors via an interaction-aware learning process using the attention backbone in Sec. IV-B, and we present the attention mechanism in Sec. IV-C. Finally, using the learned feature vectors, the policy head imitates the policy distribution \(\pi\) from the behavioral model (Sec. IV-D).

### _Input Normalization_

Given different lane dimensions as labeled in Fig. 1, we aim to normalize the inputs \(\sigma^{i},w^{i},s^{i}_{k},\mathbf{s}^{-i}_{k}\) accordingly to facilitate the neural network training. The normalization produces input vectors \(\bar{s}^{i}_{k}\) for vehicle \(i\) and a set of vectors \(\{\bar{s}^{j}_{k}\}\) for \(j\in A(i)\) according to,

\[\bar{s}^{i}_{k}=\left[\begin{array}{c}\left(x^{i}_{k}-\nicefrac{{l}}{{2}}-x _{\text{ramp}}\right)/l_{\text{ramp}}\\ \left(x^{i}_{k}+\nicefrac{{l}}{{2}}-x_{\text{ramp}}\right)/l_{\text{ramp}}\\ \left(y^{i}_{k}-y_{\text{ramp}}\right)/w_{\text{ramp}}\\ \left(\nu^{i}_{k}-v_{\text{min}}\right)/(v_{\text{max}}-v_{\text{min}})\\ w^{i}\end{array}\right],\iota\in\{i\}\cup A(i), \tag{14}\]

where \(l^{\iota}\) is the wheelbase length of vehicle \(\iota\); the first two elements of the feature vector (14) are the normalized longitudinal coordinates of the vehicle rear end and front end; \(w^{\iota}=[1/3,1/3,1/3]\) for all \(\iota\in A(i)\) per Sec. III-B.

Fig. 6: Schematic diagram of our SANN architecture: The attention backbone takes the normalized input vectors and produces their corresponding feature vectors via the attention mechanism [46]. The policy head fits a policy distribution \(\pi_{NN}\) from the resulting feature vectors incorporating the driver \(i\)’s social value orientation \(\sigma^{i}\).
### _Attention Backbone_

We first use a multi-layer perceptron (MLP), i.e., a fully connected neural network, to expand the dimension of the inputs \(\{\tilde{s}_{k}^{i}\}\) individually from \(\mathbb{R}^{7}\) to \(\mathbb{R}^{225}\) according to,

\[z_{\ell}=\sigma_{\text{ReLU}}\left(W_{\ell}z_{\ell-1}+b_{\ell}\right),\ \ \ \ell=1,\ldots,L, \tag{15}\]

where \(W_{\ell}\) and \(b_{\ell}\) are the network parameters of the \(\ell\)th layer; \(\sigma_{\text{ReLU}}(z)=\max\left\{0,z\right\}\) is an element-wise ReLU activation function; \(L\in\mathbb{Z}\) is the number of layers; the inputs are \(z_{0}=\tilde{s}_{k}^{i}\in\mathbb{R}^{5}\), \(\iota\in\{i\}\cup A(i)\); the outputs of the MLP are learned vectors \(z_{k}^{i}=z_{L}\in\mathbb{R}^{225}\), \(\iota\in\{i\}\cup A(i)\). Then, we combine the learned vectors \(\{z_{k}^{i}\}_{*}\), into one matrix \(Z=[z_{k}^{i},\ldots,z_{k}^{j},\ldots]^{T}\) where each row corresponds to a learned feature vector \((z_{k}^{i})^{T}\). The row dimension of \(Z\) can vary with the numbers of interacting vehicles in \(A(i)\), which is undesirable for forming a batch tensor in neural network training [47]. Thus, we consider a maximum of \(N_{z}-1\) adjacent vehicles in \(A(i)\) such that we can append zero rows to \(Z\) and construct \(Z\in\mathbb{R}^{N_{z}\times 225}\). Moreover, we use a mask matrix \(H\in\mathbb{R}^{N_{z}\times N_{z}}\) to mark down the indices of the appended rows for the latter masked self-attention process. The element of \(H\) in the \(i\)th row and \(j\)th column attains \(H_{i,j}=-\infty\) if the \(i\)th or \(j\)th row vector in \(Z\) is an appended zero vector, and obtains \(H_{i,j}=0\) otherwise.

Subsequently, we pass \(Z\) through three identical cascaded blocks (see Fig. 6) using the following formula,

\[\begin{split}\bar{Z}_{\ell}&=\texttt{Attention}(Z_{ \ell-1}|W_{Q,\ell},W_{K,\ell},W_{V,\ell}),\\ Z_{\ell}&=\sigma_{\text{ReLU}}(\bar{Z}_{\ell})+Z_ {\ell-1},\ \ \ell=1,2,3,\end{split} \tag{16}\]

where \(\texttt{Attention}(\cdot)\) denotes the masked self-attention block [46]; \(W_{Q,\ell}\in\mathbb{R}^{225\times|Q|}\), \(W_{K,\ell}\in\mathbb{R}^{225\times|Q|}\), and \(W_{V,\ell}\in\mathbb{R}^{225\times|V|}\) are the parameters named query, key, and value matrix of the \(\ell\)th masked self-attention; the inputs are \(Z_{0}=Z\) and each layer \(\ell=1,2,3\) produces \(Z_{\ell}\in\mathbb{R}^{N_{z}\times 225}\); the summation outside \(\sigma_{\text{ReLU}}\) corresponds to the bypass connection in Fig. 6 from the beginning of a masked self-attention block to the summation symbol \(\oplus\). This bypass connection is called residual connection [48] and is designed to mitigate the vanishing gradient issue in the back-propagation of deep neural networks. We chose to cascade three such blocks via trading-off between empirical prediction performance and computational cost in comparison with those using different numbers of this block.

### _Attention Mechanism_

In the \(\ell\)th attention block (16), we leverage the attention mechanism to interchange information between row vectors of the matrix \(Z_{\ell-1}\) in the learning process. Specifically, the \(\ell\)th masked self-attention block can be represented using the following set of equations,

\[Q_{\ell}=Z_{\ell-1}W_{Q,\ell}, \tag{17a}\] \[K_{\ell}=Z_{\ell-1}W_{K,\ell},\] (17b) \[V_{\ell}=Z_{\ell-1}W_{V,\ell},\] (17c) \[E_{\ell}=(Q_{\ell}K_{\ell}^{T})\circ\left[1/\sqrt{|Q|}\right]_{225 \times 225}+H,\] (17d) \[P_{\ell}=\texttt{Softmax}(E_{\ell},\text{dim=1}),\] (17e) \[\bar{Z}_{\ell}=P_{\ell}V_{\ell}, \tag{17f}\]

where the row vectors of matrices \(Q_{\ell}\), \(K_{\ell}\), \(V_{\ell}\) are called query, key, and value vectors, respectively, learned from the corresponding row vectors in matrix \(Z_{\ell-1}\); \(|Q|\) and \(|V|\) are the dimensions of the query and value vectors; \([x]_{a\times b}\) is a matrix of size \(a\times b\) with all entries equal to \(x\); \(\circ\) is the element-wise Hadamard product; the element \(e_{i,j}\) in \(i\)th row and \(j\)th column of \(E_{\ell}\) represents a normalized similarity score induced by dot-product between \(i\)th query vector in \(Q_{\ell}\) and \(j\)th key vector in \(K_{\ell}\); \(E_{\ell}\) essentially encodes how similar two row vectors in \(Z_{\ell-1}\) are with each other; (17e) apply \(\texttt{Softmax}(\cdot)\) to each column of \(E_{\ell}\); the element \(p_{i,j}\) in \(i\)th row and \(j\)th column of \(P_{\ell}\) is equal to \(\exp(e_{i,j})/\sum_{k}\exp(e_{k,j})\) such that each column vector of \(P_{\ell}\) is a weight vector; each row vector in \(\bar{Z}_{\ell}\) is a weighted summation of value vectors in \(V_{\ell}\) using the weights learned in \(P_{\ell}\).

Notably, in the first layer \(\ell=1\), the mask \(H\) in (17d) sets the similarities between appended zero row vectors and other row vectors in \(Z_{0}=Z\) to \(-\infty\). Subsequently, if the \(i\)th or \(j\)th row vector in \(Z\) is an appended zero vector, (17e) results in weights \(p_{i,j}=0\) in \(P_{1}\) and (17f) yields the \(i\)th or \(j\)th row also a zero vector in \(\bar{Z}_{1}\). Furthermore, the computation in (16) inherits the zero-to-row vectors from \(\bar{Z}_{1}\) to \(Z_{1}\). Inductively, through \(\ell=1,2,3\), the attention backbone preserves the zero rows in \(Z_{0}=Z\). Eventually, the attention backbone outputs \(Z_{3}=[f_{k}^{i},\ldots,f_{k}^{j},\ldots]^{T}\) where each row vector \((f_{k}^{i})^{T}\) is a learned feature vector corresponds to the input row vector \((z_{k}^{i})^{T}\) in \(Z=[z_{k}^{i},\ldots,z_{k}^{j},\ldots]^{T}\).

### _Policy Head_

Similar to (7), we use the SVO category \(\sigma^{i}\) of the \(i\)th driver to combine the learned information \(\{f_{k}^{j}\}_{j\in A(i)}\) from adjacent vehicles \(j\in A(i)\) with \(f_{k}^{i}\) of the driver \(i\) and attain a single vector \(\bar{f}_{k}^{i}\in\mathbb{R}^{225}\) according to,

\[\bar{f}_{k}^{i}=\alpha(\sigma^{i})f_{k}^{i}+\beta(\sigma^{i})\Sigma_{j\in A(i)}f_{k }^{j}. \tag{18}\]

We use another MLP similar to (15) with a bypass residual connection. This is followed by the final element-wise \(\texttt{Softmax}\) activation function that admits the following form,

\[\texttt{Softmax}(z)=\exp(z_{i})/\Sigma_{i}\exp(z_{i}), \tag{19}\]

where \(z\) is a column vector and \(z_{i}\) is the \(i\)th element in \(z\). The \(\texttt{Softmax}\) calculates a probability distribution as the policy distribution output \(\pi_{NN}\in\mathbb{R}^{225}\).

The SANN architecture provides several advantages:

1. The normalization process normalizes the input information using lane and vehicle dimensions which improves the prediction robustness to different highway structural dimensions and vehicle model types.
2. The learning process is interaction-aware. The attention backbone interchanges information between each feature vector corresponding to each interacting driver, which captures the inter-traffic dependencies in the personal reward (6).
3. The learning process is cooperation-aware. The policy head fuses the learned features using the driver's SVO \(\sigma^{i}\). This process emulates (7) and introduces the notion of cooperation/competition into learning.
4. The SANN incorporates the behavioral model priors \(\sigma^{i},w^{i}\) that are later estimated by a Bayesian filter. This offers better online transferability to different drivers.
5. The SANN is permutation invariant, namely, interchanging the order of the inputs in \(\{\tilde{s}_{k}^{j}\}_{j\in A(i)}\) will not alter the value of the learned features \(\{f_{k}^{j}\}_{j\in A(i)}\) or affect the final policies \(\pi_{NN}\). Given the interacting drivers \(j\in A(i)\) geographically located in a 2D plane, the SANN learned policy should not be affected by the artificial information carried in the input order.
6. The SANN can handle variable numbers of inputs up to a maximum number \(N_{z}\). This offers better transferability to different traffic conditions/densities.

These properties might not necessarily be preserved by other networks, e.g., Graph Neural Networks [49], the LSTM [50]. Then, we use this neural network behavioral model to predict the trajectories of interacting vehicles for the decision-making of our ego vehicle in the forced merging scenario.

## V Decision-Making Under Cooperation Intent Uncertainty

We use the Bayesian filter to infer drivers' latent model parameters from observed traffic interactions (Sec.V-A). Then, based on the behavioral model, we incorporate the predictions of interacting drivers' behavior into a receding-horizon optimization-based controller to generate reference trajectories for the ego vehicle (Sec.V-B). In Sec.V-C, we use an interaction-guided decision tree search algorithm to solve this optimization problem and integrate it with the SANN for prediction. The learned SANN improves online prediction efficiency and while the tree-search algorithm offers a probabilistic safety guarantee and good scalability.

### _Bayesian Inference of Latent Driving Intentions_

At each time step \(k\), we assume the ego vehicle \(0\) interacts with adjacent vehicle \(i\in A(0)\) and can observe its nearby traffic state \(s_{k}^{i},\mathbf{s}_{k}^{-i}\). Assuming that the \(i\)th driver's decision-making process follows the policy (12), we need to infer the latent social psychological parameters \(\sigma^{i},w^{i}\) in order to predict the future behavior/trajectory of vehicle \(i\). We assume observation history of traffic around vehicle \(i\) is available,

\[\xi_{k}^{i}=[s_{0}^{i},\mathbf{s}_{0}^{-i},s_{1}^{i},\mathbf{s}_{1}^{-i},\ldots,s_{k}^ {i},\mathbf{s}_{k}^{-i}], \tag{20}\]

which contains the state \(s_{n}^{i}\) of vehicle \(i\) and the aggregated state \(\mathbf{s}_{n}^{-i}\) of adjacent vehicles around vehicle \(i\) for all time step \(n=0,1,\ldots,k\). Then, we use the following Bayesian filter to recursively estimate a posterior distribution of the latent parameters \(\sigma^{i},w^{i}\) from \(\xi_{k}^{i}\).

**Proposition 1**.: _Given a prior distribution \(\mathbb{P}\left(\sigma^{i},w^{i}|\xi_{k}^{i}\right)\) and assume the unmodeled disturbance \(\tilde{w}_{k}^{i}\sim\mathcal{N}(0,\Sigma)\) in (2) follows a zero-mean Gaussian distribution, then the posterior distribution \(\mathbb{P}\left(\sigma^{i},w^{i}|\xi_{k+1}^{i}\right)\) admits the following form,_

\[\begin{split}&\mathbb{P}\left(\sigma^{i},w^{i}|\xi_{k+1}^{i} \right)=\frac{1}{N\left(\xi_{k+1}^{i}\right)}.\\ & D(s_{k+1}^{i},\sigma^{i},w^{i},\dot{s}_{k},\mathbf{s}_{k}^{-i}) \cdot\mathbb{P}\left(\sigma^{i},w^{i}|\xi_{k}^{i}\right),\end{split} \tag{21}\]

_where \(N\left(\xi_{k+1}^{i}\right)\) is a normalization factor, and_

\[\begin{split}& D(s_{k+1}^{i},\sigma^{i},w^{i},\dot{s}_{k}^{i}, \mathbf{s}_{k}^{-i})=\\ &\sum_{\gamma(s_{k}^{i})\in\Gamma(s_{k}^{i})}\left[\mathbb{P} \left(\tilde{w}_{k}^{i}=s_{k+1}^{i}-\gamma_{1}^{1}(s_{k}^{i})\right)\right. \right.\\ &\left.\left.\pi\left(\gamma(s_{k}^{i})|\sigma^{i},w^{i},\dot{s}_ {k}^{i},\mathbf{s}_{k}^{-i}\right)\right].\end{split} \tag{22}\]

We note that (22) represents a transition probability of a driver moving from \(s_{k}^{i}\) to \(s_{k+1}^{i}\) following the kinematics (2) and policy (12). We initialize the Bayesian filter with a uniform distribution. Meanwhile, we can replace \(\pi\) in (22) with \(\pi_{NN}\) for faster online Bayesian inference. We also provide a proof of the proposition in the following:

Proof.: We apply the Bayesian rule to rewrite the posterior distribution according to,

\[\begin{split}&\mathbb{P}\left(\sigma^{i},w^{i}|\xi_{k+1}^{i} \right)=\mathbb{P}\left(\sigma^{i},w^{i}|s_{k+1}^{i},\mathbf{s}_{k+1}^{-i},\xi_{k} ^{i}\right)\\ &=\frac{\mathbb{P}\left(\mathbf{s}_{k+1}^{i}|\xi_{k}^{i}\right)}{ \mathbb{P}\left(\mathbf{s}_{k+1}^{i},\mathbf{s}_{k+1}^{i}|\xi_{k}^{i}\right)}\cdot \mathbb{P}\left(\sigma^{i},w^{i}|\xi_{k}^{i}\right)\\ &=\frac{1}{N\left(\xi_{k+1}^{i}\right)}\cdot D(s_{k+1}^{i}, \sigma^{i},w^{i},s_{k}^{i},\mathbf{s}_{k}^{-i})\cdot\mathbb{P}\left(\sigma^{i},w^ {i}|\xi_{k}^{i}\right),\end{split}\]

where we define \(N\left(\xi_{k+1}^{i}\right)=\frac{\mathbb{P}\left(\mathbf{s}_{k+1}^{i},\mathbf{s}_{k+ 1}^{-i}|\xi_{k}^{i}\right)}{\mathbb{P}\left(\mathbf{s}_{k+1}^{-i}|\xi_{k}^{i}\right)}\) and rewrite the transition probability,

\[\begin{split}& D(s_{k+1}^{i},\sigma^{i},w^{i},\dot{s}_{k}^{i}, \mathbf{s}_{k}^{-i})=\mathbb{P}\left(s_{k+1}^{i}|\sigma^{i},w^{i},\xi_{k}^{i}\right) \\ &=\sum_{\gamma(s_{k}^{i})\in\Gamma(s_{k}^{i})}\left[\mathbb{P} \left(\tilde{w}_{k}^{i}=s_{k+1}^{i}-\gamma_{1}^{1}(s_{k}^{i})\right)\right.\\ &\left.\cdot\pi\left(\gamma(s_{k}^{i})|\sigma^{i},w^{i},\dot{s}_ {k}^{i},\mathbf{s}_{k}^{-i}\right)\right].\end{split} \tag{23}\]

### _Receding-horizon Optimization-based Control_

Leveraging the posterior from (21), we use a receding-horizon optimization-based controller to incorporate the trajectory predictions (12) of interacting vehicles \(i\in A(0)\) and plan a reference trajectory for the ego vehicle according to

\[\gamma^{*}(s_{k}^{0})=\operatorname*{argmax}_{\gamma(s_{k}^{0})\in\Gamma(s_{k}^ {0})}Q_{0}\left(\mathbf{s}_{k}^{-0},\gamma(s_{k}^{0})\right). \tag{24}\]

Similar to Sec. III-C, we compute the control signal \(u_{n}^{0}=[a_{n}^{0},\delta_{n}^{0}]^{T}\) at each time step \(n=k,\ldots,k+N^{\prime}\) to track this reference trajectory \(\gamma^{*}(s_{k}^{0})\) for one control sampling period \(N^{\prime}\Delta T\) sec. Then, we update the reference trajectory using (23) after \(N^{\prime}\Delta T\) sec. Meanwhile, the cumulative reward function \(Q_{0}\left(\mathbf{s}_{k}^{-0},\gamma(s_{k}^{0})\right)\) admits the following form,

\[\begin{split}& Q_{0}\Big{(}\mathbf{s}_{k}^{-0},\gamma(s_{k}^{0})\Big{)}= \frac{1}{|A(0)|}\sum_{i\in A(0)}\\ &\Big{[}\mathbb{E}_{\sigma^{i},w^{i}\sim\mathbb{P}\left(\sigma^{i},w ^{i}|\xi_{k}^{i}\right)}\,Q_{0}^{\prime}\Big{(}\mathbf{s}_{k}^{-0},\gamma(s_{k}^{0})| \sigma^{i},w^{i}\Big{)}\Big{]},\end{split} \tag{25}\]

where the function value \(Q_{0}^{\prime}\) is computed according to,

\[\begin{split}& Q_{0}^{\prime}\Big{(}\mathbf{s}_{k}^{-0},\gamma(s_{k}^{0})| \sigma^{i},w^{i}\Big{)}=\mathbb{E}_{\gamma(s_{k}^{i})\sim\pi\left(\cdot|\sigma^{i},w^{i},s_{k}^{i},\mathbf{s}_{k}^{-i}\right)}\\ &\Big{[}\sum\limits_{n=0}^{|N^{\prime}|}r_{0}\Big{(}\gamma_{n^{ \prime}N^{\prime}}^{(n+1)N^{\prime}}(s_{k}^{0}),\gamma_{n^{\prime}N^{\prime}}^{(n+1)N^ {\prime}}(s_{k}^{i})\Big{)}\Big{]},\end{split} \tag{26}\]

and the ego vehicle acts to minimize its traveling time and avoid collision, thereby the ego reward function \(r_{0}\) attains the following form,

\[r_{0}\left(\gamma_{n_{1}}^{n_{2}}(s_{k}^{0}),\gamma_{n_{1}}^{n_{2}}(s_{k}^{i}) \right)=-\mathbb{C}\left(\gamma_{n_{1}}^{n_{2}}(s_{k}^{0}),\gamma_{n_{1}}^{n_{2}}(s_{k}^{ i})\right)\cdot\tau(
expectation of the reward function with respect to the behavioral model parameters \(\sigma^{i},w^{i}\), while (25) samples trajectory predictions \(\gamma(s^{i}_{k})\) of vehicle \(i\) from the policy \(\pi\) conditioned on \(\sigma^{i},w^{i}\). In (25), \(\pi\) can be replaced by \(\pi_{NN}\) learned by the SANN to speed up the computations. Nonetheless, solving the problem (23) requires an exhaustive search over the trajectory set \(\Gamma(s^{0}_{k})\) that can be computationally demanding. Instead, we can treat the trajectory set \(\Gamma(s^{0}_{k})\) as a decision tree (see Fig. 3) and a tree-search-based algorithm can be developed to improve the computation efficiency.

### _Interaction-Guided Decision Tree Search_

```
0:\(\Gamma(s^{0}_{k})\), \(A(0)\), \(\pi_{NN}\), and \(\xi^{i}_{k}\), \(\Gamma(s^{i}_{k})\), \(\mathbb{P}\left(\sigma^{i},w^{i}|\xi^{i}_{k-1}\right)\) for all \(i\in A(0)\)
1: initialize \(Q_{0}\left(\mathbf{s}^{-0}_{k},\gamma(s^{0}_{k})\right)=0\) for all \(\gamma(s^{0}_{k})\in\Gamma(s^{0}_{k})\)
2:\(\mathbb{P}\left(\sigma^{i},w^{i}|\xi^{i}_{k}\right)\propto\left[\sum_{\gamma(s^ {i}_{k})\in\Gamma(s^{0}_{k})}\mathbb{P}\left(\tilde{w}^{i}_{k}=s^{i}_{k+1}- \gamma^{1}_{1}(s^{i}_{k})\right)\cdot\pi_{NN}\left(\gamma(s^{i}_{k})|\sigma^{i },w^{i},\mathbf{s}^{i}_{k},\mathbf{s}^{-i}_{k}\right)\right]\cdot\mathbb{P}\left( \sigma^{i},w^{i}|\xi^{i}_{k-1}\right)\) for all \(i\in A(0)\)\(\triangleright\) Bayesian filter using SANN imitated behavioral model
3:\(\mathbb{P}\left(\gamma(s^{0}_{k})|\xi^{i}_{k}\right)=\sum_{\sigma^{i},w^{i}} \pi_{NN}\left(\gamma(s^{i}_{k})|\sigma^{i},w^{i},\mathbf{s}^{i}_{k},\mathbf{s}^{-i}_{k }\right)\cdot\mathbb{P}\left(\sigma^{i},w^{i}|\xi^{i}_{k}\right)\) for all \(i\in A(0)\)\(\triangleright\) interaction behavior prediction using SANN imitated behavioral model
4: sort \(A(0)\) w.r.t. \(\left((x^{0}_{k}-x^{i}_{k})^{2}+(y^{0}_{k}-y^{i}_{k})^{2}\right)^{1/2}\), \(i\in A(0)\) in a descending order
5:for\(i\in A(0)\)do\(\triangleright\) prioritize search over closer interactions
6:for\(\gamma(s^{0}_{k})\in\Gamma(s^{0}_{k})\)do\(\triangleright\) parallel search
7:for\(n=0-\lfloor N/N^{\prime}\rfloor\)do\(\triangleright\) over prediction horizons
8:\(c_{n}=\sum_{\gamma(s^{i}_{k})\in\Gamma(s^{0}_{k})}\mathbb{P}\left(\gamma(s^ {i}_{k})|\xi^{i}_{k}\right)\)\(\cdot c\left(\gamma^{(n+1)N^{\prime}}(s^{0}_{k}),\gamma^{(n+1)N^{\prime}} _{N^{\prime}}(s^{i}_{k})\right)\)\(\triangleright\) probability of collision with \(i\)
9:if\(c_{n}\)\(\triangleright\) 0.5 then \(\triangleright\) trim unsafe decision tree branch
10:\(\Gamma(s^{0}_{k})\leftarrow\Gamma(s^{0}_{k})\setminus\Gamma_{\text{ unsafe}}(s^{0}_{k})\), \(\Gamma_{\text{ unsafe}}(s^{0}_{k}):=\left\{\gamma\in\Gamma(s^{0}_{k})|\gamma^{(n +1)N^{\prime}}_{N^{\prime}}=\gamma^{(n+1)N^{\prime}}_{N^{\prime}}(s^{0}_{k})\right\}\), and terminate all parallel search branches in \(\Gamma_{\text{ unsafe}}(s^{0}_{k})\)
11:else
12:\(Q_{0}\left(\mathbf{s}^{-0}_{k},\gamma(s^{0}_{k})\right)\gets Q_{0}\left(\mathbf{s} ^{-0}_{k},\gamma(s^{0}_{k})\right)+\lambda^{n}\cdot r_{0}\left(\gamma^{(n+1)N^ {\prime}}_{N^{\prime}}(s^{0}_{k}),\gamma^{(n+1)N^{\prime}}_{N^{\prime}}(s^{i} _{k})\right)\cdot\mathbb{P}\left(\gamma(s^{i}_{k})|\xi^{i}_{k}\right)\)\(\triangleright\) update discounted cumulative reward
13:endif
14:endfor
15:endfor
16:endfor
17:\(\gamma^{*}(s^{0}_{k})=\operatorname*{argmax}_{\gamma(s^{0}_{k})\in\Gamma(s^ {0}_{k})}Q_{0}\left(\mathbf{s}^{-0}_{k},\gamma(s^{0}_{k})\right)\)
18:return\(\gamma^{*}(s^{0}_{k})\)
```

**Algorithm 1** Interaction-Guided Decision Tree Search

For online deployment, we incorporate the SANN-imitated behavioral model \(\pi_{NN}\) into a decision tree search Algorithm 1 to facilitate the Bayesian filtering, trajectory prediction of interacting vehicles, and decision-making of the ego vehicle. As shown in Fig. 3, the trajectory set \(\Gamma(s^{0}_{k})\) can be viewed as a decision tree and is initiated from the current state \(s^{0}_{k}\) as the tree root. Each trajectory \(\gamma(s^{0}_{k})\) is a branch in the decision tree \(\Gamma(s^{0}_{k})\), each state in a trajectory is a node, and each (directed) edge encodes reachability from one node to another in a certain trajectory/branch. Meanwhile, two distinct branches \(\gamma^{(m_{1})}(s^{0}_{k})\), \(\gamma^{(m_{2})}(s^{0}_{k})\) can share a same trajectory segments, i.e., \((\gamma^{(m_{1})})^{n_{1}}_{0}(s^{0}_{k})=(\gamma^{(m_{2})})^{n_{1}}_{0}(s^{0}_{k })=\gamma^{n_{1}}_{0}(s^{0}_{k})\). Algorithm 1 searches over this decision tree, updates the cumulative reward (24) for each branch, and trims unsafe branches, thereby, improving the searching efficiency. For example in Fig. 3b), the on-ramp ego vehicle is trying to merge where there is a following highway vehicle in grey. The normal lane-changing trajectories/branches share the same subset of initial trajectory segments \(\gamma^{n_{1}}_{0}(s^{0}_{k})\) and are highly likely to cause a collision with the highway vehicle. Therefore, we can trim all the lane change trajectories from the decision tree (shown by semi-transparent lines in Fig. 3b) and terminate further searches along these unsafe branches.

This process is formalized in Algorithm 1. In lines 2 and 3, we use the SANN behavioral model \(\pi_{NN}\) to update the posterior distribution in the Bayesian filter (see Proposition 1) and predict trajectory distributions that are used to compute the reward (24). Since a closer interacting vehicle is more likely to collide with the ego vehicle in the near future, we rank the indices of the adjacent vehicles in \(A(0)\) in descending order with respect to their Euclidean distances to our ego vehicle. The three for-loops in line 5,6,7 of Algorithm 1 search over interactions with different vehicles, branches, and prediction horizons of a branch, respectively. In lines 8-13, the sorted set \(A(0)\) prioritizes collision checks with closer interacting vehicles and trims the unsafe branches as early as possible. We trim all branches with the common set of nodes \(\gamma^{(n+1)N^{\prime}}_{nN^{\prime}}(s^{0}_{k})\) if the ego vehicle of trajectory segment \(\gamma^{(n+1)N^{\prime}}_{nN^{\prime}}(s^{0}_{k})\) has a probability of collision with the \(i\)th vehicle higher than a threshold of 0.5. Otherwise, we will update the cumulative reward according to line 12 in Algorithm 1. Eventually, in line 17, we solve (23) by simply choosing the branch with the maximum cumulative reward. We also note that the three for-loops enable linear scalability of this algorithm with respect to both the number of interacting drivers in \(A(0)\) and the number of prediction horizons \(\lfloor N/N^{\prime}\rfloor\).

## VI Simulation and Experimental Results

We first present both qualitative and quantitative results of real-world trajectory prediction using our Behavioral Model and the Bayesian Filter in Sec. VI-A. We report the performance of the SANN imitation learning in Sec. VI-B. We provide extensive evaluations of the proposed decision-making framework in forced merging tasks using our simulation environment (Sec. VI-C), the real-world High-D traffic dataset [32] (Sec. VI-D), and the Carla simulator [33] (Sec. VI-E). We note that we **do not** need to re-tune the model hyperparameters or re-train the SANN for different forced merging evaluation environments. The demonstration videos are available in [https://xiaolisean.github.io/publication/2023-10-31-TCST2024](https://xiaolisean.github.io/publication/2023-10-31-TCST2024)

### _Real-world Trajectory Prediction_

We present qualitative (Fig. 8) and quantitative results (Fig. 7) of predicting real-world trajectories using our behavioral model and the Bayesian filter. In our first task, we aim to reproduce a real-world trajectory from a naturalistic High-D [32] dataset. In a High-D traffic segment (Fig. 8) of 12 sec, we identify a target vehicle \(i\) (in green) that is overtaking its
leading vehicle \(2\) (in orange). We initialize a virtual vehicle (in red) at \(t_{k}=0\) using the actual vehicle state \(s^{i}_{k}\), and we set \(\sigma^{i}=\) "competitive", \(w^{i}=[0,1,0]^{T}\) in the behavioral model (9). Afterwards, we control this virtual vehicle using (9) assuming no tracking error, and we set the control sampling period to \(N^{\prime}\Delta T=0.5\) sec. We compare the synthesized trajectories using our behavioral model with the actual ones in Fig. 8. Our behavioral model demonstrates a good prediction accuracy from 0-6 sec, which is adequate for a predictive controller with a sampling period \(N^{\prime}\Delta T\leq 6\) sec. We also note that the prediction error is large at \(t_{k}=12\) sec because the error accumulates over longer prediction windows. Our prediction captures the overtaking trajectories and qualitatively demonstrates the effectiveness of our method in modeling real-world driving behavior.

Fig. 7 summarizes the error statistics of the trajectory prediction. In online application scenarios, given an observed interacting history \(\xi^{i}_{k}\), we recursively infer the latent behavioral model parameters \(\sigma^{i},w^{i}\) using the Bayesian filter (21) as a posterior distribution \(\mathbb{P}\left(\sigma^{i},w^{i}|\xi^{i}_{k}\right)\). Subsequently, the interacting vehicles' trajectories are predicted as a distribution using policy (12) according to \(\mathbb{P}\left(\gamma(s^{i}_{k})|\xi^{i}_{k}\right)=\sum_{\sigma^{i},w^{i}} \pi\left(\gamma(s^{i}_{k})|\sigma^{i},w^{i},s^{i}_{k},\mathbf{s}^{-i}_{k}\right) \cdot\mathbb{P}\left(\sigma^{i},w^{i}|\xi^{i}_{k}\right)\). We quantify the prediction error between the actual trajectory \(\hat{\gamma}(s^{i}_{k})=\left\{\hat{s}^{i}_{n}\right\}_{n}^{k+k^{\prime}}\) and the predicted trajectory distribution \(\mathbb{P}\left(\gamma(s^{i}_{k})|\xi^{i}_{k}\right)\) using the following metric

\[\begin{split}&\operatorname{\mathbf{err}}\left(\hat{\gamma}(s^{i}_{ k}),\xi^{i}_{k}\right)=\mathbb{E}_{\gamma(s^{i}_{k})\sim\mathbb{P}\left( \gamma(s^{i}_{k})|\xi^{i}_{k}\right)}\\ &\left[\frac{1}{k^{\prime+1}}\sum\limits_{s^{i}_{k}\in\gamma(s^{ i}_{k}),n=k}^{n=k+k^{\prime}}\left\|\begin{bmatrix}x^{i}_{n}-\hat{x}^{i}_{n}\\ y^{i}_{n}-\hat{y}^{i}_{n}\end{bmatrix}\right\|_{2}\right],\end{split} \tag{26}\]

where \(k^{\prime}\Delta T\) in second is the duration of the actual trajectory \(\hat{\gamma}(s^{i}_{k})\). This metric computes the expected \(\ell_{2}-\)norm error in position prediction averaged over time steps.

We sample different traffic segments of different duration from the High-D dataset, and each traffic segment is bisected by time instance \(t_{k}\) into training segments \(\xi^{i}_{k}\) and prediction segment \(\hat{\gamma}(s^{i}_{k})\) corresponding to a sampled vehicle \(i\). We apply the aforementioned procedure to each training segment \(\xi^{i}_{k}\) and calculate the prediction error (26) using the corresponding prediction segment \(\hat{\gamma}(s^{i}_{k})\). Meanwhile, in the sequel, we assume \(w^{i}\) in a finite set

\[W=\left\{\begin{matrix}[0,0,1],\,[0,1,1]/2,\,[0,1,0],\,[1,1,1]/3,\\ [1,0,1]/2,\,[1,1,0]/2,\,[1,0,0]\end{matrix}\right\} \tag{27}\]

Fig. 8: Reproducing real-world overtaking trajectory: The trajectories of the virtual vehicle \(i\) (in red) are synthesized using our behavioral model (9) considering its driver of model parameters \(\sigma^{i}=\) “competitive”, \(w^{i}=[0,1,0]^{T}\). This virtual “competitive” driver overtakes the vehicle \(2\in A(i)\) minimizing the traveling time \(\tau\). The resulting trajectories in red match the actual trajectories in green.

Fig. 7: Error statistics of trajectory prediction comprises 6,774 High-D trajectories of in total 28,078 driving seconds: (a) Each grid reports the mean of prediction errors using \(\xi^{i}_{k}\), \(\hat{\gamma}(s^{i}_{k})\) of the same lengths. (b) Each subplot visualizes the mean prediction errors (red lines) corresponding to \(\hat{\gamma}(s^{i}_{k})\) of the same duration versus variable duration of \(\xi^{i}_{k}\). We use red shaded areas to denote the one standard deviation and use blue shaded areas to represent the minimum/maximum error bounds.
to reduce the dimension of parameter space \((\sigma^{i},w^{i})\). Subsequently, we have 22 possible combinations of \((\sigma^{i},w^{i})\): We assign 7 different \(w^{i}\in W\) to three \(\sigma^{i}\neq\) "altruistic"; if \(\sigma^{i}=\) "altruistic", the weights \(w^{i}\) do not matter for the altruistic driver as defined in (7), (8).

As shown in Fig. (a)a, the mean prediction errors are below \(4\ \mathrm{m}\) for predictions of 1-9 sec ahead and using training segments of 1-18 sec. We also note that longer training segments \(\xi_{k}^{i}\) (see Fig. (b)b) reduce both the prediction error standard deviation and its maximum values. However, for longer prediction windows of 7-9 sec (see Fig. (b)b), the error accumulates and leads to larger standard deviations and mean errors which are also observed in Fig. 8. For shorter prediction windows of 1-6 sec, we have a maximum error below \(5\ \mathrm{m}\) for most of the cases and the standard deviation is smaller than \(1.5\ \mathrm{m}\) (see Fig. (b)b). The results in Fig. 8 and Fig. 7 provide evidence that our algorithms have good quantitative accuracy over shorter prediction windows, and good qualitative accuracy over longer prediction windows. Based on these considerations, we set the trajectory length as \(N\Delta T=6\ \mathrm{sec}\), so that we have a good prediction performance over a shorter prediction window of 6 sec. This duration covers a complete lane change of \(T_{\text{lane}}=4\ \mathrm{sec}\), and suffice the task of predicting interacting vehicles' trajectories for ego vehicle control.

### _Imitation Learning with SANN_

The goal of imitation learning is to train the SANN to mimic the behavioral model. Namely, the predicted policy \(\pi_{\text{NN}}\) should match the actual one \(\pi\) from behavioral model (12) accurately. We leverage the High-D dataset [32] to synthesize realistic traffic \(s_{k}^{i},\mathbf{s}_{k}^{-i}\) for training the SANN. We randomly sample a frame from the High-D traffic together with a target vehicle \(i\), thereby we can extract states \(s_{k}^{i},\mathbf{s}_{k}^{-i}\) of vehicles \(i\) and its interacting vehicles. Together with sampled parameters \(\sigma^{i},w^{i}\), we can compute the actual policy distribution \(\pi\left(\cdot|\sigma^{i},w^{i},s_{k}^{i},\mathbf{s}_{k}^{-i}\right)\) using (12). We repeat this procedure to collect a dataset of 183,679 data points. We decompose the dataset into training (70%), validation (15%), and test datasets (15%). The MLP in the Attention Backbone has three layers of sizes 7, 32, 225, respectively. The MLP in the Policy Head has two layers of sizes 225, 225. Furthermore, we set \(|Q|=32\), \(|V|=225\), and the maximum number of feature vectors as \(N_{z}=9\).

We use Python with Pytorch [47] to train the SANN using the training dataset and the loss function (13). We train the SANN for 1000 epochs using a Batch Stochastic Gradient Descent algorithm with momentum (batch size of 200). We set the initial learning rate and momentum to \(0.01\) and \(0.9\), respectively. We also adopt a learning rate scheduler that decreases the learning rate by half every 200 epochs. The training process takes in total 7 hours (25 sec per epoch) on a computer with 16 GB RAM and Intel Xeon CPU E3-1264 V3. We evaluate the performance of the SANN in the task of predicting the policy distributions in the test dataset, and the results are reported in Fig. 9. For the majority of the test cases, the SANN achieves losses smaller than 0.4 which corresponds to a good statistical performance. Moreover, in the example (see Fig. 9) with relatively larger losses of 0.8514, the learned policy (in blue) also qualitatively matches the actual one (in red). To conclude, the SANN imitates the proposed behavioral model with good accuracy. Hence, as presented in Algorithm 1, we use this SANN learned policy to perform Bayesian filtering and trajectory prediction to facilitate the online decision-making for our ego vehicle.

### _Forced Merging in Simulations_

We set up a simulation environment (see Fig. 10) where the interacting vehicle is controlled by the behavioral model (9) with \(\sigma^{1}=\) "altruistic". We use the proposed Algorithm 1 to control the ego vehicle. For this and the following experiments, we set the control sampling period to \(N^{\prime}\Delta T=0.5\ \mathrm{sec}\) for both Algorithm 1 and the behavioral model. Instead of passively inferring the driver's intention from its behavior, the Algorithm 1 controls the ego vehicle and exhibits a merging strategy with proactive interaction to test the interacting driver's intention.

Specifically, in Fig. (b)b), the interacting driver 1 first drives at a constant speed from \(0\) to \(0.5\) sec. Meanwhile, from \(0\) to \(1\) sec, the ego vehicle 0 is not certain if driver

Fig. 9: Test statistics and examples of imitation learning on the test dataset. The histogram presents the loss statistics between the SANN learned policy distributions \(\pi_{NN}\) and the actual ones \(\pi\) computed from the behavioral model (12). Five qualitative examples are visualized in call-out boxes: The \(y\)-axis shows the probability of driver \(i\) taking a certain trajectory \(\gamma(s_{k}^{i})\). For comparison, policies \(\pi\) and \(\pi_{NN}\) are plotted above and below the dashed line, respectively, in mirror reflection
1 will yield the right of way for its merging attempt and, therefore it tentatively merges after vehicle 1 with longitudinal deceleration. Meanwhile, in the time interval between \(0.5\) and \(1.5\) sec, the "altruistic" driver of vehicle 1 notices the merging intention of the ego vehicle and decelerates to promote merging. In the aforementioned interactions, the ego vehicle gradually updates its belief of the latent model parameters \(\sigma^{1},w^{1}\) of driver 1. As shown in Fig. 10a), our algorithm correctly infers the "altruistic" identity of driver 1. Subsequently, the ego vehicle aborts the merging action due to safety concerns with longitudinal acceleration to build up speed advantage for future merging. Then, in the time interval between \(2.0\) and \(5.5\) sec, being aware of the yielding behavior, the ego vehicle re-merges before vehicle 1 with longitudinal acceleration. This simulation example provides evidence that our method can effectively interpret the driving intentions of other drivers. Moreover, our decision-making module can leverage this information to facilitate the forced merging task while ensuring the safety of the ego vehicle.

### _Forced Merging in Real-world Traffic Dataset_

We further evaluate the performance of our method in the High-D [32] real-world traffic dataset. There are 60 recordings in the High-D dataset where the recordings 58-60 correspond to highways with ramps. In recordings 58-60, we identify in total 75 on-ramp vehicles (High-D target vehicles) that merge into highways. For each one of the 75 vehicles, we extract its initial state at a recording frame when it appears on the ramp. Then, we initialize a virtual ego vehicle using this state and control this virtual ego vehicle using our decision-making module. Other vehicles are simulated using the actual traffic recordings, where we neglect the interaction between the virtual ego vehicle and the High-D target vehicle. Eventually, we generate 75 forced merging test cases repeating this procedure over all target vehicles. We visualize two test cases out of 75 as examples in Figs. 11 and 12. The two interacting vehicles 1 and 2 in Fig. 11 are approximately driving at constant speeds. Therefore, our decision-making module accelerates the virtual ego vehicle to a comparable speed and merges the virtual ego vehicle in between the two interacting vehicles. In the second example (see Fig. 12), our decision-making module can interpret the yielding behavior of vehicle 2 and, thereby merge the ego vehicle highway in a timely manner. In both examples, our algorithm is able to complete the forced-merging task faster than the real-world human drivers (in green boxes).

Meanwhile, we identify a test case failure if our method fails to merge the virtual ego vehicle into the highway before the end of the ramp, or if the ego vehicle collides with other vehicles and road boundaries. Otherwise, we consider it a success case. In the three recordings, we have 18, 21, and 36 test cases, respectively. Our method can achieve a success rate of 100% in all recordings. We also report the results of the time it took to merge the virtual ego vehicle in Fig. 13. Compared to the actual human driver of the target vehicle (green bars),

Fig. 11: A forced merging evaluation example on the High-D dataset: Our ego vehicle (in red) accelerates first to create sufficient gaps between highway vehicles 1 and 2, which are driving approximately at constant speeds. The ego vehicle successfully merges into the highway in \(5.2\) sec which is faster than the actual human driver (in green).

Fig. 10: A forced merging example of proactive interaction in a simulation: (a) Each subplot reports a posterior distribution of \(\mathbb{P}\left(\sigma^{i},w^{i}|\xi_{k}^{i}\right)\) at a certain time instance \(t_{k}\) from the Bayesian filter (21). The \(x\)-axis shows 7 cases of \(w^{1}\in W\), and the \(y\)-axis shows three different SVO categories with \(\sigma^{i}=\) “altruistic” stands along; (b) Each subplot visualizes a frame of highway interaction between the ego vehicle and vehicle 1 controlled by Algorithm 1 and behavioral model (9), respectively.
our decision-making module can merge the ego vehicle into the highway using a shorter time. Meanwhile, for 64 out of the 75 test cases, our method completes the task within \(5\ \mathrm{sec}\) while we model a complete lane change with \(T_{\text{lane}}=4\ \mathrm{sec}\) in Sec. II-C. The results demonstrate that our method can successfully complete the forced merging task with traffic in real-world recordings in a timely and safe manner. Though the traffic is realistic in the dataset, we also note that there is no actual interaction between the virtual ego vehicle and other vehicles. Namely, the recorded vehicles in the dataset will not respond to the action of our virtual ego vehicle.

### _Forced Merging in Diverse Carla Traffics_

We set up a forced merging scenario in the Carla simulator [33] (see Fig. 15) and test our decision-making module in diverse and reactive simulated traffic. The vehicles in the traffic are controlled by the default Carla Traffic Manager. In the previous experiments, we assumed that the ego vehicle was able to accurately track the reference trajectories \(\gamma^{*}(s_{k}^{0})\) from Algorithm 1. To mimic the real application scenario of our algorithm, we developed a PID controller for reference trajectory tracking. As visualized in the system diagram of Fig. 2, Algorithm 1 in the high-level decision-making module updates optimal reference trajectories \(\gamma^{*}(s_{k}^{0})\) every \(N^{\prime}\Delta T=0.5\ \mathrm{sec}\). Meanwhile, to track this reference trajectory \(\gamma^{*}(s_{k}^{0})\), the low-level PID controller computes the steering and throttle signal of the actual vehicle plant (simulated by Carla using Unreal Engine) at 20 Hz.

Fig. 16 illustrates our method capability to merge the ego vehicle into a dense highway platoon in Carla. From \(0\) to \(4.60\ \mathrm{sec}\), the ego vehicle attempts to merge between vehicles 1 and 2 and, eventually aborts lane change due to safety concerns. From \(4.60\ \mathrm{to}\ 6.12\ \mathrm{sec}\), vehicle 3 is decelerating due to a small headway distance. And vehicle 2 is accelerating seeing an enlarged headway distance as a result of the acceleration of vehicle 1 before \(4.60\ \mathrm{sec}\). Consequently, the gap between vehicles 2 and 3 is enlarged due to this series of interactions from \(4.60\ \mathrm{to}\ 6.12\ \mathrm{sec}\). Taking advantage of these interactions, our ego vehicle decides to accelerate and merge before vehicle \(3\ \mathrm{at}\ 4.60\ \mathrm{sec}\).

Moreover, we are interested in quantifying the state of the traffic that the ego vehicle directly interacts with, i.e., the traffic in the lane near the ramp. We first define the traffic Region Of Interest (ROI) as the traffic in the near-ramp lane between the longitudinal position \(x_{\text{ramp}}-l_{\text{ramp}}\) and \(x_{\text{ramp}}\) (see Fig. 1). Subsequently, we define the following two variables: We compute the average traffic flow as the number of vehicles entering this traffic ROI over the entire simulation time; and we calculate the traffic density as the number of vehicles located in this traffic ROI over the length of this region \(l_{\text{ramp}}\) at a certain time instance. Then, we average the traffic density over measurements taken every \(\Delta T=0.05\ \mathrm{sec}\) which yields the average traffic density.

The two variables quantify the averaged speed and density of the near-ramp traffic that the ego vehicle directly interacts with. For the example in Fig. 16, the traffic in the ROI has an average traffic density of \(0.055\ \mathrm{vehicle/meter}\) and an average traffic flow \(0.3646\ \mathrm{vehicle/sec}\). We also note that the vehicles controlled by the default Carla Traffic Manager have no interpretation of other drivers' intentions and, therefore, are unlikely to yield to the ego vehicle. This introduces more difficulties in merging into a dense vehicle platoon (such as the example in Fig. 16) even for human drivers.

Therefore, we introduce another variable, i.e., the probability of yielding, to have more diversified evaluations for our method. Specifically, for traffic with zero probability of yielding, we control all vehicles in the traffic using the default Carla Traffic Manager. For that being set to a non-zero value of \(X\%\), the vehicles in the traffic ROI have \(X\%\) probability

Fig. 12: Another forced merging evaluation example on the High-D dataset: The interacting vehicle 2 changes its lane in order to promote the merging action of the on-ramp vehicle. Our ego vehicle (in red) merges into the highway once observing the lane change behavior of vehicle 2. The ego vehicle successfully merges into the highway in \(5.72\ \mathrm{sec}\) which is faster than the actual human driver (in green).

Fig. 13: Forced merging test statistics in High-D: The results are reported separately for the three recordings, and our method achieves a 100% success rate. (Upper) We visualize the time-to-merge of the virtual ego vehicles (red bars) in comparison with the ones of the actual High-D vehicles (green bars). (Lower) Three histograms visualize the number of cases in which our method takes certain seconds to merge the virtual ego vehicle.
of yielding to the on-ramp ego vehicle. The yielding behavior is generated by the Cooperate Forced Controller [51]. Similar to the Intelligent Driver Model (IDM) [52] that controls a vehicle to follow the leading vehicle ahead, the Cooperate Forced Controller is a car-following controller derived from the IDM, but assuming the on-ramp ego vehicle is the leading vehicle.

We test our algorithm at various traffic conditions with different settings. As shown in Fig. 14, we achieve a 100 % success rate among 200 test cases where we have different traffic densities and different probabilities of yielding for interacting vehicles. In the majority of the test cases, the ego vehicle takes less than \(9\) sec to merge. Generally, in denser traffic, our algorithm needs more time to interact with more vehicles and, thus, more time to merge. With a higher probability of yielding, the algorithm takes less time to complete the merging tasks on average. Notably, our algorithm can achieve a 100% success rate even in traffic with zero probability of yielding.

## VII Conclusions

In this paper, we proposed an interaction-aware decision-making module for autonomous vehicle motion planning and control. In our approach, interacting drivers' intentions are modeled as hidden parameters in the proposed behavioral model and are estimated using a Bayesian filter. Subsequently, interacting vehicles' behaviors are predicted using the proposed behavioral model. For the online implementation, the Social-Attention Neural Network (SANN) was designed and utilized to imitate the behavioral model in predicting the interacting drivers' behavior. The proposed approach is easily transferable to different traffic conditions. In addition, the decision tree search algorithm was proposed for faster online computation; this algorithm leads to linear scalability with respect to the number of interacting drivers and prediction horizons. Finally, a series of studies, based on naturalistic traffic data and simulations, were performed to demonstrate the capability of the proposed decision-making module. In particular, we have shown that the behavioral model has good prediction accuracy, and the proposed algorithm is successful in merging the ego vehicle in various simulations and real

Fig. 14: Forced merging test statistics in the Carla simulator with different traffic conditions: The \(x\)-axis shows four different traffic settings where vehicles have different probabilities of being willing to yield to the ego vehicle. We have 50 test cases for each of the four traffic settings. Each dot is a single test case where the color and the size of the dot reflect the time needed to merge the ego into the highway. The \(y\) and \(z\) axes report the average density and flow of the traffic in the ROI.

Fig. 15: A forced merging example in the Carla simulator: the ego vehicle is the black vehicle in the middle lower side of each subplot with its trajectories in green lines and reference trajectories (from our decision-making module) in red lines.
world traffic scenarios, without the need for returning the model hyperparameters or retraining the SANN.
