Hybrid Imitation-Learning Motion Planner for Urban Driving
Cristian Gariboldi Matteo Corno Beng Jin


[
    [
    Accepted Sep 3 2024 to ApJ Letters

=======================================



empty

empty






§ ABSTRACT 

With the release of open source datasets such as nuPlan and Argoverse, the research around learning-based planners has spread a lot in the last years. Existing systems have shown excellent capabilities in imitating the human driver behaviour, but they struggle to guarantee safe closed-loop driving. Conversely, optimization-based planners offer greater security in short-term planning scenarios. To confront this challenge, in this paper we propose a novel hybrid motion planner that integrates both learning-based and optimization-based techniques. Initially, a multilayer perceptron (MLP) generates a human-like trajectory, which is then refined by an optimization-based component. This component not only minimizes tracking errors but also computes a trajectory that is both kinematically feasible and collision-free with obstacles and road boundaries. Our model effectively balances safety and human-likeness, mitigating the trade-off inherent in these objectives. We validate our approach through simulation experiments and further demonstrate its efficacy by deploying it in real-world self-driving vehicles.







§ INTRODUCTION


Autonomous cars are expected to play a crucial role in future mobility due to their potential for increased safety and road utilization. To ensure these benefits, their planning components must provide safe, comfortable, and collision-free trajectories that account for both static and dynamic traffic elements. Traditional trajectory planning approaches include rule-based, sample-based, and optimization-based methods, which rely on manually defined costs and objective functions optimized using classical techniques like A*, RRT, dynamic programming, and Model Predictive Trajectory algorithms. These methods are reliable and interpretable but struggle to scale in complex urban scenarios and do not improve with data, requiring extensive engineering effort for tuning.

The availability of open-source datasets such as nuPlan and Argoverse has advanced research in learning-based planners, which are very good at generating human-like trajectories. However, these models trained in open-loop settings do not guarantee safety in closed-loop applications, especially in novel scenarios, due to their dependence on training data. To address these limitations, perturbations can be introduced into training datasets to help vehicles recover from dangerous situations and mitigate covariate shift problems. Alternatively, a differentiable simulator can be used for closed-loop training. Despite these improvements, learning-based models still struggle to generalize well in unseen domains, making them unsafe for real-world traffic.

The paper proposes two key contributions:

1) Integration of learning-based and optimization-based techniques to create a hybrid imitation-learning model. This combination aims to generate safe, human-like trajectories, balancing the trade-offs between these objectives. This approach is the first of its kind.

2) Validation of the hybrid model on a real vehicle in urban environments, demonstrating its practical effectiveness and robustness beyond simulation.

Most research in this field is confined to simulations, which may not translate to real-world performance. The goal is to improve the short-term planning capabilities of learning-based models, ensuring their safety and reliability in real urban settings. The research focuses on planning, assuming that localization, perception, mapping, and control modules are already in place.



§ RELATED WORK


Generating a comfortable, feasible and collision-free trajectory is a complex task for autonomous driving that has attracted considerable academic interest with several approaches proposed.



 §.§ Optimization-based planners
 
Rule-based and sample-based approaches have been valuable for global and local trajectory planning [1, 2]. However, their complexity makes them unsuitable for real-world autonomous driving in complex scenarios. Consequently, optimization-based planners [3-9] have been proposed, which find optimal trajectories by minimizing predefined cost functions and apply the best control actions for tracking.

Despite their advantages, optimization-based planners face significant challenges:

1) They often struggle to find the global optimum in complex scenarios, as real-time solutions to these optimization problems are difficult, frequently resulting in convergence to local minima in non-convex problems.

2) Even when these planners generate safe, collision-free trajectories, the paths differ significantly from those a human would choose. This discrepancy can confuse and destabilize other agents around the self-driving vehicle, who are not used to predicting the behavior of autonomous cars, potentially leading to unsafe situations.

To address these issues, researchers have turned to machine learning approaches, which have shown promise due to recent advancements in the field.



 §.§ Reinforcement Learning

Reinforcement Learning for autonomous driving [10-13] removes some human engineering complexity since it uses Machine Learning techniques to learn an optimal policy by maximizing a reward (cost) function, by exploring and exploiting the environment. Even though it is possible to obtain good performance in simulation or
in laboratory's experiments, its performance does not easily translate to real world and complex scenarios and cannot guarantee safety in every driving conditions. This may happen because of its difficulty
to converge to stability.



 §.§ Imitation-Learning

Imitation-learning models learn driving policies from expert demonstrations, mapping states to actions. Recently, large datasets of human driving behavior have been released by companies and open-source projects such as Argoverse <cit.>, Lyft <cit.>, Waymo <cit.>, and nuPlan <cit.>, enhancing the development of imitation-learning in autonomous driving. This approach has led to state-of-the-art solutions for motion forecasting <cit.> and robust path planning capabilities.

Our work focuses on leveraging imitation-learning for motion planning by analyzing several methods in this category. ChaufferNet <cit.> uses a convolutional neural network to encode a top-down representation of the environment, training it to imitate human driving. The Urban Driver model <cit.> optimizes trajectories using a policy gradient method and a differentiable simulator for closed-loop training. In contrast, the Neural Motion Planner system <cit.> uses sensor and HD map data to generate 3D detections, future trajectories, and a cost volume, selecting the trajectory with the minimum learned cost.

A multimodal prediction strategy combines a transformer with a Mixture of Experts approach <cit.> to model probability distributions over multiple future trajectories, selecting the one minimizing a predefined cost function. Hybrid models, like SafetyNet <cit.>, integrate a machine learning planner with a rule-based fallback layer to ensure trajectory feasibility and safety, executing either the ML or fallback trajectory based on dynamic feasibility checks.

Another hybrid model, PDM-Hybrid <cit.>, uses trajectory fusion between sample-based and learning-based planners to achieve high scores in the nuPlan simulator. However, this model presents several issues:

1) The fusion of the two trajectories involves linear interpolation based on a correction horizon, denoted as C. Up to C, the trajectory is guided by the sample-based approach, transitioning to the learning-based trajectory beyond C. However, this method may introduce discontinuities in the final trajectory due to inconsistencies at the fusion point C;

2) While this strategy aims to produce a prediction trajectory resembling human behavior (after C), the actual path taken by the ego-vehicle aligns with the output of the sample-based approach. Consequently, the final trajectory may lack human-like characteristics, deviating from expected human behavior.






§ SYSTEM ARCHITECTURE

This section describes our hybrid imitation-learning model, combining a learning-based planner with an optimization-based component for kinematically feasible, collision-free trajectories. As outlined in Fig. 1, the system inputs the ego vehicle states, perception observations, and a goal destination to generate a sample-based trajectory with the Planner block. A Multilayer Perceptron (MLP) refines this trajectory to mimic human-like behavior, and the Model Predictive Trajectory (MPT) block optimizes it to avoid collisions with obstacles and road boundaries.






 §.§ Planner

The planner block together with the Multilayer Perceptron, was inspired by PDM-Open model [24, 25], which,
taking as inputs the poses, velocities and accelerations of the ego vehicle, the observations (used for agents forecasting) and the goal, it is responsible to find a centerline from the starting position to the end point, leveraging on the Dijkstra algorithm <cit.>, and to compute a collision-free path,
relying on a sample-based approach.

The planner computes 15 different paths in the following way:

1) Starting from the centerline, it employs 5 different Intelligent Driver Model (IDM) <cit.> policies with specific target speeds, specifically 20%, 40%, 60%, 80% and 100% of the speed limit. When there is a leading vehicle in front of the ego, the speed limit is defined as the velocity of the leading vehicle;

2) Secondly, in order to have lateral variance, we also apply 3 different offsets from the centerline, respectively +1m, -1m and 0m.


This way, we have 15 different paths with longitudinal and lateral variety which are simulated in the forecasted environment and scored according to the closed-loop metrics provided by nuPlan.
The path with the highest score is then selected and if it has an expected at-fault collision within 2 seconds, the output is overwritten with a maximum braking force maneuver.



 §.§ Multilayer Perceptron (MLP)

The multilayer perceptron is responsible for generating an
output trajectory which should be as similar as possible to the expert
driver one. In order to achieve this task, the neural network takes as inputs the ego vehicle's poses, velocities and accelerations of
longitudinal, lateral and angular axis, starting from the past 2 seconds up to the current
time step, together with the path computed by the planner block. These inputs are scaled to a 512-dimensional vector using a linear layer and then they are concatenated and fed into the MLP.

The MLP consists of two 512-dimensional linear layers with dropout (p=0.1) and ReLU activation functions. The output layer is a linear layer that regresses the future waypoints for the next 8 seconds. This output is called "Neural Network Trajectory," trained to minimize the L2 distance between its waypoints and those of the expert driver trajectory provided by the dataset, which offers more than 88-thousands scenarios with a length of 15 seconds with human driver trajectories for training purposes.



 §.§ Model Predictive Trajectory (MPT)

The optimization-based component utilizes a MPT algorithm. This algorithm integrates inputs such as the "Neural Network Trajectory" generated by the MLP, the drivable area, the ego vehicle's poses and velocities and the observations from the perception system. Its primary function is to produce an optimized trajectory that ensures both collision-free navigation and adherence to kinematic feasibility.

Aiming to solve an optimization problem, we define the following soft and hard constraints:

1) Soft Constraint: the collision-free condition is considered as a soft constraint, since if the optimized trajectory is not collision-free, we take into consideration the previously generated trajectory;

2) Hard Constraint: since the trajectory near the ego vehicle must be smooth, the only hard constraint we have is that the trajectory points near the ego must be the same as the previously generated trajectory, in order to avoid sudden steering maneuvers. This hard constraint is formulated as follow:
    
    δ_k = δ_k^prev  if   (0 ≤ i ≤ N_fix)


    Where:
    
        
  * δ_k represents the steering angle at a current trajectory point;
        
  * δ_k^prev  represents the steering angle at the previous trajectory point. It ensures that the current steering angle remains consistent with the previous one;
        
  * N_fix represents the number of fixed trajectory points. It determines the range over which the hard constraint is applied.
    


The objective function of the optimization problem minimizes the tracking errors and the steering acceleration, rate and angle of the ego vehicle. 

It can be defined as follow:










    J    =  w_y ∑_k y_k^2 + w_θ∑_kθ_k^2 + w_δ∑_kδ_k^2 
        + w_δ̇∑_kδ̇_̇k̇^̇2̇ + w_δ̈∑_kδ̈_̈k̈^̈2̈




Where at time step k, we can define the following variables:

    
  * y_k: lateral distance to reference path;
    
  * θ_k: heading angle against the reference path;
    
  * δ_k: steering angle;
    
  * δ̇_̇k̇: steering rate;
    
  * δ̈_̈k̈: steering acceleration.
    
  * w_y, w_θ, w_δ, w_δ̇, w_δ̈ are tuning weights.


The MPT, by taking as input the observations of other agents, is also able to perform adaptive cruise planning maneuvers. The role of the cruise planning is keeping a safe distance with dynamic vehicle objects with smoothed velocity transition.

The safe distance is calculated dynamically by the following equation:


    d = v_ego t_idling + 1/2 a_ego t_idling^2 + v_ego^2/2a_ego - v_obstacle^2/2a_obstacle


where:


    
  * d is the calculated safe distance;
    
  * t_idling is the idling time for the ego to detect the front vehicle's deceleration;
    
  * v_ego is the ego's current velocity;
    
  * v_obstacle is the front obstacle's current velocity;
    
  * a_ego is the ego's acceleration;
    
  * a_obstacle is the obstacle's acceleration.


To maintain a safe distance while optimizing for smooth velocity transitions, we solve an optimization problem. The objective function minimizes the deviation from the desired velocity and smoothness of acceleration:


    J =  ∑_k (w_v(v_desired - v_ego, k)^2 + w_a a_ego, k^2)


subject to constraints on safe distance d, velocity, and acceleration. By solving this problem at each time step, the ego vehicle adapts to changes and ensures safe and efficient cruising. (Note that w_v, w_a are tuning weights).
















    






§ EXPERIMENTS AND RESULTS



 §.§ Baselines

We first analyze the results from the nuPlan open-loop (OL), closed-loop non-reactive (CL-NR), and closed-loop reactive (CL-R) simulations for baseline models, as shown in TABLE 1. Scores are computed using the simulator's built-in metrics. Open-loop simulations evaluate the planner's imitation of an expert driver's route, while closed-loop simulations assess the trajectory's safety, comfort, and collision avoidance. Each simulation assigns a score between 0 and 100 based on these criteria.




Upon closer examination of TABLE 1, a discernible pattern emerges within the results.

Specifically, Urban Driver, PDM-Open and GC-PGP <cit.>, characterized as learning-based models, exhibit
commendable performance in open-loop simulations but display diminished efficacy in
closed-loop scenarios.

Conversely, the rule-based IDM and the sample-based PDM-Closed models demonstrate an inverse behaviour: underperforming in
open-loop simulations yet surpassing the learning-based models in closed-loop simulations.

These findings suggest that learning-based models excel in predicting the motion of the ego
vehicle, capable of replicating human trajectories. However, unlike rule, sample or optimization-based
approaches, they do not inherently ensure safe closed-loop driving.



 §.§ ROS Simulator

Before testing the model directly on the real vehicle, several experiments have been conducted in the simulator.

Fig. 2 shows different experimental results.



The green line is the "Neural Network Trajectory", direct output of the neural network. As expected it is not able to provide a safe closed-loop driving, as in the provided corner cases of Fig. 2 it often overcomes the boundaries of the lane, leading to unsafe and dangerous situations without the guarantee of a collision-free trajectory. Despite that, it shows good generalization capabilities, as the maps and scenarios considered during evaluation are completely different from the ones in the training stage. 

However, the pink line, which represents the "MPT Trajectory", perfectly drives the vehicle within the bounds of the lane, redefining the multilayer perceptron's output into a safe and collision-free route.

The model is also able to perform collision avoidance maneuvers with static obstacles and adaptive cruise control driving with dynamic agents.

Thanks to these experimental results, it is possible to demonstrate the effectiveness of the safe closed-loop driving capabilities of the hybrid motion planner, which is indeed able to prevent collisions and unfeasible trajectories by computing a refined output through the optimization process.

However, assessing the model's ability to mimic human-like driving style requires a qualitative analysis.

To this aim, we examine several qualitative results.

The following results in Fig. 3-8 show some comparisons between a default optimization-based planner (on the left), and the hybrid motion planner that we propose in this paper (on the right).

In addition to the shape of the trajectories, also the velocity (top) and acceleration (bottom) profiles are provided in order to better evaluate the human-likeness.



In Fig. 3, the trajectories of both the default and hybrid planners exhibit a striking similarity in shape. However, upon closer inspection, we notice an interesting distinction: the hybrid model's trajectory gracefully widens around curves, diverging from the lane centerline, mirroring human driver behavior more closely.



Furthermore, the velocity and acceleration profiles of the hybrid planner are considerably smoother compared to the optimization-based model. In the latter, abrupt maneuvers and accelerations are evident, resulting in a discontinuous overall motion.






















In Fig. 4, although the velocity and acceleration profiles looks very similar among the two planners, we can distinguish a notable difference in the shape of the trajectories. While the default planner almost perfectly follows the lane centerline, leading to a geometrical path, the hybrid model moves away from the centerline, driving along the two turns with a single maneuver.



Similarly, in Fig. 5 we obtain a human-like trajectory with the hybrid model, which widens around curves. Moreover, on the top of the image, we can notice an interesting behaviour in the shape of the trajectories. In that spot, we can see there is an abrupt step in the right bound of the lane, which also affects the default planner trajectory. Conversely, the hybrid planner completely ignore the step in the lane bound and does not affect the motion, leading to a more comfortable route.



Another interesting case is shown in Fig. 6, where an adaptive cruise control maneuver was simulated.

While the default planner abruptly accelerates at the beginning, reaching high velocities in a short time, and suddenly brakes when encountering the leading vehicle, leading to an uncomfortable and discontinuous motion, the hybrid planner employs a much smoother trajectory, inferring the right acceleration to avoid brusque and curt maneuvers.




As the experiment was conducted along a straight line, with no discernible differences between the two trajectories, we shift our focus to analyzing velocity and acceleration in the time domain in Fig. 7, where we can notice much smoother profiles in the hybrid model's motion.







In addition to our qualitative analysis, we shift our focus to quantitative results by inspecting the jerk profile in the time domain. Elevated jerk levels are characteristic of robotic maneuvers, whereas moderated levels reflect a more human-like driving style.


In Fig. 8, we can notice a high peak in the jerk profile of the optimization-based planner, while the hybrid motion planner's one remains contained.





 §.§ Real World Driving

After having conducted several successful tests in the simulator environment, we now shift our case study into real-world scenarios. The model has been deployed on the vehicle built and designed by the company Pix Moving, which is called Robobus. The Robobus is a bi-directional, level 4 autonomous vehicle, fully electric, with sensors such as lidars, radars, cameras, GNSS and IMU. It has been designed to transport up to six people, with a maximum speed of 30km/h and it is already operating in some areas in China and Japan. The experiments took place in a real traffic scenarios, with other static and dynamic agents involved, as shown in Fig. 9. The planner showed stable and robust performance while navigating in the traffic, especially at low speed (less than 15km/h). Thanks to the optimization-based component which refines the output of the neural network, the final trajectory was always within the lane boundaries and collision-free with obstacles and other agents.


















§ CONCLUSIONS


In our paper, we introduce a hybrid imitation-learning motion planner designed to ensure safe, collision-free trajectories that closely mimic human-like behavior. Our model exhibits impressive performance in simulation, demonstrating strong generalization across diverse maps, scenarios, and environments not seen during training. This underscores its robust capabilities. Moreover, our approach proves effective when deployed in real-world self-driving vehicles, particularly at low speeds. As we move forward, future research efforts should prioritize testing the model at higher speeds to better prepare it for real-world urban driving scenarios.







-12cm   
                                  
                                  
                                  
                                  
                                  


















99

c1 Steven LaValle, Planning Algorithms, Cambridge University Press, 2006.
c2 Wilko Schwarting, Javier Alonso, Daniela Rus, Planning and Decision-Making for Autonomous Vehicles, Annual Review of Control, Robotics, and Autonomous Systems, 2018.
c3 Yanjun Huang, Hong Wang, Amir Khajepour, A Novel Local Motion Planning Framework for Autonomous Vehicles Based on Resistance Network and Model Predictive Control, IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, 2020.
c4 Yadollah Rasekhipour, Amir Khajepour, Shih-Ken Chen, A Potential-Field-based Model Predictive Path Planning Controller for Autonomous Road Vehicles, IEEE Transactions on Intelligent Transportation Systems, 2017.
c5 Moritz Werling, Sören Kammel, Julius Ziegler, Optimal trajectories for time-critical street scenarios using discretized terminal manifolds, The International Journal of Robotics Research, 2011.
c6 Jie Ji, Amir Khajepour, William Melek, Path Planning and Tracking for Vehicle Collision Avoidance based on Model Predictive Control with Multi-constraints, IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, 2016.
c7 Chang Liu, Seungho Lee, Scott Varnhagen, Path Planning for Autonomous Vehicles using Model Predictive Control, IEEE Intelligent Vehicles Symposium, 2017.
c8 Alberto Franco, Vitor Santos, Short-term Path Planning with Multiple Moving Obstacle Avoidance based on Adaptive MPC, IEEE Xplore, 2019.
c9 Chaoyong Zhang, Duanfeng Chu, Shidong Liu, Trajectory Planning and Tracking for Autonomous Vehicle Based on State Lattice and Model Predictive Control, IEEE Intelligent transportation systems magazine, 2019.
c10 Changxi You, Jianbo Lu, Dimitar Filev, Advanced Planning for Autonomous Vehicles Using Reinforcement Learning and Deep Inverse Reinforcement Learning, Elsevier, 2018.
c11 Parth Kothari, Christian Perone, Luca Bergamini, DriverGym: Democratising Reinforcement Learning for Autonomous Driving, Machine Learning for Autonomous Driving Workshop, 2021.
c12 Tung Phan, Forbes Howington, Sheng Chu, Driving in Real Life with Inverse Reinforcement
Learning, arXiv, 2022.
c13 Alex Kendall, Jeffrey Hawke, David Janz, Learning to Drive in a Day, arXiv, 2018.
c14 Wenyuan Zeng, Wenjie Luo, Simon Suo, End-to-end Interpretable Neural Motion Planner, Computer Vision Foundation, 2019.
c15 Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Argoverse: 3D Tracking and Forecasting with Rich Maps, Computer Vision and Pattern Recognition, 2019.
c16 John Houston, Guido Zuidhof, Luca Bergamini, One Thousand and One Hours: Self-driving Motion Prediction Dataset, Computer Vision and Pattern Recognition, 2020.
c17 Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Scalability in Perception for Autonomous Driving: Waymo Open Dataset, Computer Vision and Pattern Recognition, 2019.
c18 Holger Caesar, Juraj Kabzan, Kok Seang Tan, nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles, arXiv, 2022.
c19 Jie Cheng, Xiaodong Mei, Ming Liu, Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoders, arXiv, 2023.
c20 Mayank Bansal, Alex Krizhevsky, Abhijit Ogale, ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst, arXiv, 2018.
c21 Oliver Scheel, Luca Bergamini, Maciej Wołczyk, Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients, 5th Conference on Robot Learning, 2021.
c22 Stefano Pini, Christian S. Perone, Aayush Ahuja, Safe Real-World Autonomous Driving by Learning
to Predict and Plan with a Mixture of Experts, Machine Learning for Autonomous Driving Workshop, 2022.
c23 Matt Vitelli, Yan Chang, Yawei Ye, SafetyNet: Safe planning for real-world self-driving vehicles using machine-learned policies, arXiv, 2021.
c24 Daniel Dauner, Marcel Hallgarten, Andreas Geiger, Parting with Misconceptions about Learning-based Vehicle Motion Planning, arXiv, 2023.
c25 Daniel Dauner, Marcel Hallgarten, Andreas Geiger, Supplementary Material for Parting with Misconceptions about Learning-based Vehicle Motion Planning, arXiv, 2023.
c26 Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Towards learning-based planning:
The nuPlan benchmark for real-world autonomous driving, arXiv, 2024.
c27 Edsger Wybe Dijkstra, A Note on Two Problems in Connexion with Graphs, Numerische Mathematik, 1959.
c28 M. Treiber, A. Hennecke, and D. Helbing. Congested traffic states in empirical observations
and microscopic simulations. Physical review E, 2000.
c29 Marcel Hallgarten, Martin Stoll, Andreas Zell. From Prediction to Planning With Goal Conditioned Lane Graph Traversals, arXiv, 2023.











