Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Driving
Yuhang Lu Yichen Yao Jiadong Tu Jiangnan Shao Yuexin Ma Xinge Zhu

Latent Watermarking of Audio Generative Models
    
Robin San Roman
Meta, FAIR 

Inria Nancy
Pierre Fernandez
Meta, FAIR
 
Inria Rennes
Antoine Deleforge
Inria Nancy

Yossi Adi
Meta, FAIR 

Hebrew University of Jerusalem
Romain Serizel
Inria Nancy



    Received XXXX; Accepted YYYY
===========================================================================================================================================================================================================




§ ABSTRACT 
   Large Vision-Language Models (LVLMs) have recently garnered significant attention, with many efforts aimed at harnessing their general knowledge to enhance the interpretability and robustness of autonomous driving models. However, LVLMs typically rely on large, general-purpose datasets and lack the specialized expertise required for professional and safe driving. Existing vision-language driving datasets focus primarily on scene understanding and decision-making, without providing explicit guidance on traffic rules and driving skills, which are critical aspects directly related to driving safety. To bridge this gap, we propose IDKB, a large-scale dataset containing over one million data items collected from various countries, including driving handbooks, theory test data, and simulated road test data. Much like the process of obtaining a driver's license, IDKB encompasses nearly all the explicit knowledge needed for driving from theory to practice. In particular, we conducted comprehensive tests on 15 LVLMs using IDKB to assess their reliability in the context of autonomous driving and provided extensive analysis. We also fine-tuned popular models, achieving notable performance improvements, which further validate the significance of our dataset. The
project page can be found at: <https://4dvlab.github.io/project_page/idkb.html>











§ INTRODUCTION


In recent years, Large Vision-Language Models (LVLMs) <cit.> have emerged as powerful tools in AI, showcasing impressive capabilities in areas such as visual dialogue and document understanding. Building on the general knowledge of LVLMs, some approaches <cit.> have leveraged these models to enhance the efficiency, robustness, and interpretability of autonomous vehicles, addressing the intricate challenges of autonomous driving in open world. However, LVLMs are often trained on vast and generic datasets, lacking the specialized expertise required for the driving domain. This gap in domain-specific knowledge can lead to potential inaccuracies when these models are applied to self-driving systems, where precision and reliability are paramount.



To address this issue, many vision-language driving datasets <cit.> towards LVLM fine-tuning have been developed. Most of these datasets simply add textual annotations to traffic images from existing datasets, which limits the complexity and diversity of the scenarios they cover. While a few datasets <cit.> are specifically collected and annotated with more challenging driving scenarios, they still primarily focus on tasks like scene perception and decision-making, rather than providing structured driving knowledge. As a result, models built on these datasets can only implicitly learn driving knowledge through the supervision of driving decisions. However, this approach differs significantly from how humans learn to drive, which involves studying driving instructions, traffic laws, driving rules, driving skills, and methods for handling emergency situations. Consequently, these models often lack a comprehensive understanding of driving knowledge, leading to unstable and unreliable performance in real-world applications.


To this end, we propose Intelligent Driving Knowledge Base (IDKB), the first large-scale vision-language dataset dedicated to professional driving knowledge and experience. Typically, humans learn to drive systematically by studying driving materials, taking theory tests, and practicing on the road. To enable LVLMs to effectively “earn a driver's license” and guarantee their driving safety, we compiled an extensive collection of driving handbooks and test questions from various countries, covering traffic laws, rules, driving techniques, and crisis management skills. In addition to theoretical data, we generated practical data by simulating diverse road scenarios in CARLA <cit.>, including variations in weather, lighting, traffic conditions, and more. This comprehensive effort resulted in a dataset of over 1 million data entries with various formats, spanning 15 countries, 9 languages, and 4 vehicle types. By capturing diverse driving regulations and practices from various regions, our dataset provides a solid foundation for thoroughly evaluating LVLMs and enhancing their ability to acquire safe and efficient driving capabilities.

Based on IDKB, we conducted an extensive evaluation for 15 existing LVLMs to assess their degree of mastery in driving knowledge and skills. As shown in Fig. <ref>, the evaluated LVLMs generally lacked strong driving domain knowledge, underscoring the need for fine-tuning with high-quality, structured, and diverse driving knowledge data for effective application in autonomous driving.

We also fine-tuned several of these models using our dataset, and the experimental results demonstrate that explicit and structured driving knowledge significantly enhances the performance of LVLMs, leading to more effective and accurate outcomes. Our findings highlight the importance of incorporating specialized, domain-specific knowledge into LVLMs to better equip them for the complex and safety-critical task of autonomous driving.

Our key contributions can be summarized as follows:


  * We introduce IDKB, the first large-scale vision-language dataset explicitly containing both driving theory and practical knowledge.

  * We evaluate 15 existing LVLMs on our dataset and provide a comprehensive analysis of their driving abilities.

  * We offer fine-tuned, open-source LVLMs trained on our dataset, which possess enhanced professional driving expertise.



















§ RELATED WORK




 §.§ LVLMs for Autonomous Driving


Recently, research on Large Vision-Language Models (LVLMs) has surged, with multimodal models such as GPT-4V <cit.>, Qwen <cit.>, and LLaVA <cit.> demonstrating strong performance across a wide range of general tasks. Leveraging these generalized capabilities, several approaches have begun to integrate LVLMs with autonomous driving algorithms to enhance self-driving car performance and interpretability. For example, DriveGPT4 <cit.> processes multimodal input data and generates both text responses and vehicle control signals by fine-tuning a LVLM on an instruction-tuning dataset. AgentDriver <cit.> converts driving situations into textual descriptions with human-like intelligence, then uses an LLM to reason and plan. Similarly, DriveVLM <cit.> employs a LVLM to output planning trajectories through a Chain-of-Thought (CoT) reasoning process. 
However, due to being trained on vast amounts of general data, LVLMs often lack the specialized driving knowledge necessary for accuracy and reliability in driving. Therefore, a dataset that covers specific and comprehensive driving knowledge is crucial for both evaluating and enhancing LVLMs for autonomous driving.





 §.§ Vision-Language Driving Datasets


With the rise of Large Vision-Language Models (LVLMs), numerous vision-language datasets for autonomous driving have been developed for better understanding driving scenes.
The pioneering works BDD-X <cit.> and BDD-OIA <cit.> annotated video datasets with textual descriptions and explanations of ego car actions. Many subsequent multimodal datasets have relabeled existing self-driving datasets. Talk2Car <cit.> adds free-form, high-quality natural language commands to the nuScenes dataset. NuScenes-QA <cit.> creates 460,000 question-answer pairs based on 3D object relationships to evaluate models' understanding and reasoning abilities. DriveLM <cit.> constructs perception, prediction, and planning question-answer pairs in a graph structure to simulate human reasoning, thus enhancing end-to-end autonomous driving systems. VLAAD <cit.> utilizes GPT-4 to generate question-answer pairs from BDD-X <cit.>, producing an instruction-following dataset that features complex reasoning, detailed descriptions, and conversation. However, these datasets rely heavily on existing datasets and often lack complex scenarios. To address this, some datasets are collected and annotated from scratch. DRAMA <cit.> identifies critical objects in traffic scenarios and provides corresponding linguistic descriptions of driving risks. DriveVLM <cit.> presents SUP-AD, a scene understanding and planning dataset with annotations on challenging and long-tail scenarios. CODA-LM <cit.> is a large-scale multimodal self-driving dataset focusing on road corner cases.

However, these works lack explicit knowledge of traffic regulations, rules, and driving techniques, limiting LVLMs in their ability to develop a comprehensive and abstract understanding of driving knowledge.

In contrast, our dataset mirrors the human process of acquiring driving knowledge by collecting detailed annotations from driving handbooks and test questions, while also integrating theoretical learning with practical application through simulated road scenarios in CARLA. By explicitly presenting this knowledge, our approach closely aligns with human learning styles, enabling LVLMs to acquire and integrate driving knowledge more efficiently and reliably, ultimately enhancing their performance in driving-related tasks.





§ METHODS



Intelligent Driving Knowledge Base (IDKB) is structured as a driving knowledge resource, mirroring the process individuals follow to acquire expertise when obtaining a driver's license. This process typically involves studying driving handbooks, taking theory tests, and practicing on the road. In this section, we introduce the data construction pipeline<ref> and present the  statistics and characteristics of our dataset.



 §.§ Data Construction





  §.§.§ Driving Handbook Data
 
Driving handbooks are highly structured and comprehensive resources, covering laws, regulations, techniques, safety, and more. They serve as the foundational step in learning to drive. By studying these handbooks, an intelligent system can develop a thorough and well-rounded understanding of the driving domain. We collected 206 documents, including traffic laws and driving handbooks, totalling 23,847 pages from 15 different countries via the Internet. As shown in Fig. <ref>, we collect and organize unordered data from these documents in systematic way. We first employ layout detector and Optical Character Recognition (OCR) technology to extract data blocks and the text within these blocks. Subsequently, we developed an algorithm to cluster and sequence the data blocks in a way that aligns with human readability. Finally, we filter out duplicate and irrelevant data. An example is presented in Fig. <ref>. 






  §.§.§ Driving Test Data

Driving test data represent an alternative format for the knowledge covered in driving handbooks. While the handbooks offer a structured and comprehensive overview, the test questions reorganize this knowledge into multiple-choice and short-answer formats. This approach allows an intelligent system to both reinforce what it has learned and assess its understanding in a more interactive manner. Engaging with these questions ensures that the system has thoroughly internalized the handbook content, making this process an essential part of learning. To construct this part of data, we extensively collected questions from driving tests of 15 different countries. As depicted in Fig. <ref>, the data collection process is similar with that of driving handbook data. We extract relevant information from various driving tests, and then reorganize these metadata into standard question-answer formats. Most entries we collected are multiple-choice questions, with one or more correct answers, while a smaller portion belongs to open QA questions. Two annotated examples are presented in Fig. <ref>. 


Data Augmentation. 
To enhance the diversity of data and expand the scale of the dataset, we employed GPT-4o model to augment Driving Test Data. To ensure the enhanced data is well-structured, we divide the item into the question stem, options, and explanation sections. Each section undergoes incremental enhancement three times using GPT-4o. After completing the enhancements, the sections are combined to form the enhanced question-answer pair. To assure quality and avoid duplication, we require GPT-4o to output three distinct enhanced versions of the data in a single response, following a specific format. Invalid data is identified and removed through format checking and manual screening afterwards. 
In addition, for each data entry containing images, we generate textual descriptions of the images using GPT-4o and incorporate these descriptions into the dataset for further application.

Data Quality Control.  
To ensure high data quality, we implemented a two-step verification process at the end of the collection pipeline. Initially, an automated program filters out obvious low-quality data, such as images with extremely low resolution or very short text. After this automated removal, we conduct a manual review to further refine and ensure the quality of the remaining data.







  §.§.§ Driving Road Data

After studying the driving handbooks and test questions, the intelligent system gains sufficient theoretical knowledge of driving. The next crucial step is to apply and reinforce this knowledge in real-world driving scenarios, which ensures that the system can effectively translate theoretical understanding into practical, real-world competence. However, existing datasets for real-world driving scenarios are often limited in scope, scale, and coverage of traffic signs, making it difficult to thoroughly understand road conditions. To address these limitations, we leverage the CARLA <cit.> simulator to generate a large, high-quality dataset that offers a more comprehensive understanding of traffic regulations at a low cost. 
Our approach begins with constructing custom simulated environments in the CARLA simulator to generate a batch of traffic sign understanding data. We further expand this dataset by extracting scenes containing traffic signs from the Bench2Drive (Jia et al., 2024) dataset, thereby creating additional annotated traffic sign data. As demonstrated in Fig. <ref>, our driving road data collection process involves the following three steps.

Scene Construction. 
In the scene construction stage, we first collect high-resolution traffic signs from different countries and create 3D models for each traffic sign using the CARLA-UE4 editor. Then we select two of CARLA’s large maps and set the traffic signs at appropriate locations within the CARLA simulator.


Camera Data Collection. 
In the camera data collection stage, we generate an ego vehicle equipped with camera sensors to collect image data by driving the vehicle manually in the CARLA simulator. To ensure the authenticity of the simulation data, we randomly generate a large number of vehicles and pedestrians in the CARLA world and control them through the Autopilot mode and the WalkerAIController provided by the CARLA, respectively, to simulate the real road conditions. To ensure the diversity and richness of the data, we set different weather conditions, times of day, and road surface slipperiness to simulate real camera views and driving scenes. While collecting the camera sensor data, we also record the position, bounding box, rotation and other necessary information of each actor in the CARLA world for the next step. In total, we drive the ego vehicle in the CARLA simulator for about 20 hours and obtain approximately 400,000 frames of camera data.

Data Annotation. 
After obtaining the image data and actor information, we automatically pick out the frames containing traffic signs based on the position, distance and direction of the sign relative to the ego vehicle. Then we manually build a dictionary to define the descriptions and explanations for each sign, and  attach the text annotations to each frame in an automated process. Finally, we obtained a total of 112,388 data samples, including both multiple-choice and question-and-answer formats. 

Data Quality Control. 
Images that lack traffic signs or are obscured by adverse weather conditions or occlusions are considered as low-quality data. To maintain high data quality, we manually check and remove any images that do not meet the required standards.





 §.§ Dataset Statistics

In total, IDKB provides 1,016,956 data entries. As shown in Fig. <ref>(a), Driving Test Data constitutes the largest portion of the dataset (84.0%), with CARLA and Driving Handbook Data accounting for 11.1% and 5.0%, respectively. Our dataset spans multiple countries and multiple languages. As presented in Fig. <ref>(b)(c), apart from English-speaking countries, we also include driving knowledge from China, Italy, Germany, and others. In terms of vehicle types, we categorized the data into four classes: Car (standard passenger vehicles including sedan, jeep, ...), Truck (large vehicles including minivan, commercial, LGV, ...), Bus (including minibus, trailer, coaches, ...), and Moto (Motobike, Motocycle). Fig. <ref>(d) presents the distribution of the vehicle types.

To better analyze the knowledge coverage of our dataset, we employed proprietary LVLMs to classify all the questions into four major categories according to their semantics, including Laws & Regulations (22.2%), Road Signs & Signals (38.6%), Driving Techniques (22.0%), and Defensive Driving (17.1%). More data details are provided in Supplementary. 




 §.§ Data Characteristics

As shown in Tab. <ref>, compared with existing vision-language autonomous driving datasets, IDKB possesses four main novel characteristics as follows.



  §.§.§ Diverse Data Type
 
Our dataset contains diverse data types, encompassing both Question and Answer (QA) and Multiple-choice Question (MCQ). Most existing datasets typically focus on a single question format. By including both QA and MCQ formats, IDKB enhances its utility for various applications, such as training models for open-ended and structured queries, thus providing a more comprehensive testing ground for autonomous driving models.




  §.§.§ Diverse Data Source

Our dataset integrates both real-world and synthetic data sources to provide a comprehensive coverage of driving scenarios. We collected driving manuals and test questions from various countries across the internet, which form the basis of our real-world data. To complement this, we enriched our dataset with data from CARLA-simulated road scenes. Relying solely on  real-world road data has its limitations, as it may not cover all possible scenarios encountered in diverse driving environments. By incorporating CARLA simulations, we address this limitation and ensure that our dataset encompasses a wider range of scenarios. This combined approach allows the system to effectively translate theoretical knowledge from driving manuals and test questions into practical operational capabilities.




  §.§.§ Diverse Data Domain

IDKB exhibits exceptional domain diversity, covering 15 different countries, 9 languages, and 4 types of vehicles. This extensive coverage makes the dataset particularly versatile, enabling its application to various regional contexts and linguistic environments. Most existing datasets are limited to a specific country or language, typically focusing on the US and English. In contrast, IDKB's global approach ensures that models trained on it are better equipped to handle international variations in driving conditions, regulations, and vehicle types, facilitating broader applicability in autonomous driving systems worldwide.






  §.§.§ Diverse Knowledge Domain

Our dataset also offers comprehensive coverage of knowledge domains relevant to autonomous driving. It includes detailed information on Traffic Laws and Regulations, Road Signs and Signals, Vehicle Control and Driving Techniques, and Defensive Driving strategies. While other datasets may focus on one or two knowledge areas, IDKB provides a holistic view of the driving environment. This broad knowledge diversity is crucial for developing models that need to understand and navigate complex driving scenarios, making IDKB an invaluable resource for advancing autonomous driving technologies.





§ EXPERIMENTS


In this section, we evaluate 15 Large Vision-Language Models (LVLMs), both open-source and closed-source, using our proposed dataset. We begin by introducing the selected LVLMs and the tasks they are required to perform. Then, we outline the evaluation methods used for each task. Next, we present a quantitative evaluation of each task. 






 §.§ Experiment Setup


Selected LVLMs. We test 15 representative LVLMs that differ in terms of parameters, open-source availability, and their vision encoders (CLIP ViT <cit.>, EVA-CLIP-ViT <cit.>, SAM <cit.>, SigLIP <cit.>) as well as their LLMs (QWen <cit.>, Vicuna <cit.>, Yi <cit.>, DeepSeek <cit.>, InternLM <cit.>, LLaMA <cit.>, ChatGLM <cit.>, FLAN-T5 <cit.>). For a fair comparison, all LVLMs are used to infer questions from our dataset based on the same prompt. Further details on the selected LVLMs are provided in the Supplementary.

Data Split. We selected all of the driving handbook data, along with ninety percent of the driving test data and driving road data, to form the training set. The remaining ten percent of the driving test data and driving road data were used to create the test set. For more detailed statistics on the training and test sets, please refer to the Supplementary.

Tasks. We evaluate the LVLM's performance using two data sources: driving test data and driving road data, each containing multiple-choice questions (MCQ) and question-and-answer (QA) tasks. In the MCQ tasks, the LVLM must select the correct answer(s) from the provided options, while in the QA tasks, it is required to generate the most relevant response to a given question.



 §.§ Evaluation Details


For MCQ tasks, we use regular expressions to extract options from the LVLM outputs and compare them with correct answers, measuring accuracy as the metric. Partially correct answers are not accepted in multi-answer questions. Rule-based extraction is challenging due to the free-form nature of LVLM outputs. To address this, we introduce a instruction-following test where the output must include the string “Option [A to F]” as prompted.

For QA tasks, we use ROUGE <cit.> and SEMScore <cit.> to measure similarity between LVLM outputs and the reference answers. ROUGE evaluates N-gram overlap, while SEMScore assesses semantic similarity using sentence embeddings.

We calculate average scores for both MCQ and QA metrics across data sources, creating Test Data and Road Data Scores. The mean of these two scores is the IDKB Score, reflecting the LVLM's overall mastery of driving knowledge. More details about metrics are provided in Supplementary.




 §.§ Main Results

In this subsection, we analyze various LVLMs on our test set, focusing on overall performance, distinctions between driving test and road data, handling of single versus multiple answers, comparison between proprietary and open-source LVLMs, and adherence to instructions. We also highlight the impact of fine-tuning with our dataset, showcasing key improvements in model performance. More details and analysis will be provided in Supplementary. 



Overall Evaluation 
In Tab. <ref>, we present the performance results of various VLMs on our test set. GPT-4o achieved the highest overall performance with an IDKB score of 0.64. Among the open-source models, XComposer2 stood out with the best performance, achieving an IDKB score of 0.45. Most open-source models fell within an IDKB score range from 0.35 to 0.4. However, BLIP2, XComposer, Yi-VL and VisualGLM underperformed, with IDKB scores of 0.27, 0.28, 0.28 and 0.29, respectively. Overall, the evaluated LVLMs generally did not demonstrate strong driving domain knowledge, highlighting the need for high-quality, structured and diverse driving knowledge data for effective applications in autonomous driving.

Driving Test Data vs. Driving Road Data 
LVLMs generally perform better on driving road data than on driving test data. Most open-source LVLMs scored around 0.25 on Test Data Score, while the average Road Data Score was 0.44. A similar trend was observed in the two proprietary LVLMs. This suggests that while many LVLMs have a basic understanding of traffic signs, they lack a deeper comprehension of traffic laws, regulations, and driving skills—areas more thoroughly assessed in driving test data.

Single Answer vs. Multiple Answer
The performance of LVLMs on multi-answer questions showed clear polarization. About half of the LVLMs outperformed on multi-answer questions compared to single-answer questions. Conversely, the other half performed significantly worse on multi-answer questions than on single-answer ones. Notably, models such as VisualGLM-6B, Qwen-VL, CogVLM, XComposer, and Monkey-chat struggled particularly with multi-answer questions, failing to provide accurate responses. This disparity in performance suggests that the complexity of driving knowledge required for multi-answer questions presents a substantial challenge for certain VLMs, underscoring the variability in their capabilities.

Instruction Follow Ability
Overall, proprietary models excel in adherence to instructions, with accuracy of 0.99, indicating minimal deviation from prescribed answer templates. In contrast, among open-source models, only VisualGLM-6B, XComposer, ShareGPT4V-6B, XComposer2, and Monkey-chat demonstrate a relatively high level of instruction-following capability. The remaining open-source models are prone to producing non-compliant text, with Yi-VL-6B exhibiting the most significant issue—only 11% of its outputs align with the input requirements. Some LVLMs may struggle with consistency and accuracy in adhering to specified instructions, underscoring the need for additional fine-tuning to enhance their ability to comply with instructions in practical applications.

Proprietary vs. Open-Source
Proprietary LVLMs generally outperform open-source LVLMs in evaluation results, likely due to their larger number of parameters and more extensive knowledge base. This advantage is more pronounced with driving test data, which requires external knowledge of laws and rules—areas where proprietary models excel. In contrast, when dealing with driving road data, which emphasizes traffic sign recognition, some open-source LVLMs like XComposer2 can achieve performance levels comparable to those of proprietary models.



Significant Improvement through Fine-Tuning
To better evaluate the impact of structured driving knowledge data on model performance in this domain, we fine-tuned four representative LVLMs, each with a different visual encoder or LLM. As shown in Table <ref>, the fine-tuned LVLMs achieved IDKB scores comparable to, or even matching, those of proprietary models with significantly larger parameters.
The improvement in the Test Data Score is particularly striking, with MiniCPM-Llama3-V2.5's performance doubling in this metric. This suggests that our data equips LVLMs with valuable expertise in driving laws, rules, techniques, and handling special situations. 
Additionally, the model's ability to interpret traffic signs, as indicated by the Test Road Score, showed improvement over the original, suggesting an enhanced understanding of traffic regulations in road scenarios. 
These results underscore the importance of our dataset in enhancing LVLM competence within the driving domain. By equipping models with structured and diverse driving knowledge, our dataset plays a crucial role in strengthening LVLMs' expertise, ultimately contributing to the development of safer and more reliable autonomous driving systems.






§ BENEFITS OF DRIVING KNOWLEDGE FOR DOWNSTREAM AUTONOMOUS DRIVING TASKS


In this section, we showcase the application of the Qwen-VL-chat model, fine-tuned on the IDKB dataset, for the planning task using the nuScenes <cit.> dataset. This demonstrates how integrating driving knowledge data can significantly enhance performance in downstream tasks.



 §.§ Setup

Based on Agent-Driver <cit.>, we generated fine-tuning data for nuScenes trajectory planning and fine-tuned Qwen-VL-chat for this task. To demonstrate the value of IDKB data for autonomous driving, we also fine-tuned another Qwen-VL-chat model using a combination of the nuScenes planning data and our IDKB data.

For inference, as shown in Fig. <ref>, we follow a prompt method similar to DriveVLM <cit.>. The LVLM first identifies the traffic signs on the road and then receives key information to predict the trajectory for the next three seconds.







 §.§ Results and analysis

The nuScenes planning task relies on imitation learning from human-generated planning data, while the IDKB dataset is designed to align with the human model of driving knowledge acquisition. This alignment allows the LVLM fine-tuned with IDKB to gain a deeper understanding of driving knowledge, including traffic laws, regulations, and driving skills, which in turn leads to more rational and safer route planning.

Our experimental results support this hypothesis. As shown in the table, the model fine-tuned with IDKB data demonstrates superior performance, with a 32% reduction in average L2 distance and a 65% decrease in collision metrics, indicating safer and more rational planning.

Additionally, as illustrated in Fig. <ref>, the model fine-tuned with both nuScenes and IDKB data successfully identifies the 'ROAD WORK AHEAD' sign, understands the need to slow down, and drive with caution. The planned trajectory confirms this, with the decreasing offset between consecutive frames reflecting deceleration.




§ CONCLUSIONS


In this paper, we introduced IDKB, a pioneering large-scale dataset designed to bridge the gap in domain-specific driving knowledge within LVLMs. IDKB includes over 1 million entries on driving regulations, scenarios, and practices from 15 countries and 9 languages. 
We evaluated 15 LVLMs using IDKB to assess their performance in the context of autonomous driving and provided extensive analysis. Additionally, we fine-tuned several popular models, achieving significant improvements in their performance, which further highlights the importance of our dataset.









ieeenat_fullname












The supplementary first provides more details of our dataset, covering data collection, augmentation, and key statistics, with sample presentations in Sec. <ref>. We then outline the experimental setup, including datails of selected LVLMs, dataset division, assessment methods, and metric calculations in Sec. <ref>. Next, we analyze the benchmark results in Sec. <ref>, focusing on LVLMs' performance across different data domains and knowledge coverage. Finally, in Sec. <ref>, we show that models fine-tuned with our dataset, in addition to nuScenes <cit.>, produce safer, human-compliant trajectory planning, highlighting the impact of enriched driving knowledge.



§ MORE DETAILS ABOUT DATASETS


In this section, we present detailed information on the data collection process (Sec. <ref>), provide comprehensive statistics on the dataset (Sec. <ref>), and include additional data visualizations (Sec. <ref>).



 §.§ Data Construction





  §.§.§ Driving Handbook Data Collection

Driving handbooks are highly structured and comprehensive resources, covering laws, regulations, techniques, safety, and more. To extract unordered data from these documents, we first utilize layout detection and Optical Character Recognition (OCR) technology to extract data blocks and the text within these blocks. Specially, we use LayoutLMv3 <cit.> as our layout detection and PaddleOCR as our OCR toolkit. Next, we utilize DBSCAN <cit.> to cluster blocks to let each column in a multi-column PDF falls into the same cluster. After that, we sequence the data blocks based on the position and which column it's in. Finally we filter out duplicate and irrelevant data.




  §.§.§ Driving Road Data Collection

In the CARLA, the position and rotation of an actor are represented by carla.Location and carla.Rotation, respectively. The carla.Location format is given as (x, y, z) in a rectangular coordinate system, measured in meters. While CARLA is based on the UE4, the carla.Rotation differs in its definition, using (pitch, yaw, roll) to describe orientation. Specifically, pitch represents rotation around the Y-axis, yaw represents rotation around the Z-axis, and roll represents rotation around the X-axis, all measured in degrees.

During the Camera Data Collection stage, we captured the position and rotation of the ego vehicle and traffic signs. Subsequently, in the Data Annotation stage, images containing traffic signs were identified based on the position, distance, and rotation of the traffic signs relative to the ego vehicle. The distance is calculated as follows:

    distance = √((x_sign-x_ego)^2 + (y_sign-y_ego)^2)

where, x_sign, y_sign represent the location of the traffic sign and x_ego, y_ego represent the location of the ego vehicle.

In CARLA, the yaw ranges from -180^∘ to 180^∘. A yaw of 0^∘ indicates that the actor is facing the positive X-axis. A positive yaw value corresponds to a counterclockwise rotation (left turn), while a negative yaw value indicates a clockwise rotation (right turn).

The locational angle of the traffic sign relative to the ego vehicle is calculated as:

    yaw_loc = (arctan(y_sign - y_ego/x_sign - x_ego) ×180/π) - yaw_ego

where, yaw_ego represent the rotation around the Z-axis of the ego vehicle.

Then the yaw_loc is normalized to the range [-180^∘, 180^∘] by:

    yaw_loc = yaw_loc 360



    yaw_loc = yaw_loc - 360    if yaw_loc > 180 
    yaw_loc   otherwise


The directional angle of the traffic sign relative to the ego vehicle is calculated as follows:

    yaw_rot = yaw_sign - yaw_ego

where, yaw_sign represent the rotation around the Z-axis of the traffic sign.

Then the yaw_rot is normalized to the range [-180^∘, 180^∘] by the same way as above.

After testing, we finally determined that the yaw_loc, distance and yaw_rot are respectively ([-35^∘, 35^∘]), ((0, 30m]), ([-180^∘, -160^∘])⋃([160^∘, 180^∘]), and judged in turns.



  §.§.§ Data Augmentation.

To enhance the diversity of data and expand the scale of the dataset, we employed GPT-4o model to augment data. To ensure the enhanced data is well-structured, we divide the item into the question stem, options, and explanation sections and enhance them separately. After that enhancaed sections are combined to form the enhanced question-answer pair. Prompt we used are shown in Tab. <ref>.










 §.§ Data Statistics


Tab. <ref> provides a summary of detailed statistics for IDKB, including numbers on dataset split, question types, image position, average text length, and other relevant statistics. Note that we calculate average question/option length on the unaugmented data, and special tokenizers are utilized to count words for Chinese, Korean and Japanese text. Tab. <ref> presents detailed numbers on country distribution. The annotation language of each country's data is also listed. 

To obtain a statistical view of the knowledge domains within IDKB, we designed specific prompts and employed proprietary LVLMs to classify all the questions into four major categories based on their semantics. Here we provide detailed definitions of the four kinds of knowledge domains: 

    
  * Traffic Laws and Regulations:
This category focuses exclusively on the statutory and legal aspects of driving. It includes questions about state and local traffic laws, understanding of legal driving ages, alcohol and drug influence laws, and penalties for traffic violations. This category ensures that drivers are aware of legal obligations and the consequences of their actions on the road, which can be a substantial part of a driving test.

  * Road Signs, Signals, and Lane Markings:
This category is dedicated to testing knowledge of road signs (stop, yield, pedestrian crossing, etc.), traffic signals, and different lane markings. This includes interpreting what various signs and markings mean and how to respond to them while driving. The category serves to evaluate a driver’s ability to navigate roads safely and appropriately, understanding that correct responses to signs and signals are crucial for safe driving.

  * Vehicle Control and Driving Techniques:
Here, the focus is on practical driving skills and vehicle management. Questions could include scenarios dealing with vehicle control in adverse weather conditions, the correct use of mirrors, turning and passing maneuvers, and parking techniques. This category tests a driver’s hands-on ability to operate a vehicle safely and maintain control in everyday driving situations as well as in unexpected conditions.

  * Driver Responsibility and Defensive Driving:
This category assesses knowledge related to driver’s behavior and responsibilities. It includes defensive driving techniques, the importance of seat belts, methods to prevent distracted driving, and how to act responsibly in various driving situations (school zones, neighborhoods). Questions about accident procedures, emergency responses, and first aid may also be included. It underscores the importance of personal responsibility and proactive safety measures in driving.




 §.§ More Data Examples


To provide a more intuitive overview of our IDKB dataset, we present additional data examples in this section, as illustrated in Fig.<ref>, <ref> and <ref>, for Driving Handbook Data, Driving Test Data, and Driving Road Data respectively.





§ MORE DETAILS ABOUT EXPERIMENT SETUP


In this section, we provide additional information on the selected LVLMs in Sec. <ref>, outline the evaluation methods for multiple-choice and question-and-answer questions in Sec. <ref>, and explain the computation of the three overall metrics in Sec. <ref>. Then in Sec. <ref> we introduce the prompts we used in the paper.




 §.§ LVLMs Model Details


Tab. <ref> provides details of the LVLMs used in this paper, including their parameter sizes, visual encoders, and LLMs. The selected open-source LVLMs have parameter sizes ranging from 6.6B to 12.1B, featuring 4 different vision encoders and 11 distinct LLMs. Finetuning and inference times also vary across models. For instance, finetuning Qwen-VL-chat with LoRA <cit.> on our dataset, using 8 NVIDIA A40 GPUs with a maximum token length of 2048, took 66 hours and required approximately 24GB of memory.






 §.§ Evaluation Details


Multiple-Choice Questions
For multiple-choice questions, we use regular expressions to extract answers according to the logic outlined in Algorithm <ref>. For instruction-following, we simply check whether the LVLM's output contains the pattern “Option + letter”.


Question-and-Answer
For question-and-answer, we employ ROUGE <cit.> and SEMScore <cit.> to measure similarity between LVLM outputs and the reference answers. ROUGE evaluates N-gram overlap between outputs and the reference answers. For each question we consider enhanced answers as reference too to duel with free-form nature of LVLM outputs, especially for ROUGE. We compute ROUGE-1 and ROUGE-L between the output and each reference answer, and then select the maximum score for both ROUGE-1 and ROUGE-L.

For SEMScore, we first utilize sentence transformer <cit.> to embed LVLM outputs and the reference answers. Specifically we use “paraphrase-multilingual-mpnet-base-v2” as our model. Then we measured cosine distance between embeds which represents the difference between output and the reference answers in semantic level. Similar to Rouge, we take the maximum SEMScore as the SEMScore for the output.



 §.§ Overall Metrics


The overall score for MCQs is calculated as the weighted accuracy of both single-choice and multiple-choice questions, with the quantity of each question type used as the weighting factor. The value of instruction-following is excluded from this calculation and is used only as a reference.

For QAs, the overall score is a weighted combination of Rouge-1, Rouge-L, and SEMScore, with weights of 0.15, 0.15, and 0.70, respectively. Given the free-form nature of LVLM outputs, semantically similar responses may receive low ROUGE scores due to limited N-gram overlap. To address this, we assign a higher weight to SEMScore to better capture human-like evaluation.

The Test Data Score and Road Data Score are calculated as the averages of the overall MCQ and QA scores from the driving test data and driving road data, respectively. The IDKB score is then derived as the average of the Test Data Score and Road Data Score.



 §.§ Prompt


We include the country, vehicle type, and question type in our prompt to provide background knowledge to the LVLMs. Additionally, The prompt also includes a detailed description of the required output format to ensure the responses meet the specified standards. Prompt we used to evaluate LVLMs are shown in Tab. <ref>.



§ FURTHER ANALYSIS OF LVLM RESULTS


In this section, we analyze the entire test set, including both the Driving Test Data and Driving Road Data. We examine the performance of all evaluated models, focusing on overall performance (Sec. <ref>), performance across different countries (Sec. <ref>), languages (Sec. <ref>), and vehicle types (Sec. <ref>), as well as performance across various knowledge domains (Sec. <ref>).



 §.§ Evaluate LVLMs Performance on test set


As shown in Fig. <ref> and Fig. <ref>, for MCQ questions, the proprietary model significantly outperformed the open-source models, with the accuracy gap between the best and worst open-source performers (CogVLM vs. VisualGLM) being nearly twofold. However, for QA questions, the gap is less pronounced, with many open-source models performing comparably to proprietary models.




 §.§ Evaluate LVLMs Performance by Country


We provide the performance of different models on questions from different countries in the Fig. <ref> and Fig. <ref>. MCQ questions involve 14 countries, while QA questions cover 3 countries.
For QA questions, the performance gap between models is consistent across the three countries. However, for questions from India, all models scored lower compared to the other two countries.
In the case of MCQ questions, we observed significant variations in LVLM performance across countries. LLaVA-v1.5-7B, XComposer, and ShareGPT4V performed exceptionally well on Japanese questions, significantly outperforming other models. LLaVA-v1.5-7B and ShareGPT4V also delivered strong results on Italian questions. For UK questions, GPT-4 achieved an accuracy exceeding 80%, which is particularly impressive. However, on questions from Japan, Korea, and Spain, some LVLMs had an accuracy of 0. This disparity highlights the varying capabilities of different LVLMs across countries and underscores the value of our dataset's diversity in providing a comprehensive evaluation



 §.§ Evaluate LVLMs Performance by Language


Fig. <ref> illustrate the performance of various models on questions across different languages. Since the QA questions in the test set are exclusively in English, this analysis focuses solely on the MCQ questions. The performance disparity between models is more pronounced for minority languages compared to English. LLaVA-v1.5-7B, XComposer, and ShareGPT4V performed exceptionally well on Japanese questions, whereas GPT-4 faced some challenges in this area. CogVLM and BLIP2 showed strong performance across several minority languages, including Spanish, German, French, Korean, and Italian. In contrast, MiniCPM-LLaMA-V2.5 struggled with minority languages, achieving a correct rate of 0 on Spanish and Korean questions, although it performed well on Traditional Chinese. These results demonstrate that models exhibit different strengths depending on the language and region. While some models excel in specific languages or regions, they may underperform in others. This variation underscores the importance of evaluating models across a diverse set of languages and regions to fully understand their capabilities and limitations.



 §.§ Evaluate LVLMs Performance by Vehicle Type


Since all QA questions in the test set are of the “car” type, our analysis concentrates on the MCQ questions. Fig. <ref> illustrates the performance of various models across different vehicle types. Among the open-source models, BLIP2 demonstrates the strongest performance in the motor, truck, and bus categories, while LLaVA-v1.5-7B excels in the car category. Notably, Qwen-VL-chat shows consistent but average performance across motor, truck, and car categories. Interestingly, the models tend to perform better on questions related to motor, truck, and bus than on car-related questions, suggesting that the car category poses unique challenges that may require further exploration.



 §.§ Evaluate LVLMs Performance by Knowledge Coverage


Fig. <ref> and Fig. <ref> present the performance of various models across different knowledge domains. For MCQ questions, proprietary models show higher accuracy in Driving Techniques and Defensive Driving compared to the other categories. In contrast, most open-source models underperform in these areas, indicating that proprietary models may possess better common-sense knowledge.
For QA questions, the Signs & Signals category achieves a significantly higher average score than the other categories, although Yi-VL performs poorly in this area. Scores for Driving Techniques and Defensive Driving also surpass those for Laws & Regulations, highlighting that models tend to perform better in more application-oriented categories than in regulatory ones.




























