OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving
Julong Wei Shanshuai Yuan Pengfei Li Qingda Hu Zhongxue Gan Wenchao Ding


Inverse Design of Winding Tuple for Non-Hermitian Topological Edge Modes
    Yong Hu
    September 9, 2024
========================================================================



§ ABSTRACT 


The rise of multi-modal large language models(MLLMs) has spurred their applications in autonomous driving. Recent MLLM-based methods perform action by learning a direct mapping from perception to action, neglecting the dynamics of the world and the relations between action and world dynamics. In contrast, human beings possess world model that enables them to simulate the future states based on 3D internal visual representation and plan actions accordingly. To this end, we propose OccLLaMA, an occupancy-language-action generative world model, which uses semantic occupancy as a general visual representation and unifies vision-language-action(VLA) modalities through an autoregressive model. Specifically, we introduce a novel VQVAE-like scene tokenizer to efficiently discretize and reconstruct semantic occupancy scenes, considering its sparsity and classes imbalance. Then, we build a unified multi-modal vocabulary for vision, language and action. Furthermore, we enhance LLM, specifically LLaMA, to perform the next token/scene prediction on the unified vocabulary to complete multiple tasks in autonomous driving. Extensive experiments demonstrate that OccLLaMA achieves competitive performance across multiple tasks, including 4D occupancy forecasting, motion planning, and visual question answering, showcasing its potential as a foundation model in autonomous driving.















§ INTRODUCTION



Rencent years, we have witnessed a significant breakthrough in multi-modal large language models(MLLMs) capable of integrating various modalities, such as language, image, audio, which has accelerated the development of embodied artificial intelligence (Embodied AI). Nevertheless, a general agent, which can address multiple tasks in real-world has yet to emerge. This is inherently because existing MLLMs perform action by learning a direct mapping from perception to action, neglecting the dynamics of the world and the relations between action and world dynamics. In contrast, human beings possess world model that enables them to simulate the future states based on 3D internal visual representation and plan actions accordingly. Therefore, exploring how to construct the agent's world model is crucial for the advancement of embodied intelligence.


Autonomous driving, as a representative application of Embodied AI, has witnessed extensive research on world models. However, the precise definition of a world model for autonomous driving remains an open question. Current world models for autonomous driving focus on sensor prediction tasks such as video prediction <cit.>, point cloud prediction <cit.> and occupancy prediction <cit.>. Yet they fail to simultaneously achieve forecasting scene evolution, language reasoning, and interaction with the real world. Thus, we propose that a model capable of unifying the modeling of vision, language, and actions(VLA), akin to human abilities, would be a promising candidate for an autonomous driving world model.

However, two challenges are crucial and need to be solved for building a VLA world model. The first is to build a general 3D visual representation that facilitates both understanding and generation, and the second is to develop a multi-modal framework capable of accommodating VLA modalities. Recent years, semantic occupancy (Occ) has gained significant attention as a general 3D visual representation. It can describe fine-grained 3D structure, while also containing high-level semantic information, making it well-suited for aligning space and semantics. Meanwhile, the feasibility of vision generation based on autoregressive language models has been thoroughly validated, with performance comparable to diffusion models, which are specialists in visual generation. These provide valuable insights for addressing the challenges and constructing a VLA world model based on an autoregressive model with Occ visual representation.



Based on the above observations, we propose OccLLaMA, a unified 3D occupancy-language-action generative world model, which unifies VLA-related tasks including but not limited to scene understanding, planning, and 4D occupancy forecasting, as shown in <Ref>. To enable OccLLaMA with the ability to understand and generate vision modality, we choose Occ as a general visual representation and introduce a novel scene tokenizer to construct discrete scene vocabulary effectively, considering sparsity and classes imbalance. Then, by combining scene vocabulary, language vocabulary, and action vocabulary, we construct a unified multi-modal vocabulary for VLA-tasks, which provides a foundation for integrating VLA in one model. Furthermore, we enhance LLM, specifically LLaMA <cit.>, to implement next token/scene prediction on the unified multi-modal vocabulary, building a VLA world model similar to humans.








We summarize our contributions as follows:




  * An occupancy-language-action generative world model, OccLLaMA, which uses Occ as visual representation and involves multiple tasks through unified multi-modal vocabulary and enhanced autoregressive model based on LLaMA.


  * A novel scene tokenizer that efficiently discretize and reconstruct Occ scenes, considering sparsity and classes imbalance.


  * Extensive experiments compared to SOTA methods, which achieves the competitive performance across multiple tasks, including 4D occupancy forecasting, motion planning, and visual question answering.





















































§ RELATED WORKS




 §.§ MLLMs in Autonomous Driving


The advancement of LLMs explore new paradigms in autonomous driving, including scene understanding <cit.>, end-to-end decision-making <cit.>. 
LLM-driven decision-making methods have shown potential in addressing interpretability and generalization challenges in learning-based systems by making inferences in text space. For real-world autonomous driving, various techniques have emerged to convey environmental information to models, with research advancing to extend input modalities more effectively.

It includes template-based scene description in natural language <cit.>, vector embedding input combined with language prompts <cit.>, camera-perceived image embedding <cit.>,etc. It remains to be validated whether more modalities mean high performance. For example, point clouds do not show the ability to enhance performance in DriveMLM <cit.>. Moreover, the output modality in past work tends to be relatively homogeneous, which somewhat limits the model's accuracy and stability of decision making and closed-loop feedback capability. Thus there is an opportunity for multimodal large language models(MLLM) and world models(WM) to shake hands.





 §.§ World Model in Autonomous Driving


World Models aim to predict future scene based on the action and observations <cit.>.In autonomous driving, world models are often used for data generation and decision making. Various models represents the scene in different spaces, they can be divided into 2D image representation <cit.>, 3D point clouds representation <cit.> and 3D occupancy representation <cit.>. 
Visual world models using 2D image representations offer scalability due to sensor flexibility but lack 3D scene comprehension. While 3D point clouds representations address this problem, they lack semantic information.

Some works <cit.> focus on multi-modal representation, but it is difficult to align the features of results generated in different modality. Therefore, integrating the 3D scene representation and semantic understanding is a promising way to model the scene evolution. Unlike the paradigm of  <cit.>generating the 3D occupancy without semantics meaning attached and scene representation in camera and point cloud form separately. Refer to the paradigm of Occupancy World <cit.>,representation on 3D occupancy space with semantics is adopted in this work.



 §.§ Autoregressive Visual Generation


Autoregressive (AR) Visual Generation refers to models use autoregressive methods to generate images. Early models such as VQVAE <cit.>, VQGAN <cit.> and Dalle <cit.>, convert images into discrete tokens and generate them sequentially faced limitations in output performance, scalability. Then diffusion models <cit.>dominated the field of visual generation due to distinct paradigms. 
Recently, the simplicity of the autoregressive model has enabled uniform understanding and generation, effectively scaling up big data, leading to notable success and increased attention.

VAR <cit.> model enables GPT-based autoregressive models to surpass diffusion models in image generation. Llama-Gen <cit.> outperforms diffusion models in conditional image generation, showing that pure autoregressive models can serve as a foundation for image generation without visual signal inductive bias. Integrating AR language models with vision generation remains challenging, particularly in creating unified models for both language and vision tasks.





§ METHOD



 §.§ Overview
















We propose OccLLaMA, a uniform occupancy-language-action framework. As illustrated in <Ref>, the core components of OccLLaMA include the scene tokenizer (<Ref>) and the occupancy-language-action generative world model (<Ref>). To involve multitask, we introduce a three-stage training scheme (<Ref>) for the training of scene tokenizer, occupancy-language-action pre-training, and instruction tuning.



 §.§ Scene Tokenizer
 

To represent scenes using discrete tokens, a common approach is to employ a VQVAE-like architecture <cit.>. However, approximately 90% of the grids <cit.> in occupancy are filled with air, leading to significant sparsity. 

Existing methods that apply dense convolutional operators to the air category are both  expensive and inefficient. Additionally, the imbalance between categories further hinders learning efficiency. To address these challenges, we introduce a sparse encoding strategy for the encoder, inspired by point cloud processing techniques. Meanwhile, we decouple the non-occupied category from other semantic categories, allowing for more efficient scene reconstruction.




  §.§.§ Encoder
 The original scene is represented as x ∈ℝ^H × W × D, where the 3D space is divided into dense H × W × D voxels, and each voxel is assigned a semantic label l ∈ℝ. We then sparsify x into y ∈ P^H × W by discarding the air voxels and representing the semantic occupancy voxels as a 1D pseudo-point cloud set P={p_i}_i=1^N arranged along the BEV direction, where N is the number of non-air voxels within the current pillar. Each point p_i is a vector (d,l) with height d and semantic label l. We then aggregate the pseudo-point cloud features using a pillars embedding <cit.> and employ a swin-transformer block <cit.> to obtain the BEV feature map z = E(y) ∈ℝ^H/r×W/r× c, where r is downsampling rate, and c is the latent feature dimension.



  §.§.§ Quantification
 To obtain discrete representations, we next transform z into a collection of codebook entries ẑ through vector quantization. The learnable codebook Z = {ẑ_i}^K_i=1 consists of K vectors, each with a dimension of c. The quantization process Q(·) replaces each z_i with its nearest codebook entry ẑ_k in Z, expressed as:

    ẑ_i=Q(z_i):=min_ẑ_k∈ Z z_i - ẑ_k _2



  §.§.§ Decoder
 Due to the loss of height information in the BEV feature map after quantization, the decoder restores dense 3D voxel features by stacking convolution block and up-sampling layer. Specifically, to address classes imbalance, we instantiate lightweight voxel head and class head separately to decode geometric and semantic information of occupancy. Notably, the voxel head provides an occupied mask for the class head, allowing us to supervise the semantics of the occupied voxels only.


  §.§.§ Loss

To train this scene tokenizer, we follow OccWorld <cit.> to utilize three loss functions for optimizing, where cross-entropy loss ℒ_c and lovasz-softmax loss ℒ_l for geometry ge and semantics se reconstruction learning, and the embedding loss ℒ_e for codebook learning.

    ℒ = λ_1 ℒ_c^ge + λ_2ℒ_l^ge +
    λ_3 ℒ_c^se + λ_4ℒ_l^se
    + λ_5ℒ_e









 §.§ Generative World Model





  §.§.§ Unified Vocabulary
 
Employing scene tokenizer in <Ref>, an occupancy scene x can be mapped and flattened into a sequence ẑ^1:L∈ℝ^c, where L = H/r×W/r, allowing for joint representation with similar language vocabulary V_t = { v_t^i}_i=1^K_t in original LLMs. Specifically, we first represent scene tokens ẑ^1:L as a sequence of indices s^1:L = {s^i}_i=1^L, where s^i corresponds to the code index number of scene tokens ẑ^1:L. So we can build a scene vocabulary V_s = { v_s^i}_i=1^K_s, which is order-preserving to our scene codebook Z. Since it is non-trivial to output fine-grained numerical results using general LLMs, we divide the coordinates of waypoints into N bins empirically based on the statistics of the trajectory set and map waypoint to the nearest bin, to build an action vocabulary V_a = { v_a^i}_i=1^K_a. Additionally, we add several special functional tokens { v_f^i}_i=1^K_f, such as , , ,  to denote modality boundaries;  to assist in next scene prediction. Thus, we can build a unified occupancy-language-action vocabulary V = { V_s, V_t, V_a, { v_f^i}_i=1^K_f} to formulate diverse tasks in a generate format, where both input and output can be one of the three modalities or a mixture, depending on the specific task to be solved.



  §.§.§ Next Token / Scene Prediction
 
We observe that both language and action are temporal sequences, making the tokens within these sequences naturally suitable for temporal attention with original causal masks and next token prediction mechanisms. However, the tokens within a scene sequence do not inherently follow a temporal order, and the sequence length tend to greater than language and action. If next token prediction is performed line by line within a scene, it fails to capture the spatial relationships and incurs high computational costs. To address these issues, we introduce a next scene prediction while preserving the next token prediction.

As illustrated in <Ref>, we implement spatial attention at the positions corresponding to scene tokens to better capture the spatial relationships within the scene. Correspondingly, we initialize learnable scene queries to predict the entire scene in one forward step, enabling better interaction among tokens within the scene and significantly reducing inference time. In <Ref>, we provide a detailed explanation of the mechanism for executing next token / scene prediction simultaneously.















 §.§ Train Stage


Our training scheme includes three stages:



  §.§.§ Training of scene tokenizer
 We first focus on learning the scene codebook to represent occupancy as discrete tokens, using the objective defined in <Ref>. Once optimized, the scene tokenizer remains unchanged throughout the subsequent stages of the pipeline.



  §.§.§ 3D Occupancy-Language-Action pre-training
 We focus on aligning occupancy-language-action modalities in this stage. We use world model objective and scene-caption objective for full parameter pre-training, the former supervises the alignment between occupancy and action to learn the evolution of the world, the latter supervises the alignment between occupancy and language to learn the semantic understanding of the 3D scene.



  §.§.§ Instruction tuning
 In this stage, we fine-tunes the model based on prompt-based instructions for different scene understanding and planning tasks by LoRA <cit.>.






§ EXPERIMENTS



 §.§ Experiment Settings




  §.§.§ Dataset
 NuScenes <cit.> is a widely recognized foundation dataset in autonomous driving. The dataset comprises 700 training videos and 150 validation videos, each spanning 20 seconds with a key frame rate of 2Hz. Occ3D <cit.> is a large-scale dataset for 3D occupancy based on NuScenes, providing a semantic occupancy representation for each frame. NuScenes-QA <cit.> is a multi-modal visual question answering dataset based on Nuscenes. It encompasses five categories of questions: existence, counting, query-object, query-status, and comparison, which further categorized into zero-hop and one-hop by complexity. For aligning occupancy and language modalities, we collecte a large caption dataset based on NuScenes. Specifically, this dataset matches occupancy frames with the positions, classes, states, and future trajectories of objects appearing on them.



  §.§.§ Implementation Details
 
For most comparisons, we set language model backbone as LLaMA-3.1-8b and the scene tokenizer parameters as 50×256×2048. For VQA comparison, we set language model backbone as LLaMA-2-7b and scene tokenizer resolution as 25×25 for fairness. We employ the AdamW optimizer for all the training. Scene tokenizer is trained with learning rate of 10^-4, batch size of 4, λ_1=λ_3 = 10, λ_2 = λ_4 = 5, and λ_5 = 5, while Generative Model is trained with learning rate of 10^-4 and batch size of 1 in pre-training strage, 5 × 10^-5 and 4 in instruction tuning stage. The Scene tokenizer undergoes 100 epoch on 8 RTX 4090 GPUs, while the Generative Model undergoes 10 epoch in pre-training stage and 5 epoch in instruction tuning stage on 8 V100 GPUs.




























 §.§ Results and Analysis



  §.§.§ 4D Occupancy Forecasting
 The task aims to forecast the future 3D occupancy scene given a few historical occupancy inputs. Specifically, we follow existing works <cit.> and used a 2-second historical frames to forecast the subsequent 3 seconds and use mIoU and IoU as the main evaluation metrics. As illustrated in <Ref>, we compare OccLLaMA with the state-of-the-art approach, OccWorld, in two settings: using ground-truth 3D occupancy (-O), using predicted results from FBOCC <cit.> based on camera data (-F). Firstly, we observe that our scene tokenizer demonstrates superior scene reconstruction capabilities. Additionally, OccLLaMA achieves a competitive forecasting result within 1s, and significantly outperformes OccWorld over longer time, highlighting its enhanced long-term prediction capabilities. Furthermore, OccLLaMA-F can be regarded as an end-to-end pipeline as it takes cameras as input. Despite the complexity of the task, OccLLaMA consistently exhibits strong predictive performance. We present visualizations consistent with the above conclusions in <Ref>. 











































  §.§.§ Motion Planning
 

As illustrated in <Ref>, We compare the motion planning capabilities of OccLLaMA with several strong baselines that utilize various inputs and supervisions. We also compare our model with OccWorld under different settings as those in 4D occupancy forecasting task. We observe that UniAD <cit.> delivers the best performance, while the supervision annotations, limiting its scalability to large-scale datasets. As an alternative, OccLLaMA achieves competitive performance relying solely on 3D semantic occupancy, demonstrating its potential to scale as a foundational model in autonomous driving. Compared to methods using occupancy as input, OccLLaMA significantly outperforms OccNet <cit.>, further highlighting the superiority of the Autoregression. Additionally, surpassing the autoregressive state-of-the-art method OccWorld demonstrates the effectiveness of our pipeline. Moreover, the non-trivial performance achieved by integrating existing methods showcases the generalizability of our approach. Notably, outputting trajectories without alternating scene predictions results in performance drop, suggests that world model paradigm holds greater potential.


































  §.§.§ Visual Question Answering


To the best of our knowledge, we are the first MLLM to utilize Occupancy data with textual instructions as input and implement a series of 3D tasks in autonomous driving. We choose LiDAR-LLM <cit.>, the state-of-the-art on NuScenes-QA benchmark that integrates LiDAR into LLMs, as our primary baseline for comparison. Additionally, we evaluated a robust 2D LLM on the NuScenes-QA benchmark using depth images and raw images as input separately. We assess model performance using Top-1 accuracy metric and conduct separate evaluations for different question types. To ensure fairness, we implemente our pipeline under LLaMA2-7b, the same base model as LiDAR-LLM <cit.> and LLaVA <cit.>.
As illustrated in <Ref>, we observe OccLLaMA delivers the best performance overall. Compared to LiDAR-LLM, OccLLaMA can capture semantic information in 3D space better, which is essential for object-related questions. Additionally, OccLLaMA incorporates spatial information as input and aligns semantic and spatial data naturally, which is beneficial for questions involving spatial relationships.
























 §.§ Ablation Study



  §.§.§ Scene Tokenizer Parameters


<Ref> compares the impact of different hyperparameters on the reconstruction performance of the Scene Tokenizer, including latent space resolution, feature dimension, and codebook size. A larger codebook leads to overfitting and inefficient codebook utilization. A smaller codebooks and feature dimensions fail to effectively model the scene distributions. The resolution is positively correlated with reconstruction ability and has the most significant impact. However, larger resolution result in a greater token number to reconstruct a scene, thereby increasing the burden on forecasting. 





















  §.§.§ Generative Model Components


We compares the impact of different components of the generative model on forecasting and planning performance. As illustrated in <Ref>, w/o spatial attention means that the tokens in one scene maintain their original causal attention based on the flattened sequence order. w/o action tokenization means that waypoints are formed by concatenating tokens from original language vocabulary. We observe that using action-specific tokens, rather than relying on language vocabulary, results in performance gains on forecasting and planning. This improvement can be attributed to that action-specific tokens preserve the physical priors of waypoints while avoiding the inductive bias in language vocabulary. Additionally, we find that employing spatial attention to model spatial dependencies within the scene is essential for forecasting. However, it leads to a slight decrease in planning performance, which we attribute to spatial attention locally disrupting the global causal attention.























  §.§.§ Benefits of Pretraining

As shown in <Ref>, we compare the impact of different training settings on QA performance: instruction fine-tuning from pretraining (<Ref>) versus instruction fine-tuning from scratch. We observe that pretraining for modality alignment lead to an overall improvement on VQA. This indicates that when OccLLaMA gains an understanding of basic 3D scenes and world dynamics, it can better accomplish high-level QA tasks.

















§ CONCLUSION

In this paper, we propose OccLLaMA, a 3D occupancy-language-action generative world model in  autonomous driving for multitask. We introduce a novel scene tokenizer for discretization and reconstruction of occupancy scene. Furthermore, we build a unified multi-modal vocabulary that involve occupancy, language, and action modalities. Base on this vocabulary, we adapt LLM to perform the next token / scene prediction to complete multitask. Through extensive experiments on 4D occupancy forecasting, motion planning, and VQA, we demonstrate the multitask effectiveness of OccLLaMA. In the future, we will increase data diversity to further enhance the capabilities of OccLLaMA. We will also explore model quantization and distillation to address the inference delay caused by the large number of parameters.

































































