# Image Aesthetics Assessment via Learnable Queries

###### Abstract

Image aesthetics assessment (IAA) aims to estimate the aesthetics of images.
Depending on the content of an image, diverse criteria need to be selected to
assess its aesthetics. Existing works utilize pre-trained vision backbones
based on content knowledge to learn image aesthetics. However, training those
backbones is time-consuming and suffers from attention dispersion. Inspired by
learnable queries in vision-language alignment, we propose the Image
Aesthetics Assessment via Learnable Queries (IAA-LQ) approach. It adapts
learnable queries to extract aesthetic features from pre-trained image
features obtained from a frozen image encoder. Extensive experiments on real-
world data demonstrate the advantages of IAA-LQ, beating the best state-of-
the-art method by 2.2% and 2.1% in terms of SRCC and PLCC, respectively.

Index Terms—  Aesthetics Assessment, Learnable Queries

##  1 Introduction

Image aesthetics assessment (IAA) is a computer vision task that aims to
evaluate the aesthetic quality of images. Such a capability is beneficial for
downstream applications including image recommendation, enhancement,
retrieval, and generation [1]. Due to the inherent subjectivity and ambiguity
associated with image aesthetics, the ground truth of image aesthetics is
usually determined by the opinions of different reviewers in the form of the
mean opinion score (MOS) or the distribution of opinion scores (DOS).

Depending on the content of the image, there can be different emphases
involved in aesthetics assessment. Early works proposed to split the images
into different semantic groups and extract different sets of aesthetic
features [2, 3, 4]. The features can be handcrafted under the guidance of
photography rules [2], or extracted by deep neural networks [3, 4]. However,
there may be special cases where an image does not belong to any predefined
semantic group. Explicitly splitting these images based on semantics can
result in the relationships among different semantic groups being overlooked.

Therefore, later works [5, 6, 7] attempted to implicitly extract aesthetic
features from pre-trained semantic backbones. Since fine-tuning the entire
backbone is computationally expensive [5, 7] and might lead to attention
dispersion [6], the backbone is only used as a feature extractor. However,
since the vision backbones are pre-trained primarily for image classification
[5, 7] or scene recognition [6], they lack knowledge regarding the aesthetic
attributes (e.g., composition) that are less related to semantics. Moreover,
some of these works [5, 7] require input images in full resolution and
additional feature extraction stages in advance, which lack efficiency and
practical applicability.

More recently, with the prevalence of pre-trained large vision-language
models, works that utilize such models together with specially designed
prompts for IAA are starting to emerge [8, 9]. The vision-language models used
are either pre-trained solely on general image-text pairs [8], or further pre-
trained on aesthetic image-text pairs [9]. Either way, their pre-trained
models are not limited to aesthetic-related semantic patterns but also cover
relatively abstract knowledge related to image aesthetics. However, they use
relatively simple single prompts (e.g., “good image”) or prompt pairs (e.g.,
“good photo” and “bad photo”) to extract aesthetic-related knowledge, which
cannot deal with complex IAA tasks.

Inspired by BLIP-2 [10], a vision-language pre-training model that uses
learnable queries to align vision and language pre-trained features extracted
from frozen unimodal models, we propose the Image Aesthetics Assessment via
Learnable Queries (IAA-LQ) approach. It trains learnable queries to extract
aesthetic features from the pre-trained vision backbones. With a flexible
quantity, the learnable queries can extract the most aesthetics-related image
patterns from the frozen vision backbone. Extensive experiments on real-world
data demonstrate the advantages of IAA-LQ, beating the best state-of-the-art
method by 2.2% and 2.1% in terms of Spearman’s rank correlation coefficient
(SRCC) and Pearson linear correlation coefficient (PLCC), respectively.

##  2 The Proposed IAA-LQ Approach

The proposed IAA-LQ approach is shown in Figure 1. It consists of a frozen
image encoder to extract pre-trained image features, a set of learnable
queries, a querying transformer for self- and cross-attention, and a
prediction header for IAA.

Fig. 1: The design of IAA-LQ. It learns embeddings for learnable queries
through a querying transformer, where pre-trained image features extracted
with a frozen image encoder are inserted once in every two transformer blocks
for cross-attention. The learned query embeddings are averaged and passed
through a feed-forward layer and Softmax to output the predicted aesthetic
DOS.

###  2.1 Encoding an Image

A given image can be expressed as $(I_{n},\boldsymbol{d}_{n})$, where $I_{n}$
is the $n$-th image in an IAA dataset, and
$\boldsymbol{d}_{n}=\\{d_{n}^{k}\\}_{k=1}^{K}$ is its ground truth aesthetic
DOS which satisfies $\sum_{k=1}^{K}d_{n}^{k}=1$. Note the aesthetic MOS
$s_{n}$ can be derived from $\boldsymbol{d}_{n}$ as:

$s_{n}=\sum_{k=1}^{K}(k\cdot d_{n}^{k}).$ (1)

In our design, we adopt pre-trained vision transformers (ViT) as the image
encoder. It splits the input image into $N_{p}$ patch tokens and adds a [CLS]
token as the first token. Therefore, the extracted pre-trained image
embeddings $\boldsymbol{e}_{v}^{n}$ of image $I_{n}$ can be expressed as:

$\boldsymbol{e}_{v}^{n}=E_{\theta_{v}}(I_{n}),$ (2)

where $E_{\theta_{v}}(\cdot)$ is the ViT parameterized by frozen $\theta_{v}$,
and $\boldsymbol{e}_{v}^{n}\in\mathbb{R}^{(1+N_{p})\times H_{v}}$ with $H_{v}$
denoting the hidden size of the ViT.

###  2.2 Learnable Queries & Querying Transformer

Suppose we use $M$ learnable queries with hidden size $H_{q}$ to extract the
aesthetic features from the pre-trained image features. The pre-trained image
embeddings are inserted once every two transformer blocks into the querying
transformer for cross-attention with the queries. In each block, the queries
first interact with each other through self-attention, and then possibly
conduct cross-attention with the image embeddings. Finally, they are passed
through a feed-forward layer to output query embeddings. Formally, for queries
$\boldsymbol{q}\in\mathbb{R}^{M\times H_{q}}$ and pre-trained image embeddings
$\boldsymbol{e}_{v}^{n}$, the output query embeddings $\boldsymbol{e}_{q}^{n}$
with aesthetic knowledge extracted from the image is expressed as:

$\boldsymbol{e}_{q}^{n}=E_{\theta_{q}}(\boldsymbol{q}|\boldsymbol{e}_{v}^{n}),$
(3)

where $E_{\theta_{q}}(\cdot)$ is the querying transformer parameterized by
$\theta_{q}$, and $\boldsymbol{e}_{q}^{n}\in\mathbb{R}^{M\times H_{q}}$.

###  2.3 Prediction Header for IAA

After the output query embeddings $\boldsymbol{e}_{q}^{n}$ are obtained, we
take the average of the query embeddings to obtain a compact aesthetic
embedding:

$\boldsymbol{e}_{a}^{n}=\bar{\boldsymbol{e}}_{q}^{n},$ (4)

where $\boldsymbol{e}_{a}^{n}\in\mathbb{R}^{H_{q}}$. Based on the aesthetic
embedding, a prediction header for IAA is appended. It is a feed-forward layer
with an input size of $H_{q}$ and an output size of $K$ followed by a Softmax
layer. Together, the predicted $K$-scale aesthetic DOS can be computed as:

$\hat{\boldsymbol{d}}_{n}=E_{\theta_{p}}(\boldsymbol{e}_{a}^{n}).$ (5)

$E_{\theta_{p}}(\cdot)$ is the prediction header parameterized by
$\theta_{p}$. Following [11], we adopt Earth Mover’s Distance (EMD) loss to
optimize the predicted DOS towards the ground truth:

$\mathcal{L}(\boldsymbol{d}_{n},\hat{\boldsymbol{d}}_{n})=\sqrt{\frac{1}{K}\sum_{k=1}^{K}\lvert
CDF_{\boldsymbol{d}_{n}}(k)-CDF_{\hat{\boldsymbol{d}}_{n}}(k)\rvert^{2}}.$ (6)

$CDF_{\boldsymbol{d}_{n}}$ and $CDF_{\hat{\boldsymbol{d}}_{n}}$ are cumulative
density functions for the ground truth DOS and predicted DOS, respectively.
The overall objective is to minimize the loss over the whole dataset:

$\min_{\boldsymbol{q},\theta_{q},\theta_{p}}\left\\{\sum_{n=1}^{N}\mathcal{L}(\boldsymbol{d}_{n},\hat{\boldsymbol{d}}_{n})\right\\}.$
(7)

##  3 Experimental Evaluation

###  3.1 Experiment Settings

The dataset adopted for our experimental evaluation is the benchmark IAA
dataset, the AVA dataset [12]. It contains over 250,000 images, each of which
received 78 to 549 aesthetic scores (average of 210) on a scale of 1 to 10
(i.e., $K=10$), where 1 and 10 denote the lowest and highest aesthetics,
respectively. The ground truth DOS could be derived by computing the ratio of
reviewers under each score level to the total number of reviewers. The same
train-test split in [5, 9] is adopted for our experiments, where 235,574 and
19,928 images are allocated for training and testing, respectively. We also
include the generic testing set of the PARA dataset [13] to evaluate the
generalization ability and attribute-level performance of IAA-LQ. It contains
3,000 images with overall and attribute-level aesthetic annotations.

Method | SRCC | PLCC
---|---|---
NIMA [11] | 0.612 | 0.636
AFDC + SPP [14] | 0.649 | 0.671
GPF-CNN [15] | 0.671 | 0.682
MaxViT [16] | 0.708 | 0.745
MUSIQ [17] | 0.726 | 0.738
MLSP [5] | 0.756 | 0.757
TANet [6] | 0.758 | 0.765
GAT×3-GATP [18] | 0.762 | 0.764
VILA-R [9] | 0.774 | 0.774
IAA-LQ | 0.791 | 0.790
Table 1: Experiment results on the AVA dataset.

Following [10], the pre-trained image embeddings are retrieved from the second
last layer of the pre-trained ViT. The image resolution is set to $224\times
224$ with a patch size of $14\times 14$, yielding 257 image embeddings (i.e.,
$N_{p}=256$). We set the query hidden size $H_{q}$ to 768, initialize the
Q-Former with BERT${}_{\text{base}}$ [19], and randomly initialize the cross-
attention layers. We train the Q-Former with a batch size of 128 for 10 epochs
with Adam optimizer. The learning rate is set to $3\times 10^{-5}$ initially,
and multiplied by $0.1$ every two epochs. To evaluate the performance of the
comparison approaches, we report Spearman’s rank correlation coefficient
(SRCC) and Pearson linear correlation coefficient (PLCC) between the predicted
and ground truth aesthetic MOSs.

###  3.2 Comparison Results

Table 1 shows the performances of IAA-LQ compared with 9 state-of-the-art
(SOTA) methods under the AVA dataset. It can be observed that IAA-LQ achieves
the best performance, exceeding the best-performing SOTA method VILA-R [9] by
$2.2\%$ and $2.1\%$ in terms of SRCC and PLCC, respectively. To obtain this
model, we employ Horizontal Flipping (HF) with $p=0.5$ on training images, use
ViT-G/14 from EVA-CLIP [20] as the frozen vision backbone which outputs pre-
trained image embeddings with $H_{v}=1408$, and $M=2$ learnable queries to
extract the aesthetic features. Similar to VILA-R, IAA-LQ takes input images
with a low resolution rather than the full resolution as required by previous
works [5, 17]. It demonstrates that IAA-LQ can effectively learn from the most
critical aesthetics-related image clues.

###  3.3 Ablation Studies

Effects of image padding and augmentations: Since image aesthetics is
empirically found to be sensitive to image paddings and augmentations, we
investigate their effects in IAA-LQ. The two most common augmentations used in
IAA, Horizontal Flipping (HF) and Random Cropping (RC), are considered. As
shown in Table 2, RC deteriorates our model performance, which could be due to
the content and composition changes that are highly related to image
aesthetics as described in photographic rules such as the Rule of Thirds. On
the other hand, HF brings slight performance improvement because it generally
does not affect image aesthetics, while bringing in more samples to train on.

Padding | Augmentation | SRCC | PLCC
---|---|---|---
True | None | 0.782 | 0.783
True | HF | 0.784 | 0.785
True | RC | 0.781 | 0.781
True | HF + RC | 0.780 | 0.780
False | None | 0.790 | 0.790
False | HF | 0.791 | 0.790
False | RC | 0.784 | 0.785
False | HF + RC | 0.785 | 0.784
Table 2: Performance of IAA-LQ with different padding and augmentation strategies. HF denotes Horizontal Flipping with $p=0.5$. RC denotes resizing to $272\times 272$ followed by Random Cropping of $224\times 224$. M | 32 | 1 | 2 | 3 | 4
---|---|---|---|---|---
SRCC | 0.768 | 0.788 | 0.791 | 0.790 | 0.788
PLCC | 0.766 | 0.788 | 0.790 | 0.789 | 0.788
Table 3: Performance of IAA-LQ with different numbers of learnable queries. M denotes the number of queries. Vision Backbone | Embeddings | SRCC | PLCC
---|---|---|---
ViT${}_{\text{CLIP}}$ | CLS | 0.735 | 0.735
ViT${}_{\text{CLIP}}$ | CLS+P | 0.626 | 0.628
ViT${}_{\text{CLIP}}$ | LQ | 0.774 | 0.775
ViT${}_{\text{EVA-CLIP}}$ | CLS | 0.743 | 0.743
ViT${}_{\text{EVA-CLIP}}$ | CLS+P | 0.713 | 0.712
ViT${}_{\text{EVA-CLIP}}$ | LQ | 0.791 | 0.790
Table 4: Performance of IAA-LQ with different vision backbones and different
embeddings for the aesthetic prediction. CLS denotes the embedding of the
[CLS] token. CLS+P denotes the average of the embedding of the [CLS] token and
patch embeddings. LQ denotes the average of the learned query embeddings.

Effects of number of queries: The number of learnable queries $M$ is a crucial
hyperparameter in IAA-LQ as it directly affects the richness of the aesthetic
features extracted. As shown in Table 3, we start exploring the optimal $M$ by
setting it to 32, the same as in BLIP-2 [10]. However, it can be observed that
with 32 queries, IAA-LQ stops improving at very early stages and easily
overfits. Therefore, we then attempt to set $M=1$ and increase it gradually.
Experiment results demonstrate that IAA-LQ achieves the best performance when
only two queries are used. It demonstrates its ability of using a small number
of queries to capture adequate aesthetic features for IAA.

Effectiveness of the learnable queries: The frozen vision backbone determines
the pre-trained image patterns that can be used for IAA. In our experiments,
we consider ViT-L/14 from CLIP [21] and ViT-G/14 from EVA-CLIP [20], which
output image embeddings with $H_{v}=1024$ and $H_{v}=1408$, respectively. To
demonstrate the effectiveness of the learnable queries, we compare the
performances of learning image aesthetics from the pre-trained [CLS]
embedding, from the average of pre-trained [CLS] and patch embeddings, and
from the query embeddings of the learnable queries. As shown in Table 4, the
outstanding performance of the learnable queries demonstrates the
effectiveness of the design of the IAA-LQ learnable queries and querying
transformer in extracting relevant aesthetic features from pre-trained image
features.

###  3.4 Model Interpretation

Fig. 2: Examples of the IAA-LQ MOS prediction results. Images from the top row
to the bottom row are example images with relatively high, moderate, and
relatively low ground truth MOSs. The blue and (green) numbers beneath each
image are its predicted and (ground truth) MOSs, respectively.

In Figure 2, we show some examples of the aesthetic MOS prediction results
using the proposed IAA-LQ approach. The example images are retrieved from the
testing set of AVA. It can be observed that IAA-LQ can estimate image
aesthetics for various image contents, such as landscapes, objects, and
portraits. However, it does suffer from increased prediction errors when the
test image has relatively extreme aesthetic quality (either too high or too
low). It can be attributed to the data distribution of the AVA dataset, where
the majority of the images have aesthetic MOS around the borderline (i.e., 5).
This can be improved by training IAA-LQ on more images with relatively extreme
aesthetics MOSs.

Attribute | SRCC | PLCC
---|---|---
Aesthetics | 0.701 | 0.739
Composition | 0.702 | 0.737
Content | 0.686 | 0.730
Quality | 0.684 | 0.725
DOF | 0.667 | 0.711
Light | 0.642 | 0.694
Color | 0.619 | 0.667
Table 5: Attribute-level performance of IAA-LQ.

To evaluate the generalization capability of IAA-LQ and its ability to
indicate fine-grained aesthetic attributes, we directly perform inference on
the testing set of PARA using IAA-LQ and compare the estimated aesthetics MOSs
with the ground truth MOSs of different aesthetic attributes. Table 5 shows
that even without further supervision from the training set of PARA, IAA-LQ
can still estimate the overall aesthetic qualities reasonably well. The high
correlation with the composition and content attributes could be credited to
the relatively simple augmentations used and the frozen pre-trained image
encoder. Interestingly, our predicted MOS has a relatively low correlation
with the color attribute. This can be attributed to the large amount of near-
gray low-saturation images in AVA. Nevertheless, the proposed IAA-LQ approach
demonstrates strong generalization capability and attribute-level aesthetics
awareness.

##  4 Conclusions and Future Work

In this paper, we propose the IAA-LQ approach, which learns image aesthetics
using learnable queries. With a small number of learnable queries, our
approach can effectively extract the most relevant aesthetic features from the
pre-trained image features obtained with a frozen image encoder. Extensive
experiments demonstrate that IAA-LQ largely outperforms state-of-the-art
approaches on the benchmark AVA dataset, and achieves strong generalization
ability.

In subsequent research, we plan to improve the explainability of IAA-LQ with
additional outputs in terms of aesthetic attributes and captions or additional
logical reasoning in the model design.

## References

  * [1] Jiajing Zhang, Yongwei Miao, and Jinhui Yu,  “A comprehensive survey on computational aesthetic evaluation of visual art images: Metrics and challenges,”  IEEE Access, vol. 9, pp. 77164–77187, 2021.
  * [2] Xiaoou Tang, Wei Luo, and Xiaogang Wang,  “Content-based photo quality assessment,”  IEEE Transactions on Multimedia, vol. 15, no. 8, pp. 1930–1943, 2013.
  * [3] Yueying Kao, Kaiqi Huang, and Steve Maybank,  “Hierarchical aesthetic quality assessment using deep convolutional neural networks,”  Signal Processing: Image Communication, vol. 47, pp. 500–510, 2016\.
  * [4] Weining Wang, Mingquan Zhao, Li Wang, Jiexiong Huang, Chengjia Cai, and Xiangmin Xu,  “A multi-scene deep learning model for image aesthetic evaluation,”  Signal Processing: Image Communication, vol. 47, pp. 511–518, 2016\.
  * [5] Vlad Hosu, Bastian Goldlucke, and Dietmar Saupe,  “Effective aesthetics prediction with multi-level spatially pooled features,”  in Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19), 2019, pp. 9375–9383.
  * [6] Shuai He, Yongchang Zhang, Rui Xie, Dongxiang Jiang, and Anlong Ming,  “Rethinking image aesthetics assessment: Models, datasets and benchmarks,”  in Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJCAI’22), 2022, pp. 942–948.
  * [7] Zhiwei Xiong, Han Yu, and Zhiqi Shen,  “Federated learning for personalized image aesthetics assessment,”  in 2023 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2023, pp. 336–341.
  * [8] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy,  “Exploring clip for assessing the look and feel of images,”  in Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI-23), 2023, pp. 2555–2563.
  * [9] Junjie Ke et al.,  “Vila: Learning image aesthetics from user comments with vision-language pretraining,”  in Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’23), 2023, pp. 10041–10051.
  * [10] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi,  “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,”  arXiv preprint arXiv:2301.12597, 2023.
  * [11] Hossein Talebi and Peyman Milanfar,  “Nima: Neural image assessment,”  IEEE Transactions on Image Processing, vol. 27, no. 8, pp. 3998–4011, 2018.
  * [12] Naila Murray, Luca Marchesotti, and Florent Perronnin,  “Ava: A large-scale database for aesthetic visual analysis,”  in Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’22), 2012, pp. 2408–2415.
  * [13] Yuzhe Yang, Liwu Xu, Leida Li, Nan Qie, Yaqian Li, Peng Zhang, and Yandong Guo,  “Personalized image aesthetics assessment with rich attributes,”  in Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’22), 2022, pp. 19861–19869.
  * [14] Qiuyu Chen et al.,  “Adaptive fractional dilated convolution network for image aesthetics assessment,”  in Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20), 2020, pp. 14114–14123.
  * [15] Xiaodan Zhang, Xinbo Gao, Wen Lu, and Lihuo He,  “A gated peripheral-foveal convolutional neural network for unified image aesthetic prediction,”  IEEE Transactions on Multimedia, vol. 21, no. 11, pp. 2815–2826, 2019.
  * [16] Zhengzhong Tu et al.,  “Maxvit: Multi-axis vision transformer,”  in Proceedings of the 2022 European Conference on Computer Vision (ECCV’22), 2022, pp. 459–479.
  * [17] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang,  “Musiq: Multi-scale image quality transformer,”  in Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (CVPR’21), 2021, pp. 5148–5157.
  * [18] Koustav Ghosal and Aljosa Smolic,  “Image aesthetics assessment using graph attention network,”  in Proceednigs of the 2022 International Conference on Pattern Recognition (ICPR’22), 2022.
  * [19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  arXiv preprint arXiv:1810.04805, 2018.
  * [20] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao,  “Eva: Exploring the limits of masked visual representation learning at scale,”  in Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’23), 2023, pp. 19358–19369.
  * [21] Alec Radford et al.,  “Learning transferable visual models from natural language supervision,”  in Proceedings of the 2021 International Conference on Machine Learning (ICML’21), 2021, pp. 8748–8763.


# Occupancy Prediction-Guided Neural Planner for Autonomous Driving

Haochen Liu, Zhiyu Huang, and Chen Lv∗ Code is available at:
https://github.com/georgeliu233/OPGPH. Liu, Z. Huang, and C. Lv are with the
School of Mechanical and Aerospace Engineering, Nanyang Technological
University, 639798, Singapore. (E-mails: {haochen002<EMAIL_ADDRESS>lyuchen@ntu.edu.sg)This work was supported in part by the A*STAR under MTC IRG
Grant (No. M22K2c0079) and the SUG-NAP Grant (No. M4082268.050) of Nanyang
Technological University, Singapore.∗Corresponding author: C. Lv

###### Abstract

Forecasting the scalable future states of surrounding traffic participants in
complex traffic scenarios is a critical capability for autonomous vehicles, as
it enables safe and feasible decision-making. Recent successes in learning-
based prediction and planning have introduced two primary challenges:
generating accurate joint predictions for the environment and integrating
prediction guidance for planning purposes. To address these challenges, we
propose a two-stage integrated neural planning framework, termed OPGP, that
incorporates joint prediction guidance from occupancy forecasting. The
preliminary planning phase simultaneously outputs the predicted occupancy for
various types of traffic actors based on imitation learning objectives, taking
into account shared interactions, scene context, and actor dynamics within a
unified Transformer structure. Subsequently, the transformed occupancy
prediction guides optimization to further inform safe and smooth planning
under Frenet coordinates. We train our planner using a large-scale, real-world
driving dataset and validate it in open-loop configurations. Our proposed
planner outperforms strong learning-based methods, exhibiting improved
performance due to occupancy prediction guidance.

##  I Introduction

Accurately predicting robust, socially compatible joint futures for multiple
traffic participants (agents) to inform the decision-making module is a
crucial capability for autonomous driving systems (ADS) [1, 2, 3]. However,
integrating prediction and planning presents significant challenges due to
several factors. First, ADS must process a vast array of complex traffic
scenes, each featuring combinations of socially interactive heterogeneous
traffic participants [4]. Second, the motion predictor must manage joint
patterns of future states for numerous traffic actors in the vicinity [5].
Moreover, path-level planning decisions necessitate feasible guidance from
joint predictions to execute safe and smooth planning performances for ADS.

To achieve optimal prediction guidance, the majority of prediction research
focuses on multi-agent trajectory prediction (MATP), which directly maps the
joint sequences of future locations for selected surrounding participants [6,
7]. However, this approach is hindered by the search cost, which grows
linearly with the number of participants and exponentially for marginal motion
predictions [8]. Recently, researchers have turned to forecasting occupancy
grids [9].

Figure 1: Proposed two-stage occupancy prediction-guided neural planner
(OPGP). Predicted occupancy and multi-modal planning outputs from an
integrated learning-based framework. We then refine the planning from
transformed prediction and planning via prediction-guided optimizations.

As a visual representation, occupancy prediction provides a more efficient and
scalable form for multiple participants. In addition, it can predict the joint
future for various participants at a single glance [10], offering enhanced
safety for occlusion predictability compared to MATP when integrating planning
decisions. However, capturing the social awareness of all interactive actors
in the context of joint occupancy forecasting remains an open question.

Conversely, learning-based motion planning has gained increasing attention
[11]. Its ability to handle diverse driving scenarios has achieved impressive
results through the use of a unified neural network. Nevertheless, the
robustness and safety of learning-based planning can be undermined by
instability. To address this issue, integrating prediction guidance has
emerged as a standard approach for this learning structure [12, 13, 14]. By
learning active responses and motion constraints based on neighboring
predictions, a safer, more robust, and socially compliant planning system is
anticipated [15]. One common method involves integrating predictions by using
the future trajectories of surrounding actors for planning [16, 17]. Despite
the difficulties of using MATP to model joint trajectories, limited research
has explored occupancy as a source of prediction guidance. Predicting joint
future occupancy presents considerable challenges in integrating visual-based
prediction features within the neural network for planning. Furthermore,
transforming informative predicted occupancy into feasible planning outcomes
is an issue that warrants further investigation.

To tackle these challenges, as shown in Fig. 1, we propose a two-stage
learning-based framework (named OPGP) that integrates joint predictions for
future occupancy and motion planning with prediction guidance. In the first
stage, an integrated network of occupancy prediction-guided planning is
established upon Transformer backbones. Building upon our previous work [18],
occupancy predictions for all types of traffic participants are output
simultaneously, taking into account interaction awareness for both visual
features and vectorized context. Meanwhile, encoded scene features and
occupancy are shared and conditionally queried in the planner head, which
conducts multi-modal motion planning for the ADS’s trajectories. The second
stage focuses on modeling explicit guidance from occupancy prediction for
motion planning refinement in an optimization-feasible manner. More
specifically, we construct an optimization pipeline in Frenet space [19] for
planning refinement using transformed occupancy predictions. The contributions
of our proposed framework are summarized as follows:

  1. 1.

We propose a two-stage occupancy prediction guided neural planner pipeline.
The first stage involves an integrated occupancy prediction-guided motion
planning framework. Predicted occupancy is combined with multi-modal planning
trajectories in Transformer-based structures, taking into account the social
interactions among traffic participants and scene context.

  2. 2.

In the second stage, we design a transformed occupancy prediction guided
optimization for planning refinement to further enhance planning performance.

  3. 3.

We validate the two-stage framework on a large-scale real-world driving
dataset, and the proposed pipeline achieves compelling performance.

##  II Related Work

###  II-A Joint motion prediction

A growing number of learning-based techniques have been developed for joint
motion predictions, proving highly effective [1]. This can be attributed to
the capability of deep neural networks, particularly Transformers and GNNs
[20], to handle intricate traffic scenarios involving multiple interacting
participants and diverse scene contexts. Agent-centric methods predict multi-
agent-wise future trajectories (MATP) anchored on each detected traffic
participant. DenseTNT [6], M2I [21], and HEAT [22] score joint predictions in
combinations of marginal ones for each actor with GNN backbones. While they
demonstrate accuracy for each actor, they also raise computational costs with
occasional inconsistency. Joint methods [23, 24] directly estimate the future
distribution of all actors from heatmap scoring or sampling. However, these
methods require a maximum number of predictions and have linear cost growth
with multiple agents, especially in crowded urban areas. The use of occupancy
grids to forecast future motions has been a technique employed in autonomous
driving for several years. One notable example is ChauffeurNet [25], which
utilizes occupancy maps to predict future movements and facilitate behavior
planning. StopNet [10] further conducts integrated trajectory and occupancy
predictions for scalable and real-time predictions. Building on our previous
method [18], we incorporate planning information into the learning framework
during prediction training. We have refined our occupancy prediction pipeline
to include all types of traffic participants and consider their interactions
within the scene context for enhancement to guide planning.

###  II-B Motion planning with prediction guidance

Motion planning is a well-established field that has received extensive study
over time. It has developed through various approaches such as path
optimization [26], sampling [27], and more recently, learning-based techniques
[28, 11]. However, cruising safely in interactive and sophisticated traffic
requires further guidance from the predicted states of participants’
behaviors. PiP [29] makes conditional predictions iteratively on sampling-
based planning rescheduling, which only considers marginal futures and is
limited by generated planning paths. DIPP [16] ties differentiable planning
objectives with joint trajectory predictions, enabling responsive planning.
While previous methods focus on prediction guidance in an agent-centric
manner, we seek scene-centric guidance in terms of scalability and invariance
for growing actors. PredictionNet [30] demonstrates occupancy prediction
guidance with post-processed trajectories to inform planning and control.
However, it focuses on occupancy prediction for traffic simulation and does
not specifically design integration for planning. End-to-end methods such as
MP3 [31] and Interfuser [32] directly map prediction and planning from raw
visual inputs. Nevertheless, they focus more on unifying the perception part
into prediction, and the planning outputs are simply retrieved from extensive
trajectory storage. In our work, we leverage occupancy prediction to guide
both stages of our proposed neural planner. Specifically, the prediction
features are queried interactively by the learning-based planning decoder, and
the resulting predictions are utilized to refine the planning in a feasible
manner.

##  III Methodology

###  III-A Problem Formulation

Figure 2: An overview of the integrated occupancy prediction and planning
learning framework in OPGP. Building on our previous work, visual and vector
scene inputs $\mathcal{S}$ are separately encoded in scene encoders; the
encoded scene features are distributed by occupancy prediction heads for all
type occupancy predictions $\mathbf{\hat{O}}_{1:T}$ and multi-modal planning
heads for initial planning outputs $\mathbf{\hat{Y}}^{e}_{1:T}$ for the
subsequent stage.

As shown in Fig. 1, the occupancy prediction guided neural planner consists of
two stages. In the first stage, motion-planning decisions with occupancy
prediction guidance can be formulated as a multi-task learning paradigm.
Conditioned on scene context $\mathcal{S}$ as inputs, the integrated learning
framework $f$ simultaneously outputs occupancy predictions
$\mathbf{\hat{O}}{1:T}$ and multi-modal imitated planning trajectories
$\mathbf{\hat{Y}}^{e}{1:T}$ in future horizons $T$. Subsequently, the second
stage utilizes a Frenet transformation [19] for both prediction and planning
to achieve attainable prediction-guided optimization that generates refined
planning outputs $\tau^{*}$, minimizing the defined costs $C$. Mathematically,
the two-stage task is formulated as:

$\displaystyle\mathbf{\hat{Y}}^{e}_{1:T},\mathbf{\hat{O}}_{1:T}$
$\displaystyle=f(\mathcal{S}|\theta),$ (1)
$\displaystyle\tau^{*}=\mathop{\arg\min}_{\tau}\ C($
$\displaystyle\mathbf{\hat{Y}}^{\mathbf{T}}_{1:T},\mathbf{\hat{O}}^{\mathbf{T}}_{1:T},\mathcal{S}),$

where $\theta$ denotes the model parameters. More specifically, the input
scene context $\mathcal{S}$ comprises multiple modalities under historical
horizon $T_{h}$ with detailed formulations as follows: 1) Visual features: We
construct a combination from the historical occupancy grids
$\mathbf{O}_{T_{h}:0}$ and rasterized BEV roadmap $\mathcal{M}$ [33]
representing the spatial-temporal status of traffic participants under a
specific scenario. Each occupancy $\mathbf{O}_{t}$ separately contains 3 types
of traffic participants (vehicles, pedestrians, and cyclists), and we further
add an occlusion dimension for predictions $\mathbf{\hat{O}}_{1:T}$ for
currently unseen actors. 2) Vector features: The vectorized context initially
concerns the dynamic context from a maximum of $n_{x}+1$ actors centered on
the autonomous vehicle
$\mathbf{X}_{-T_{h}:0}={X^{ego}_{-T_{h}:0},X^{1}_{-T_{h}:0},\cdots,X^{n}_{-T_{h}:0}}$,
each representing a historical motion sequence: $(x,y,v_{x},v_{y},\theta)$.
One-hot encoding is associated with respective types. To provide map context
for the autonomous (ego) vehicle, we collect map data using a graph search of
Lanelets [34], which includes $m$ waypoints $\mathcal{W}_{m}$ and reference
routes $\mathcal{I}$. Each map segment contains coordinate information as well
as road features such as road type, speed limit, and the current state of
nearby traffic lights.

###  III-B Integrated Prediction and Planning Framework

Fig. 2 illustrates the overall structure of our learning-based framework,
which integrates occupancy predictions and planning and consists of three
modules. First, we adopt the separate encoding for multi-modal scene inputs
$\mathcal{S}$ from our previous work. Encoded visual and vector features are
subsequently delivered to occupancy prediction and planner heads. For
occupancy heads, we maintain the previous aggregation and decoding pipelines,
which then conduct a late fusion of encoded vector and visual features. The
prediction heads ultimately output all types of occupancy predictions
$\mathbf{\hat{O}}{1:T}$ for each second of the future horizon. For multi-modal
planning heads, we follow the Transformer decoder paradigm and employ a
single-layer decoding head for multi-modal path planning
$\mathbf{\hat{Y}}^{e}{1:T}$. More specifically, we initially introduce a set
of $k$ planning anchors ${\mathcal{K}_{1},\cdot,\mathcal{K}_{k}}$ to guide $k$
modes for planning heads. Each anchor $\mathcal{K}$ can be either static end-
point clusters or dynamic ones for learnable embedding that heuristically
guide the planning decoding. These anchors then serve as context queries Q for
encoded visual and vector features after self-attentions.

For the multi-head cross-attention module (MHCA), we aim to adaptively obtain
interaction-aware features from visual context, vectorized dynamics, and scene
topology for the ego vehicle. We implement separate queries using different
MHCA blocks for visual and vector features, serving as
$\textbf{K},\textbf{V}$. This is due to the length of padded visual features
that will sparse the attention scoring when considering concatenated attention
heads. After subsequent feed-forward networks (FFNs), each mode will decode to
an output of a planning path $\hat{\mathbf{y}}^{k}_{1:T}$ and corresponding
probabilities $\hat{p}_{k}$ for imitation likelihoods.

We formulate the first-stage learning framework as a multi-task learning
paradigm, and the objectives are offline updated jointly using large-scale
datasets. For occupancy predictions, as each grid cell forecasts the occupying
probabilities under a certain future horizon, we employ focal loss [35] to
balance the occupied samples for each type of actor:

$\vspace{-0.1cm}\mathcal{L}_{pred}=\frac{1}{HWTU}\sum_{x,y,t,u}^{H,W,T,U}\operatorname{FL}(\mathbf{\hat{O}}^{u}_{t}(x,y),\mathbf{O}^{u}_{t}(x,y)),\vspace{-0.1cm}$
(2)

where $H,W$ represent the occupancy scales and $U$ denotes the actor types.
For multi-modal planning, the imitation learning objectives are updated as
follows:

$\vspace{-0.2cm}\mathcal{L}_{plan}=\mathop{\arg\min}_{k}\frac{1}{T}\sum_{t}^{T}\operatorname{SL_{1}}(\hat{y}^{k}_{t}-y^{k}_{t})-p_{k}\log{\hat{p}_{k}},$
(3)

where $\operatorname{SL_{1}}$ denotes smooth L1 loss, and the planning modes
with minimum distance to ground truth will be updated for the initial planning
path $\hat{\mathbf{Y}}^{e}_{1:T}$.

###  III-C Prediction-guided planning refinement

Given the occupancy prediction for scene traffic participants
$\mathbf{\hat{O}}_{1:T}$ as well as the multi-modal initial planning results
$\hat{\mathbf{Y}}^{e}_{1:T}$, the second stage of OPGP constructs a
prediction-guided pipeline for planning refinement. First, this section
designs a transformation paradigm using Frenet [19] coordinates for both
prediction and planning. Then, we introduce a planning optimization guided by
the transformed occupancy predictions.

Transformation: We leverage Frenet coordinates for transformation targets, as
they relax the difficulties in planning optimizations. Suppose a generated
reference route $\mathcal{I}$, each reference point $r\in\mathcal{I}$ is
dynamically assigned with a curvilinear frame by tangential and normal
vectors: $[\vec{t}_{r},\vec{n}_{r}]$. Then, the current Cartesian coordinate
$\vec{y}=(x,y)$ can be transformed into Frenet $\vec{r}=(s,d)$ through the
transformation $\mathcal{F}:\vec{y}\rightarrow\vec{r}$ following the
relations:

$\vec{y}(s(t),d(t))=\vec{r}(s(t))+d(t)\vec{n}_{r}(s(t)).$ (4)

For initial planning results $\hat{\mathbf{y}}^{k}_{1:T}$, we first select the
top-scoring trajectories
$\hat{\mathbf{Y}}^{e}_{1:T}=\mathop{\arg\max}_{p_{k}}\hat{y}^{k}{1:T}$, and
then transform them by
$\hat{\mathbf{Y}}^{\mathbf{T}}_{1:T}=\mathcal{F}(\hat{\mathbf{Y}}^{e}_{1:T})$.
For predicted occupancy $\mathbf{\hat{O}}_{1:T}$, our objective is to filter
out the occupancy prediction centered on $\mathcal{I}$ and stretch onto
occupancy under the Frenet frame. This transformation focuses the searching
space of scalable occupancy on planning without occlusions. Moreover, the
Frenet-transformed occupancy relieves the convexity for planning optimization.
More specifically, given predefined mesh-grid indices on the Frenet frame:
$\mathbf{I}_{sd},s\in[0,S];d\in[-\frac{D}{2},\frac{D}{2}]$, this procedure
transforms reversely from Frenet firstly onto the Cartesian frame by
$\mathcal{F}^{-1}$. Then, following the relations from Cartesian to occupancy
grids pixel $(w,h)$ denoted as $\mathcal{P}$:

$(w,h)=\operatorname{int}(\frac{1}{n}(x-x^{e}_{0},y-y^{e}_{0})),$ (5)

where $\operatorname{int}$ is the rounding function and $n$ represents pixels
per meter. The transformed occupancy prediction
$\mathbf{\hat{O}}^{\mathbf{T}}{1:T}$ is then gathered by:

$\displaystyle\vspace{-0.2cm}\mathbf{\hat{O}}^{\mathbf{T}}_{1:T}(s,d)$
$\displaystyle=\mathbf{\hat{O}}_{1:T}(\mathbf{I}_{w},\mathbf{I}_{h}),$ (6)
$\displaystyle[\mathbf{I}_{w};\mathbf{I}_{h}]$
$\displaystyle=\mathcal{P}(\mathcal{F}^{-1}(\mathbf{I}_{sd})).$

In OPGP pipelines, the output transformed planning
$\hat{\mathbf{Y}}^{\mathbf{T}}_{1:T}$ will then be optimized with the guidance
of predictions from $\mathbf{\hat{O}}^{\mathbf{T}}_{1:T}$ for planning
refinement.

Optimization: To ensure enhanced safety and motion performance for planning
results, here we formulate an open-loop optimization problem under finite
horizons. The optimization searches for an optimal sequence
$\tau^{*}=\\{\tau_{1},\cdots,\tau_{T}\\}$ that minimizes the cost function
sets $C$ guided by transformed occupancy predictions
$\mathbf{\hat{O}}^{\mathbf{T}}_{1:T}$. More specifically, $C$ is the weighted
$\omega_{i}$ squared sum of a set of cost functions $c_{i}$:

$\displaystyle\vspace{-0.2cm}\tau^{*}=$
$\displaystyle\mathop{\arg\min}_{\tau}C(\tau,\mathbf{\hat{O}}^{\mathbf{T}}_{1:T},\mathcal{S}),$
(7) $\displaystyle C=\sum_{i}\|$
$\displaystyle\omega_{i}c_{i}(\tau^{i},\mathbf{\hat{O}}^{\mathbf{T}}_{1:T},\mathcal{S})\|^{2},\tau^{i}\subseteq\tau.$

The cost function set $C$ embraces various cost terms $c_{i}$ with carefully-
devised weights $\omega_{i}$ that consider a variety of planning performances,
including driving progress, driving comfort, adherence to rules, and most
importantly, driving safety. The cost set is explicitly cataloged as follows:

1) Driving progress: To encourage efficient driving, we assist a consistent
longitudinal speed cost within speed limits $v^{s}_{\operatorname{limit}}$,
allowing more on-road progress:

$\textbf{c}^{\operatorname{progress}}_{t}=\dot{s}_{t}-v^{s}_{\operatorname{limit}}.$
(8)

2) Driving comfort: To facilitate planning comfort and driving smoothness, we
introduce the minimization for accelerations and jerks:

$\textbf{c}^{\operatorname{comfort}}_{t}=\ddot{s}_{t}+\ddot{d}_{t}+\dddot{s_{t}}.$
(9)

3) Adherence to rules: We expect the designed ADS to conform to traffic rules
and stay within reference lanes. Thus, we propose an off-route cost and a
traffic light cost based on max-margin objectives on $(s,d)$ respectively:

$\textbf{c}^{\operatorname{route}}_{t}=d_{t}.$ (10)
$\textbf{c}^{\operatorname{tl}}_{t}=\left\\{\begin{aligned}
&\mathbf{1}(s_{\operatorname{red}})(s_{t}-s_{\operatorname{red}}),\
s_{t}>s_{\operatorname{red}},\\\ &0,\
\operatorname{otherwise}.\end{aligned}\right.$ (11)

where $s_{\operatorname{red}}$ is the stop point for red-light reference lane.

4) Prediction guided safety: To enhance driving safety for planning
refinement, we leverage the guidance of transformed occupancy predictions
$\mathbf{\hat{O}}^{\mathbf{T}}_{1:T}$ as dynamic obstacles for safe driving.
The occupancy prediction is considered as a potential collision cost weighted
by actor types:

$\vspace{-0.2cm}\textbf{c}^{\operatorname{ogm}}_{t}(s,d)=\sum_{u}^{U}\lambda_{u}\mathbf{\hat{O}}^{\mathbf{T}u}_{t}(s,d),$
(12)

Due to the issues of preserved occupancy between frames, the predicted actors
in occupancy are possibly enlarged with long tails. This may lead to over-
conservative driving for ADS. Therefore, we design a confidence safety
distance $s^{\operatorname{safe}}_{t}$ for the confident collision cost over
threshold $\epsilon$:

$\vspace{-0.2cm}s^{\operatorname{safe}}_{t}=\min_{s}(\sum_{d}^{D}\textbf{c}^{\operatorname{ogm}}_{t}(s,d)>\epsilon),$
(13)

For optimization simplicity, we only conduct longitudinal guidance for
planning refinement. Considering the forward sign distance:

$\operatorname{sgnd}(s_{t})=(s_{t}-\mathbf{I}_{s})\mathbf{1}(s_{t}>\mathbf{I}_{s}),$
(14)

The prediction-guided safety cost is modeled as a weighted predicted collision
cost sum for sign distance that surpasses $s^{\operatorname{safe}}_{t}$:

$\textbf{c}^{\operatorname{safe}}_{t}=\sum_{s}^{S}\sum_{d}^{D}\operatorname{sgnd}(s_{t})\textbf{c}^{\operatorname{ogm}}_{t}(s,d)\mathbf{1}(s>s^{\operatorname{safe}}_{t}).$
(15)

TABLE I: Open-loop Testing Results on Waymo Motion Dataset | Safety (%) | Motion ($\operatorname{ms}^{-2},\operatorname{ms}^{-3}$) | Planning errors (m)
---|---|---|---
Models | Collisions | Off route | Red light | Jerk | Acc. | Lat.Acc. | 1s | 3s | 5s
Vanilla-IL | 7.575 | 2.76 | 1.671 | 4.323 | 0.676 | 0.129 | 0.195 | 1.076 | 2.759
DIM-OPGP[36] | 9.275 | 14.27 | 3.246 | 7.226 | 0.977 | 0.252 | 0.407 | 1.865 | 4.473
Vanilla-Test | - | - | - | 4.02 | 0.692 | 0.167 | - | - | -
OPGP | 3.018 | 1.809 | 0.501 | 3.869 | 0.551 | 0.156 | 0.226 | 1.21 | 3.184

To solve this non-linear optimization problem, as depicted in Equ. 7. We
employ Gauss-Newton method [37] that iterative refine the planning variable
$\tau$ based on $\hat{\mathbf{Y}}^{\mathbf{T}}_{1:T}$ as initial value.

##  IV Experiments

###  IV-A Experimental Setup

We utilize the Waymo Open Motion Dataset (WOMD) [38], which contains over
500,000 samples that cover a wide range of real-world driving scenarios. It
captures the complex interactions and dynamics among various types of traffic
agents, including vehicles, cyclists, and pedestrians. For scene context
$\mathcal{S}$ as inputs at the first stage, the historical states for each
actor are sampled at 10Hz for the past one second ($T_{h}=10$), while the
prediction and planning objectives are over the future 5 seconds at 1Hz for
prediction and 10Hz for planning ($T=50$). The BEV image resolution for input
and output is $H,W=128$, rendering $n=1.6$ pixels per meter in the real world.
We set the hidden dimension as 96. For vector inputs, we maintain a maximum of
$n_{x}=31$ surrounding actors sorted by their average distance to the AV.
Following the standard measure, we set the anchors $k=6$ in multi-modal
planning decoding. For the second stage of OPGP, the reference route
$\mathcal{I}$ is smoothly intersected every 0.1m, and the size of transformed
occupancy prediction $\mathbf{\hat{O}}^{\mathbf{T}}_{t}$ is $S=1000$, $D=20$.
For planning purposes, we randomly split the recorded 20 seconds of WOMD,
resulting in 297,669 samples for training and selected scenarios of interest
for validation and testing (47,728 each).

Metrics: To ensure a comprehensive evaluation of OPGP performance, for
occupancy predictions, we adopted the standard metrics proposed in challenges
[9]. The prediction metrics include AUC and sIOU for $\mathbf{\hat{O}}_{1:T}$.
For planning metrics, we focus on the driving performance concerning human-
likeness, motion capabilities, and safety during testing.

Baselines: To validate the prediction and planning performance of the OPGP
framework, we compare it with several strong baselines. 1) For occupancy
prediction, we provide the vanilla STrajNet [18] without a planning head and
VectorFlow-SwinT, the improved baseline predictor with simplified aggregation
and encoding using the Swin Transformer bottleneck [39]. 2) For open-loop
planning, we compare it with Vanilla IL, i.e., only the first stage of OPGP,
and DIM [36] devised with OPGP bottlenecks. These baselines also serve as
ablative methods without some of the proposed modules in OPGP.

###  IV-B Implementation Details

We select the GELU activation function and apply a dropout rate of 0.1 after
each layer to mitigate overfitting issues. Due to the large volume of data, we
employ a distributed strategy on four NVIDIA A100 GPUs with a total batch size
of 64. The AdamW optimizer is used with an initial learning rate of 1e-4, and
a cosine annealing learning rate strategy is employed. The total number of
training epochs is set to 30.

###  IV-C Quantitative Results

1) Prediction Performance: We report the testing performance of OPGP in
comparison with other strong baselines. As shown in Table II, when compared
with the Vanilla STrajNet, which is specifically designed for the occupancy
prediction task, the prediction performance of OPGP exhibits a slight
decrease. This is due to OPGP being a multi-task learning pipeline with both
prediction and planning heads. More importantly, the current design only
considers feature guidance from prediction towards planning. Consequently, the
training objectives for multi-modal planning require further contributions to
the decoding head for occupancy prediction.

TABLE II: Testing Results on Occupancy Predictions Models | Vec-AUC | Vec-sIOU | Occ-AUC | Occ-sIOU
---|---|---|---|---
STrajNet[18] | 0.856 | 0.696 | 0.146 | 0.023
VectorFlow[39] | 0.813 | 0.656 | 0.112 | 0.017
OPGP | 0.854 | 0.679 | 0.128 | 0.02

2) Planning Performance: We conduct open-loop planning testing. As shown in
Table I, Vanilla IL, which employs the first stage of OPGP, achieves the
lowest planning errors, demonstrating the performance gain of multi-modal
configurations using $k$ anchors. In contrast, a uni-modal Gaussian for
imitative planning in DIM cannot resolve the uncertainty in complex scenarios.
The proposed OPGP yields low planning errors while enhancing safety compared
to IL. The collision and off-route rates are significantly improved due to the
occupancy prediction-guided optimization in the second stage. The well-
designed cost function also promotes driving comfort by reducing jerks in
motion metrics.

Figure 3: Qualitative results of occupancy predictions from the testing set.
The sub-figures display several selected driving scenarios with specific
features: a) T-intersections with merging; b) Parking lots with numerous
pedestrians; c) Occluded intersections; d) Fork roads with adjacent cyclists.

###  IV-D Qualitative Results

To gain a better understanding of the effectiveness of our OPGP, we first
employ visualizations of occupancy prediction to assess its performance on
multiple representative driving scenarios during testing. As shown in Fig. 3,
the proposed method effectively handles both observed actors (a, c, d) and
high-occlusion risk scenarios (a, c). Prediction accuracy is well-maintained
for vehicles, but current methods can only ensure 3s prediction performance
for smaller actors (b, d). To demonstrate the prediction-guided capability for
planning, we display the planning result with corresponding transformed
occupancy predictions.

Figure 4: An example of refined planning results with corresponding
transformed occupancy prediction for guidance. The AV (red) stops due to the
approaching pedestrians. The prediction results circled in red dots refer to
undetected actors in agent-centric methods.

As illustrated in Fig. 4, the planning rendered in cyan causes the AV (red) to
stop due to the predicted approaching pedestrians. Moreover, as indicated by
the red dotted circle, the occupancy prediction in OPGP can predict undetected
actors that would be missed by agent-wise methods.

###  IV-E Limitations and Future Work

Although the current two-stage OPGP shows promising results, it also raises
some topics that require further improvements. The first issue is the slight
drop in occupancy prediction performance in multi-task learning. This requires
joint interactions from network design to the learning scheme for prediction
and planning. Another issue is the costly tuning for handcrafted weights and
conservative prediction guidance, which forces us to manually define a safety
distance. Occupancy prediction requires additional guidance in eliminating
intractable pixels. Future work will aim to resolve these issues by improving
network structures based on OPGP pipelines.

##  V Conclusions

In this paper, we present a two-stage Occupancy Prediction-Guided Neural
Planner (OPGP) that refines learning-based planning through prediction
guidance in a joint manner. We develop an integrated learning-based framework
with Transformer backbones, designed for comprehensive occupancy predictions
and multi-modal planning objectives. Following the first stage outputs for
prediction and planning, a transformed occupancy-guided optimization, built
upon a curvilinear frame, achieves direct planning refinement through the use
of handcrafted cost function designs. The prediction and planning performance
are extensively validated using large-scale, real-world datasets (WOMD).
Exhibiting robust performance in comparison to vanilla strong prediction
baselines, the planning results demonstrate enhanced safety and driving
smoothness. Furthermore, qualitative results substantiate the effectiveness of
the transformed occupancy prediction guidance, revealing increased scalability
in handling undetected and occluded actors when compared to agent-wise
methods.

## References

  * [1] S. Mozaffari, O. Y. Al-Jarrah, M. Dianati, P. Jennings, and A. Mouzakitis, “Deep learning-based vehicle behavior prediction for autonomous driving applications: A review,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 1, pp. 33–47, 2020.
  * [2] L. Chen, Y. Li, C. Huang, B. Li, Y. Xing, D. Tian, L. Li, Z. Hu, X. Na, Z. Li _et al._ , “Milestones in autonomous driving and intelligent vehicles: Survey of surveys,” _IEEE Transactions on Intelligent Vehicles_ , 2022.
  * [3] C. Badue, R. Guidolini, R. V. Carneiro, P. Azevedo, V. B. Cardoso, A. Forechi, L. Jesus, R. Berriel, T. M. Paixao, F. Mutz _et al._ , “Self-driving cars: A survey,” _Expert Systems with Applications_ , vol. 165, p. 113816, 2021.
  * [4] W. Wang, L. Wang, C. Zhang, C. Liu, L. Sun _et al._ , “Social interactions for autonomous driving: A review and perspectives,” _Foundations and Trends® in Robotics_ , vol. 10, no. 3-4, pp. 198–376, 2022.
  * [5] X. Mo, Y. Xing, and C. Lv, “Interaction-aware trajectory prediction of connected vehicles using cnn-lstm networks,” in _IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society_. IEEE, 2020, pp. 5057–5062.
  * [6] J. Gu, C. Sun, and H. Zhao, “Densetnt: End-to-end trajectory prediction from dense goal sets,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 15 303–15 312.
  * [7] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone, “Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16_. Springer, 2020, pp. 683–700.
  * [8] Z. Huang, H. Liu, and C. Lv, “Gameformer: Game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving,” _arXiv preprint arXiv:2303.05760_ , 2023.
  * [9] R. Mahjourian, J. Kim, Y. Chai, M. Tan, B. Sapp, and D. Anguelov, “Occupancy flow fields for motion forecasting in autonomous driving,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 5639–5646, 2022.
  * [10] J. Kim, R. Mahjourian, S. Ettinger, M. Bansal, B. White, B. Sapp, and D. Anguelov, “Stopnet: Scalable trajectory and occupancy prediction for urban autonomous driving,” _arXiv preprint arXiv:2206.00991_ , 2022.
  * [11] H. Zhou, J. Laval, A. Zhou, Y. Wang, W. Wu, Z. Qing, and S. Peeta, “Review of learning-based longitudinal motion planning for autonomous vehicles: research gaps between self-driving and traffic congestion,” _Transportation research record_ , vol. 2676, no. 1, pp. 324–341, 2022.
  * [12] S. Aradi, “Survey of deep reinforcement learning for motion planning of autonomous vehicles,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 2, pp. 740–759, 2020.
  * [13] H. Liu, Z. Huang, X. Mo, and C. Lv, “Augmenting reinforcement learning with transformer-based scene representation learning for decision-making of autonomous driving,” _arXiv preprint arXiv:2208.12263_ , 2022.
  * [14] J. Ngiam, B. Caine, V. Vasudevan, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal _et al._ , “Scene transformer: A unified architecture for predicting multiple agent trajectories,” _arXiv preprint arXiv:2106.08417_ , 2021.
  * [15] J. L. V. Espinoza, A. Liniger, W. Schwarting, D. Rus, and L. Van Gool, “Deep interactive motion prediction and planning: Playing games with motion prediction models,” in _Learning for Dynamics and Control Conference_. PMLR, 2022, pp. 1006–1019.
  * [16] Z. Huang, H. Liu, J. Wu, and C. Lv, “Differentiable integrated motion prediction and planning with learnable cost function for autonomous driving,” _arXiv preprint arXiv:2207.10422_ , 2022.
  * [17] A. Cui, S. Casas, A. Sadat, R. Liao, and R. Urtasun, “Lookout: Diverse multi-future prediction and planning for self-driving,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 16 107–16 116.
  * [18] H. Liu, Z. Huang, and C. Lv, “Strajnet: Occupancy flow prediction via multi-modal swin transformer,” _arXiv preprint arXiv:2208.00394_ , 2022.
  * [19] M. Werling, J. Ziegler, S. Kammel, and S. Thrun, “Optimal trajectory generation for dynamic street scenarios in a frenet frame,” in _2010 IEEE International Conference on Robotics and Automation_. IEEE, 2010, pp. 987–993.
  * [20] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” _arXiv preprint arXiv:1710.10903_ , 2017.
  * [21] Q. Sun, X. Huang, J. Gu, B. C. Williams, and H. Zhao, “M2i: From factored marginal trajectory prediction to interactive prediction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 6543–6552.
  * [22] X. Mo, Z. Huang, Y. Xing, and C. Lv, “Multi-agent trajectory prediction with heterogeneous edge-enhanced graph attention network,” _IEEE Transactions on Intelligent Transportation Systems_ , 2022.
  * [23] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde, “Gohome: Graph-oriented heatmap output for future motion estimation,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 9107–9114.
  * [24] ——, “Thomas: Trajectory heatmap output with learned multi-agent sampling,” _arXiv preprint arXiv:2110.06607_ , 2021.
  * [25] M. Bansal, A. Krizhevsky, and A. Ogale, “Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst,” _arXiv preprint arXiv:1812.03079_ , 2018.
  * [26] P. Hang, C. Lv, C. Huang, J. Cai, Z. Hu, and Y. Xing, “An integrated framework of decision making and motion planning for autonomous vehicles considering social behaviors,” _IEEE transactions on vehicular technology_ , vol. 69, no. 12, pp. 14 458–14 469, 2020.
  * [27] Z. Huang, J. Wu, and C. Lv, “Driving behavior modeling using naturalistic human driving data with inverse reinforcement learning,” _IEEE Transactions on Intelligent Transportation Systems_ , 2021.
  * [28] D. González, J. Pérez, V. Milanés, and F. Nashashibi, “A review of motion planning techniques for automated vehicles,” _IEEE Transactions on intelligent transportation systems_ , vol. 17, no. 4, pp. 1135–1145, 2015.
  * [29] H. Song, W. Ding, Y. Chen, S. Shen, M. Y. Wang, and Q. Chen, “Pip: Planning-informed trajectory prediction for autonomous driving,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16_. Springer, 2020, pp. 598–614.
  * [30] A. Kamenev, L. Wang, O. B. Bohan, I. Kulkarni, B. Kartal, A. Molchanov, S. Birchfield, D. Nistér, and N. Smolyanskiy, “Predictionnet: Real-time joint probabilistic traffic prediction for planning, control, and simulation,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 8936–8942.
  * [31] S. Casas, A. Sadat, and R. Urtasun, “Mp3: A unified model to map, perceive, predict and plan,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 403–14 412.
  * [32] H. Shao, L. Wang, R. Chen, H. Li, and Y. Liu, “Safety-enhanced autonomous driving using interpretable sensor fusion transformer,” in _Conference on Robot Learning_. PMLR, 2023, pp. 726–737.
  * [33] Z. Huang, X. Mo, and C. Lv, “Recoat: A deep learning-based framework for multi-modal motion prediction in autonomous driving application,” _arXiv preprint arXiv:2207.00726_ , 2022.
  * [34] P. Bender, J. Ziegler, and C. Stiller, “Lanelets: Efficient map representation for autonomous driving,” in _2014 IEEE Intelligent Vehicles Symposium Proceedings_. IEEE, 2014, pp. 420–425.
  * [35] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [36] N. Rhinehart, R. McAllister, and S. Levine, “Deep imitative models for flexible inference, planning, and control,” _arXiv preprint arXiv:1810.06544_ , 2018.
  * [37] M. Bhardwaj, B. Boots, and M. Mukadam, “Differentiable gaussian process motion planning,” in _2020 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2020, pp. 10 598–10 604.
  * [38] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou _et al._ , “Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9710–9719.
  * [39] X. Huang, X. Tian, J. Gu, Q. Sun, and H. Zhao, “Vectorflow: Combining images and vectors for traffic occupancy and flow prediction,” _arXiv preprint arXiv:2208.04530_ , 2022.