11institutetext: Hong Kong Polytechnic University 22institutetext: Joins
Hopkins University
22email: {csrhli<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://github.com/lslrh/DMA

# Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding

Ruihuang Li 11 Zhengqiang Zhang 11 Chenhang He 11 Zhiyuan Ma 11
Vishal M. Patel 22 Lei Zhang(🖄) 11

###### Abstract

Recent vision-language pre-training models have exhibited remarkable
generalization ability in zero-shot recognition tasks. Previous open-
vocabulary 3D scene understanding methods mostly focus on training 3D models
using either image or text supervision while neglecting the collective
strength of all modalities. In this work, we propose a Dense Multimodal
Alignment (DMA) framework to densely co-embed different modalities into a
common space for maximizing their synergistic benefits. Instead of extracting
coarse view- or region-level text prompts, we leverage large vision-language
models to extract complete category information and scalable scene
descriptions to build the text modality, and take image modality as the bridge
to build dense point-pixel-text associations. Besides, in order to enhance the
generalization ability of the 2D model for downstream 3D tasks without
compromising the open-vocabulary capability, we employ a dual-path integration
approach to combine frozen CLIP visual features and learnable mask features.
Extensive experiments show that our DMA method produces highly competitive
open-vocabulary segmentation performance on various indoor and outdoor tasks.

###### Keywords:

3D Scene understanding Open-vocabulary Multimodal alignment

##  1 Introduction

3D scene understanding, which aims to achieve accurate comprehension of
objects as well as their attributes and relationships within a scene, has
gained significant attention in recent years due to its popular applications
in autonomous driving [32], virtual reality (VR) [2, 40, 50] and robot
navigation [3], _etc_. However, the annotation of large-scale 3D data is very
costly [7, 11], impeding the training of generalizable models for open-
vocabulary scene understanding. Though many existing methods [10, 20, 58, 46,
41, 31, 29, 30, 9] have achieved significant advancements in recognizing
closed-set categories for specific tasks, they fail to identify novel
categories and other types of queries [42] without 3D supervision, hindering
the application of existing 3D scene understanding methods to real-world
settings, where the number of possible classes is unlimited.

In contrast to the limited 3D data, modalities such as images and texts are
more abundantly available. Existing pre-trained multimodal models, such as
CLIP [43] and ALIGN [24], have shown impressive zero-shot recognition ability
by training on large-scale noisy image-text pairs, and have been successfully
adapted for open-vocabulary classification [53, 54], detection [38, 5] and
segmentation tasks [33, 52, 47]. Based on these observations, researchers have
attempted to use image or natural language modalities to provide supervisory
signals for learning 3D representations [55, 13, 42, 36]. Some methods use
fixed 2D features as supervision and distill the knowledge from either the
pre-trained 2D encoder of CLIP [36] or 2D open-vocabulary segmentation (OVSeg)
models [42] into 3D representations (NeRF or point clouds). However, they
overlook the fact that 3D models can in turn enhance 2D models by leveraging
the strong 3D structural information. Besides, the 2D OVSeg models compromise
their open-vocabulary ability since they are primarily fine-tuned on in-
vocabulary datasets. There are also some methods that directly align 3D
features to semantic captions [55, 13, 45]. However, they only capture coarse
image- or region-level descriptions without establishing dense point-to-text
correspondences or exploiting image features that involve rich semantic
contexts and more variations. Though some methods [53, 54] simultaneously
leverage visual and textual supervisions, they only conduct coarse multimodal
alignment for object-level point cloud classification.

In order to leverage the synergistic benefits of multiple modalities for dense
prediction tasks, we propose a dense multimodal alignment (DMA) strategy to
co-embed 3D points, image pixels, and text strings into a shared latent space.
To build dense associations across different modalities, the primary
bottleneck is how to obtain rich and reliable text descriptions without
relying on manual labeling. To this end, we generate two types of prompts
using large Vision-Language Models (VLMs). Firstly, we employ the tagging
model such as RAM [57] to detect as many categories as possible from an image,
ensuring alignment with complete semantic patterns. Considering that category
names might not provide sufficient details and contextual information, we
incorporate Multimodal Large Language Models (MLLM) such as LLaVA [35] to
generate linguistically expressible scene descriptions, thereby enhancing the
scalability of text queries. In addition, we use the GPT to filter out the
noise in the generated texts for improving the reliability. As a result, we
establish a highly scalable and informative text modality, enhancing the
overall understanding of 3D scenes.

As for the image modality, we adopt a dual-path integration strategy to
extract robust 2D features as supervision. Specifically, we employ the FC-CLIP
[56] as the feature extractor. On one hand, we fix its CLIP visual encoder to
maintain the open-world recognition ability. On the other hand, by fine-tuning
its mask head, we incorporate 3D structural priors into 2D features, better
adapting the model to 3D dense tasks. Then we build triplets of points,
pixels, and their corresponding texts by taking image modality as the bridge.
Given the generated triplets of different modalities and their dense
correspondences, we finally adopt the mutually inclusive loss function to
align multiple modalities. In this way, we can effectively unleash the
potential of existing foundation VLMs and maximize the complementary effects
of multiple modalities.

In summary, (1) we first present a dense multimodal alignment framework, which
establishes dense correspondences among points, pixels and texts, to learn
robust 3D representations for open-vocabulary 3D scene understanding. (2) To
generate complete and scalable language modality without relying on manual
annotations, we leverage a tagging model and an MLLM to extract category
information and scene descriptions, respectively. (3) Finally, to improve the
segmentation ability without compromising the open-vocabulary ability, we
integrate 3D priors into 2D features by fine-tuning the 2D mask head with the
backbone frozen. Extensive experiments demonstrate the outstanding open-world
3D segmentation ability of our DMA model on various indoor and outdoor tasks.

##  2 Related Work

Open-Vocabulary 3D Scene Understanding. 3D scene understanding is a popular
research topic in computer vision. Most previous methods [10, 15, 19, 20, 58]
focus on training models on manually labeled close-set categories, and have
yielded promising performance on popular 3D benchmarks [11, 4]. However, most
of these methods are designed for a specific task, such as object
classification [51], detection [8], semantic/instance segmentation [58, 10,
15], and they cannot identify novel categories, restricting their applications
to real-world settings. To overcome this limitation, recent works have been
focused on the open-vocabulary scene understanding problem [36]. Rozenberszki
_et al_. [45] proposed a language-driven pre-training method to enforce 3D
feature to be close to text embeddings, and finetune the 3D encoder with
ground-truth annotations. PLA [13] and RegionPLC [55] explicitly associate 3D
points with image- and region-level image captions, respectively. However,
existing image captioning models can only identify sparse and salient objects
while missing other important categories. Besides, textual signals lack
variations and contexts, making them insufficient for dense prediction tasks.
Some methods [42, 36] distill knowledge from large-scale pre-trained 2D
models, such as image-text contrastive learning models [43] and open-
vocabulary segmentation models [33, 52, 14, 56]. However, the performance of
pre-trained models drops a lot on the downstream datasets due to the large
domain shift. These methods also overlook the fact that 3D models can in turn
enhance 2D models by leveraging the strong structural information inherent in
3D data.

Figure 1: Framework of our proposed Dense Multimodal Alignment (DMA) method.
We generate comprehensive language modality data by leveraging a tagging model
and an MLLM. As for 2D modality, we fix the CLIP visual backbone ${\bf
f}^{2D}_{clip}$ but finetune the mask head ${\bf f}^{2D}_{mask}$ for better
adaptation to downstream 3D tasks without compromising the open-vocabulary
ability. Then the dense correspondences between pixels ${\bf f}^{2D}$ and
texts ${\bf f}^{T}_{tag}$/${\bf f}^{T}_{llm}$ can be built by computing their
feature similarities, resulting in semantic score maps
$S^{2D}_{tag}$/$S^{2D}_{llm}$. By taking image modality as the bridge, we
back-project text labels to each point and obtain the 3D label maps
$M^{3D}_{tag}$/$M^{3D}_{llm}$. Finally, we co-embed point ${\bf f}^{3D}$,
pixel ${\bf f}^{2D}$, and text embeddings ${\bf f}^{T}$ into a common space to
learn a robust 3D representation by optimizing the mutually inclusive loss
function.

Vision-Language Foundation Models. Recent vision-language foundation models
have exhibited remarkable generalization ability on zero-shot prediction
tasks. Segment Anything Model (SAM) [26] leads a new trend of universal image
segmentation and exhibits promising results on diverse downstream tasks.
Recognize Anything Model (RAM) [57] presents a novel paradigm for image
tagging (multi-label classification) by leveraging large-scale image-text
pairs for training without manual annotations. The recent success of ChatGPT
and GPT4 have stimulated tremendous interests in developing multimodal large
language models (LLMs). LLaVA [35] is an early exploration to apply LLMs to
the multimodal fields by connecting a vision encoder to LLM for general-
purpose visual and language understanding. The recent open-vocabulary methods
[52, 33, 56] shed lights on the direct use of pre-trained foundation models
for handling different visual tasks. ODISE [52] explores the potential ability
of pretrained text-to-image diffusion models [44] for open-vocabulary panoptic
segmentation. FC-CLIP [56] utilizes a shared frozen convolutional CLIP
backbone to maintain the ability of open-vocabulary classification without
compromising accuracy.

##  3 Method

As illustrated in Fig. 1, we propose a dense multimodal alignment (DMA)
framework for open-vocabulary 3D scene understanding, where we construct dense
correspondences across 2D image pixels, 3D points and 1D texts, and embed them
into a common latent space. In this section, we will elaborate the
construction of text and image modalities, and explain how we associate and
align them in a dense manner.

###  3.1 Comprehensive Text Modality Generation

Learning a robust 3D model that is generalizable to open vocabularies is
challenging since it is unclear how to acquire the dense text labels for point
clouds. Although well-trained human annotators could potentially provide
detailed language descriptions of 3D scenes, such a method is costly and lacks
scalability. To overcome this limitation, we leverage a tagging model and an
MLLM to extract complete category information and scalable scene descriptions,
respectively.

Complete Category Information. The scene tagging process is illustrated in
Fig. 2. Firstly, we use the image tagging foundation model such as RAM [23] to
extract all possible categories from an image, and utilize category names and
short descriptions derived from the metadata as the text query, referred to as
$T_{tag}$, such as “There is a {category name} in the scene”, “A photo of a
{category name}”, _etc_. Unlike image captioning models [28, 1] that can only
identify sparse and salient objects in a scene, RAM can recognize as many tags
as possible without missing important parts, ensuring a high recall rate and
alignment with complete semantic patterns. The more complete and accurate the
detected categories are, the easier we can establish precise dense
correspondences between text and 3D modalities, and hence the open-vocabulary
capability of the 3D model can be enhanced.

Figure 2: Scene tagging generation. (1) We first employ RAM [57] to generate
view-level tags, and then (2) reduce the tag noise with GPT. Finally, scene-
level tags are generated by (3) multi-view voting.

Reliable GPT-based Denoising. While there are many tags recognized by RAM,
some redundant or irrelevant tags are also included, such as non-noun words
(“purple”, “blue”, “lay”, _etc_.) and objects that do not exist in the image
(“bed”, “ukulele”, _etc_.), as shown in Fig. 2. We address this issue in two
steps. Firstly, we utilize GPT to filter potential noisy vocabulary. Given the
input list, we instruct GPT to examine the words one by one and perform
reasoning according to the chain of thought, outputting a boolean list
indicating whether a word is an outlier. Please refer to Fig. 1 of
supplemental material for the detailed instructions and examples to reduce the
noisy tags. Secondly, to decrease the non-existent categories in a scene, we
conduct multi-view voting and neglect categories that appear in fewer than
five views. Please refer to Fig. 2 of the supplemental material for the
denoised scene tagging results and the corresponding visualizations of 3D
label maps.

Scalable Scene Description. Although scene-level tags have already covered
most of the categories, the limited scalability and variation of category
names hinders their provision of rich contexts and details. To address this
limitation and enable arbitrary queries for 3D networks, we additionally
leverage MLLMs such as LLaVA [35] to generate diverse and linguistically
expressible descriptions, denoted by $T_{llm}$. Owing to the exposure to a
diverse range of linguistic patterns and contextual nuances, the MLLMs can
generate comprehensive and in-depth descriptions based on input images.
Consequently, these LLMs can enhance the richness and granularity of the
generated textual representations, thereby facilitating a more comprehensive
understanding of the 3D scenes. Please refer to Fig. 3 of supplemental
material for the examples of scene-level captions and corresponding
visualizations.

Finally, we generate the text embeddings $\textbf{f}^{T}_{tag}$ and
$\textbf{f}^{T}_{llm}$ using CLIP text encoder based on the generated tags
$T_{tag}$ and scene descriptions $T_{llm}$, respectively, which are utilized
to supervise the training of 3D networks subsequently.

Figure 3: Segmentation results using 2D and 3D models. 2D model has advantages
in segmenting background objects (in blue boxes), while 3D model is more
favorable for foreground objects with distinct structures (in red boxes).

###  3.2 Structure-aware Image Feature Extraction

Compared to language modality, the image modality offers a wealth of
contextual information and exhibits significant variations among different
pixels, which could provide more effective supervision. Inspired by this
observation, OpenScene [42] distills the knowledge from frozen open-vocabulary
2D segmentation models, such as LSeg [27] and OpenSeg [14]. However, these
methods suffer from two major limitations. Firstly, they are fine-tuned on in-
vocabulary datasets, which leads to a misalignment between image and text
features and consequently results in poor performance on open-vocabulary
categories. Secondly, all of these methods freeze 2D networks, failing to
perceive the 3D structure of objects and leading to inaccurate supervision. As
shown in Fig. 3, we visualize the segmentation results using 2D and 3D
features. One can observe that although 2D features are more advantageous in
segmenting background objects with ambiguous geometry, such as “bookshelf”,
“door” and “blackboard”, they are less effective in segmenting objects with
distinct shapes, such as “table” and “chair”. Therefore, it is necessary to
distill the structural priors of 3D networks into 2D ones as well in order to
facilitate fine-grained scene understanding.

In this paper, we adopt FC-CLIP [56] as the backbone to extract image
features. On one hand, we use the frozen CLIP visual encoder to ensure the
intactness of image-text alignment, obtaining CLIP features
$\textbf{f}^{2D}_{clip}$. On the other hand, to facilitate the synergistic
benefits of both 2D and 3D modalities, we fine-tune the mask head and attain
the mask features $\textbf{f}^{2D}_{mask}$. In contrast to previous methods
that rely on potentially noisy fixed image features for supervision, the fine-
tuned mask features enhance the adaptability to downstream 3D tasks. We
explore different fine-tuning strategies, such as LoRA [18], Adapter [17], and
full parameter fine-tuning and compare them in experiments.

###  3.3 Dense Associations across Modalities

Once the text and image modalities are constructed, the subsequent step is to
associate each point to its corresponding pixel and text. We utilize the image
modality as a bridge to establish separate associations between pixels and
other modalities. Firstly, we construct the associations between image and
language modalities by taking $C$ different text embeddings
$\textbf{f}^{T}=\\{\textbf{f}^{T}_{1},\cdots,\textbf{f}^{T}_{C}\\}$ as
classifier to assign text labels to each pixel, obtaining a 2D semantic score
map, denoted by $S^{2D}\in\mathbb{R}^{H\times W\times C}$. This process can be
formulated as follows:

$\displaystyle
S^{2D}_{c}(u,v)=\sigma(<\textbf{f}^{2D}(u,v),\textbf{f}^{T}_{c}>/\tau_{1}),$
(1)

where $S^{2D}_{c}(u,v)$ denotes the probability that the pixel at location
$(u,v)$ belongs to the $c$-th text label, and $<\cdot>$ represents the cosine
similarity between two $\ell_{2}$-normalized feature vectors. $\tau_{1}$ is a
temperature parameter. Then we establish the associations between images and
point clouds by back-propagating 3D points $\textbf{p}=(x,y,z)$ onto 2D
positions $(u,v)$ using a projection matrix $T\in\mathbb{R}^{3\times 4}$,
_i.e_., $[u,v,w]=T[x,y,z,1]$, where $w$ is a scaling factor.

Finally, we associate the text and 3D modalities by taking image as the
bridge. Given $K$ different projection views for one point p, we compute its
average semantic score, denoted as $\bar{S}^{3D}$, across $K$ views
$[S^{2D}_{1}(u_{1},v_{1}),\cdots,S^{2D}_{K}(u_{K},v_{K})]$. Based on the
aggregated 3D semantic scores, the final text-to-3D label map, denoted as
$M^{3D}\in\mathbb{R}^{N\times C}$, can be derived by:

$\displaystyle M^{3D}_{c}=\left\\{\begin{matrix}1,&\text{if}\ \
\bar{S}^{3D}_{c}>\text{threshold}\\\ 0,&else\end{matrix}\right.,$ (2)

where $N$ denotes the number of points and $M^{3D}_{c}$ indicates whether the
point belongs to the $c$-th text label or not. $M^{3D}$ can be regarded as the
pseudo label map for point cloud, serving as the supervision signal for
training 3D models.

It is noteworthy that instead of generating one-hot label through the argmax
operation, we select all confident text labels whose scores exceed the
threshold. This is because the generated text categories may exhibit
similarities in semantics (like ‘suitcase’ and ‘luggage’) or inclusion
relationships (such as ‘kitchen’ and ‘stove’). As a result, it is highly
possible that one single point corresponds to multiple text labels
simultaneously.

###  3.4 Dense Multimodal Alignment

After obtaining the triplets of different modalities and their dense
correspondences, the subsequent objective is to align the 3D points with their
corresponding text and pixel embeddings. This alignment process involves
several steps. Firstly, we extract 3D features for the point cloud by
utilizing a 3D network, denoted as $\varepsilon_{3D}$. These features are then
projected to match the dimension of the CLIP features.

Next, we assign text labels to different 3D points by computing the cosine
similarities between point and text embeddings $\textbf{f}^{T}$, yielding a 3D
segmentation probability map $P^{3D}$:

$\displaystyle
P^{3D}_{i,c}=\sigma(<\textbf{f}^{3D}_{i},\textbf{f}^{T}_{c}>/\tau_{2}),$ (3)

where $\textbf{f}^{3D}_{i}$ denotes the feature of the $i$-th point, and
$P^{3D}_{i,c}$ denotes the probability that the $i$-th point belongs to the
$c$-th text label. Here we employ the Sigmoid activation function
$\sigma(\cdot)$ since it will not lead to mutually exclusive relationships
among different categories.

Text-to-3D Supervision. We use the text-to-3D label map $M^{3D}$ as the pseudo
label to facilitate the alignment of point and text features. Different loss
functions are employed for aligning the point embeddings with the tag ${\bf
f}_{tag}^{T}$ and scene description ${\bf f}_{llm}^{T}$ embeddings. As can be
seen in Fig. 1, we build dense associations between ${\bf f}_{tag}^{T}$ and
the entire point cloud, resulting in $M_{tag}^{3D}$. Consequently, The Binary
Cross Entropy (BCE) loss is used to effectively penalize both positive and
negative samples:

$\displaystyle\mathcal{L}_{3d-text(tag)}=\mathcal{L}_{BCE}(P^{3D},M_{tag}^{3D}).$
(4)

As for ${\bf f}_{llm}^{T}$, since it corresponds only to salient objects, we
can only obtain the mask for partial points, denoted as $M_{llm}^{3D}$.
(Visualizations of $M_{tag}^{3D}$ and $M_{llm}^{3D}$ are given in Fig. 2 and
Fig. 3 of the supplementary material, respectively.) We utilize the cosine
similarity loss to supervise only the positive samples:

$\displaystyle\mathcal{L}_{3d-text(llm)}=1-\cos({\bf
f}_{llm}^{T},\textbf{f}^{3D}).$ (5)

Mutually Inclusive Loss. In this work, we do not employ the Cross-Entropy loss
because it would result in a mutually exclusive relationship between different
classes, meaning that each point is assigned to only one class of interest.
However, in text-to-3D alignment, one point may simultaneously associate with
multiple text prompts, such as ‘bed’ and ‘bedroom’, ‘chair’ and ‘office
chair’, ‘curtain’ and ‘drape’, _etc_. To handle this issue, we employ mutually
inclusive losses (MIL), such as BCE loss and cosine similarity loss, to ensure
that each point is aligned with all its corresponding tags/descriptions
simultaneously, avoiding the potential conflicts between categories with
overlapping or similar semantics.

2D-to-3D Supervision. For 3D-2D pairs, we follow the previous work [42] to
fuse pixel embeddings across $K$ different views, represented as
$[\textbf{f}^{2D}_{1},\cdots,\textbf{f}^{2D}_{K}]$, into a single feature
vector $\bar{\textbf{f}}^{2D}$, and align 2D and 3D features by minimizing the
cosine similarity loss:

$\displaystyle\mathcal{L}_{3d-2d}=1-\cos(\bar{\textbf{f}}^{2D},\textbf{f}^{3D}).$
(6)

Since 2D mask head is also trainable, we additionally add the text-to-2D
supervision and compute the BCE loss between 2D predictions and 2D masks,
obtaining $\mathcal{L}_{text-2d}$. Finally, the overall objective function to
perform dense multimodal alignment is defined as:

$\displaystyle\mathcal{L}_{3D}=\mathcal{L}_{3d-text(tag)}+\mathcal{L}_{3d-text(llm)}+\mathcal{L}_{3d-2d}+\mathcal{L}_{text-2d},$
(7)

where the language modality provides comprehensive textual descriptions, and
the image modality gives precise supervision on object edges and contextual
information. Additionally, the 3D modality reveals crucial structural
information of objects. By densely aligning these modalities in a shared
space, our method can maximize the synergistic benefits among them and achieve
outstanding segmentation performance without compromising the open-vocabulary
classification ability of the model.

##  4 Experiments

###  4.1 Setups

Datasets. To demonstrate the effectiveness of our proposed method, we employ
three popular datasets,  _i.e_., ScanNet [11], Matterport3D [6], and nuScenes
[4]. The first two datasets are indoor ones, comprising RGBD images and 3D
meshes. The third one is an outdoor dataset, consisting of data collected from
two sensors,  _i.e_., LiDAR and camera. We conduct comparisons with state-of-
the-art methods on each of these datasets. The mean Intersection-of-Union
(mIoU), mean Accuracy (mACC), Precision, and Recall are employed as the
evaluation metrics.

Implementation Details. In this work, MinkowskiNet [10] is employed as the 3D
backbone, whose voxel size is set to 2cm for ScanNet and Matterport3D and 5cm
for nuScenes. As for the 2D backbone, we use OpenSeg [14] and FC-CLIP [56]
that perform mask-wise classification. The parameters $\tau_{1}$ in Eq. 1 and
$\tau_{2}$ in Eq. 3 are both set to 0.1. We use Adam [25] as the optimizer and
the initial learning rate is set to $1e-4$. The model is trained for 100
epochs. We set the batch size as 8 for indoor datasets and use one single
NVIDIA RTX A6000 for training. As for nuScenes dataset, we use 8 GPUs for
training and set the batch size as 16.

| Methods | mIoU | mACC | mIoU(F) | mIoU(B) | Latency
---|---|---|---|---|---|---
fully-supervised | TangentConv [49] | 40.9 | $-$ | $-$ | $-$ | $-$
TextureNet [21] | 54.8 | $-$ | $-$ | $-$ | $-$
ScanComplete [12] | 56.6 | $-$ | $-$ | $-$ | $-$
Mix3D [41] | 73.6 | $-$ | $-$ | $-$ | $-$
VMNet [20] | 73.2 | $-$ | $-$ | $-$ | $-$
MinkowskiNet [20] | 69.0 | $-$ | $-$ | $-$ | $-$
Zero-shot | PLA [13] | 17.7 | 33.5 | $-$ | $-$ | 0.07s
RegionPLC [55] | 43.8 | 65.6 | $-$ | $-$ | 0.07s
2-7 | OpenScene [42](LSeg)-3D | 52.9 | 63.2 | $-$ | $-$ | 0.07s
| OpenScene(LSeg)-2D3D | 54.2 | 66.6 | $-$ | $-$ | 102.6s
| OpenScene†(OpenSeg)-3D | 46.6 | 66.5 | 50.0 | 47.1 | 0.07s
| OpenScene†(OpenSeg)-2D3D | 47.9 | 71.7 | 49.5 | 51.0 | 89.4s
2-7 | DMA(OpenSeg)-text only | 50.5 | 63.7 | 56.7 | 48.0 | 0.07s
| DMA(OpenSeg)-3D | 53.3 | 70.3 | 58.3 | 51.5 | 0.07s
| DMA(LSeg)-3D | 54.8 | 66.9 | 59.9 | 51.9 | 0.07s
| DMA(FC-CLIP)-3D | 51.8 | 68.7 | 56.0 | 51.4 | 0.07s

Table 1: Comparison on the ScanNet [11] validation set. “F” and “B” denote
foreground and background classes, respectively. † denotes our reproduced
results.

| Methods |  Anno. | mIoU |  mIoU (Base) |  mIoU (Long-Tail)
---|---|---|---|---|---
fully-sup. | RangeNet++ [39] |  100% | 65.5 |  76.4 |  56.4
Cylinder3D [58] | 75.4 |  84.1 |  69.4
SPVNAS [48] | 74.8 |  82.3 |  67.2
AMVNet [34] | 77.0 |  83.9 |  70.8
weakly-sup. | ContrastiveSC [16] |  0.9% | 64.5 |  79.7 |  53.8
LESS [37] | 74.8 |  81.6 |  68.7
2-6 | ContrastiveSC |  0.2% | 63.5 |  78.4 |  51.6
| LESS | 73.5 |  81.1 |  66.6
Zero-shot | OpenScene [42](LSeg)-2D3D |  No | 36.7 |  55.0 |  22.3
OpenScene(OpenSeg)-2D3D | 42.1 |  52.6 |  33.8
2-6 | DMA(OpenSeg)-3D |  No | 45.1 |  59.3 |  33.9
| DMA(FC-CLIP)-3D | 47.4 |  61.4 |  35.3

Table 2: Comparison on the nuScenes [4] validation set. We partition all
categories into base and long-tail classes according to their frequencies.

###  4.2 Comparison with State-of-the-Arts

We compare the proposed DMA with fully-/weakly-supervised and zero-shot
methods [42, 13, 55]. Tab. 2 presents the segmentation results on the ScanNet
[11] dataset. To facilitate comparison, we measure the results of OpenScene by
using 3D and 2D-3D integrated features as supervision. As can be seen in Tab.
2, although OpenScene(LSeg) attains better results (54.2% mIoU) by using both
2D and 3D encoders, it results in significantly increased inference latency.
This is because the parameter size of 2D encoder is much larger than 3D
encoder, and the 2D encoder needs to perform inference on multi-view images of
the scene. Our DMA(OpenSeg) using only 3D model for prediction outperforms
OpenScene(OpenSeg)-2D3D by 5.4% mIoU at a significantly lower latency, wherein
the mIoU (F) and mIoU (B) are improved by 8.8% and 0.5%, respectively. This is
because we perform additional alignment with text modality, thereby
compensating for the decreased open-vocabulary ability of 2D model. When using
text supervision only, our method outperforms the text-supervised approach
RegionPLC [55] by 9.5%, and even surpasses OpenScene(OpenSeg)-2D3D by 2.6% in
terms of mIoU. This indicates that, compared to previous methods that generate
image- and region-level captions, our method establishes dense and precise
correspondences between text and 3D points by taking 2D modality as the
bridge, achieving more precise supervision. The suboptimal performance of our
method using FC-CLIP as the 2D encoder may be attributed to the low resolution
of the images (320$\times$240), which limits the capabilities of FC-CLIP.

Figure 4: Qualitative results of different methods on both indoor and outdoor
datasets.

Outdoor Scenes. To validate the effectiveness of our method on outdoor point
clouds, we evaluate the performance of DMA on the nuScenes dataset [4]. Due to
the highly imbalanced class distribution of outdoor scenes, we additionally
measure the performance on base and long-tail categories. As shown in Tab. 2,
by densely aligning with the tagging information and the detailed description
extracted from each scene, our DMA(OpenSeg) using only 3D encoder
significantly improves the performance over OpenScene(OpenSeg)-2D3D by 3.0%
mIoU. Additionally, the final performance is further improved by 2.3% and
attains 47.4% mIoU by employing FC-CLIP [56] to extract 2D features. This is
because FC-CLIP could achieve more precise segmentation while maintaining
outstanding open-vocabulary recognition ability of CLIP. Besides, by fine-
tuning the mask head, FC-CLIP could incorporate the 3D structural priors into
mask features and produce better results.

| mIoU | mACC | Precision | Recall
---|---|---|---|---
| Head | Common | Tail | All | Head | Common | Tail | All | Head | Common | Tail | All | Head | Common | Tail | All
OpenScene†[42]-2D3D | 21.2 | 8.4 | 4.0 | 6.2 | 35.5 | 15.7 | 9.4 | 12.0 | 43.9 | 18.6 | 7.8 | 14.5 | 34.5 | 15.7 | 9.4 | 12.0
PLA [13] | $-$ | $-$ | $-$ | 1.8 | $-$ | $-$ | $-$ | 3.1 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
RegionPLC [55] | $-$ | $-$ | $-$ | 6.5 | $-$ | $-$ | $-$ | 15.9 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
1-17 DMA-text only | 23.2 | 7.6 | 2.0 | 6.9 | 32.6 | 13.4 | 5.9 | 11.3 | 40.0 | 15.1 | 5.8 | 13.7 | 32.6 | 13.4 | 5.9 | 11.3
DMA(OpenSeg)-3D | 25.3 | 10.8 | 5.5 | 7.6 | 36.7 | 18.2 | 10.7 | 14.6 | 44.5 | 23.6 | 10.5 | 14.9 | 36.7 | 18.2 | 10.7 | 13.2
DMA(FC-CLIP)-3D | 27.2 | 11.5 | 5.8 | 7.9 | 37.4 | 19.2 | 11.2 | 15.2 | 46.2 | 24.9 | 11.3 | 15.7 | 38.2 | 20.4 | 11.1 | 14.0
Fully-Sup | 45.5 | 13.6 | 3.4 | 20.8 | $-$ | $-$ | $-$ | $-$ | 66.8 | 55.7 | 23.3 | 34.4 | 57.6 | 19.1 | 5.8 | 27.8

Table 3: Comparison on ScanNet200 [45] validation set. ${\dagger}$ means our
reproduced results.

Long-Tail Datasets. As shown in Tab. 3, we validate the open-vocabulary
methods on the more challenging long-tail 3D scene understanding datasets,
_i.e_., ScanNet200 [45]. Following [45], we partition the 200 categories into
three splits, _i.e_., head, common, and tail sets, facilitating a more
comprehensive comparison across categories with different frequencies. When
training on head classes (ceiling, curtain, window, _etc_.), the fully-
supervised method performs much better than zero-shot methods due to the
sufficient 3D labels for supervision. However, on the common and tail splits,
our DMA method approaches to or even surpasses the fully-supervised
competitors. This is because there are only a few instances available on these
long-tail categories, which is not sufficient to train a robust model from
scratch. Our method does not rely on ground truth 3D labels but instead
distill knowledge from pretrained vision-language models, thus it is more
robust to rare objects.

To further validate the robustness of our method on rare objects/classes, we
evaluate on the most frequent $K$ classes of Matterport3D [6], where
$K=21,40,80,160$. We train a 3D model by taking our generated textual
descriptions and image features as supervision, and perform inference on
different $K$ categories. As shown in Tab. 4, when employing the same 2D
network, _i.e_., OpenSeg, our method demonstrates superior zero-shot
segmentation capability on both common and rare categories. Specifically, our
DMA(OpenSeg)-3D surpasses OpenScene(OpenSeg)-3D by 3.8%, 4.5%, 1.6%, and 1.2%
in terms of mIoU at different $K$. This can be attributed to that OpenScene
heavily relies on 2D model for supervision without aligning with text prompts,
which limits its open-vocabulary ability. Our method, however, directly aligns
with the textual modality, overcoming the limitations of 2D models.

|  | mIoU | mACC
---|---|---|---
| # of classes K | 21 | 40 | 80 | 160 | 21 | 40 | 80 | 160
fully-sup. | TangentConv [49] | $-$ | $-$ | $-$ | $-$ | 46.8 | $-$ | $-$ | $-$
TextureNet [21] | $-$ | $-$ | $-$ | $-$ | 63.0 | $-$ | $-$ | $-$
ScanComplete [12] | $-$ | $-$ | $-$ | $-$ | 44.9 | $-$ | $-$ | $-$
DCM-Net | $-$ | $-$ | $-$ | $-$ | 66.2 | $-$ | $-$ | $-$
VM-Net [20] | $-$ | $-$ | $-$ | $-$ | 67.2 | $-$ | $-$ | $-$
MinkowskiNet [20] | 54.2 | $-$ | $-$ | $-$ | 64.6 | $-$ | $-$ | $-$
Zero-shot. | OpenScene [42](LSeg)-3D | 41.9 | 25.4 | 12.0 | 5.9 | 51.2 | 30.7 | 15.2 | 7.5
OpenScene(LSeg)-2D3D | 43.4 | 26.8 | 13.1 | 6.4 | 53.5 | 33.0 | 17.4 | 8.6
OpenScene(OpenSeg)-3D | 41.3 | 33.4 | 18.1 | 8.2 | 55.1 | 46.7 | 26.2 | 13.9
OpenScene(OpenSeg)-2D3D | 42.6 | 34.2 | 18.8 | 8.4 | 59.2 | 47.5 | 27.1 | 14.5
2-10 | DMA-text only | 39.8 | 25.4 | 11.7 | 6.2 | 49.5 | 31.6 | 16.1 | 8.0
| DMA(OpenSeg)-3D | 45.1 | 37.9 | 19.7 | 9.4 | 57.6 | 47.7 | 26.7 | 14.1
| DMA(FC-CLIP)-3D | 46.2 | 38.4 | 20.1 | 9.8 | 58.4 | 48.3 | 26.5 | 15.2

Table 4: Comparison on the Matterport [6] test set.

Qualitative Comparison. Fig. 4 visualizes the segmentation results of
different methods. We can observe that OpenScene [42] with only 3D encoder
exhibits poor performance in segmenting objects that lack spatial structures,
such as “door”, “window”, “counter”, _etc_. In contrast, text supervision
offers more refined guidance by establishing dense correspondences between the
texts and points, thereby enabling more precise alignment. Our approach
leverages the advantages of both language and 2D modalities, and achieves
excellent segmentation results for both foreground and background classes
using only the 3D model.

###  4.3 Ablation Study

2D Features _vs_. 3D Features. In Fig 6, we compare the segmentation
performance on ScanNet by using different features. ‘F’ and ‘B’ denote
foreground and background classes, respectively. For OpenScene [42], we
observe that its 2D features are more advantageous for segmenting background
categories with ambiguous geometry than 3D ones, _i.e_., 49.1% _vs_. 47.1%
mIoU(B), while 3D features excel at segmenting the foreground objects with
distinct shapes, _i.e_., 50.0% _vs_. 42.5% mIoU(F). Although the 2D-3D hybrid
feature can leverage the strengths of both features simultaneously, utilizing
2D models for inference introduces significant computational overhead (please
refer to the latency in Tab. 2). By additionally aligning with our generated
text modality, our method can achieve outstanding performance on both
foreground (58.3%) and background (51.5%) categories using only 3D features.
Besides, DMA achieves comparable performance to using both 2D and 3D encoders
by solely utilizing the 3D encoder, _i.e_., 53.3% _vs_. 53.5% mIoU(F), and
hence significantly reducing inference time.

Tagging Models _vs_. MLLMs. In Tab. 6, we compare the results of using
different tagging models and MLLMs on ScanNet. For the enhanced version, we
replace RAM with RAM++ [22], and LLaVA-7B with LLaVA-13B. We can observe that
our method outperforms RegionPLC [55] by a large margin (about 6.7%) by
building dense point-to-text correspondences. The tagging model plays a key
role for performance improvement since it encompasses extensive semantic
patterns, while MLLM further enhances the final performance by incorporating
rich contextual information. By filtering out noisy tags with GPT, the
performance can be improved by 2.6% and 1.3% for the basic and the enhanced
versions, respectively. The final performance can be further improved when
stronger tagging models/MLLMs are employed.

Figure 5: Comparisons of text, 2D, and 3D features. “F” and “B” denote
foreground and background classes.

Figure 6: Comparisons of tagging models and MLLMs.

Method | mIoU | mACC | mIoU (In) | mIoU (Out)
---|---|---|---|---
CLIP feature | 35.2 | 51.3 | 36.7 | 31.8
Mask feature (w/o FT) | 40.1 | 55.4 | 48.3 | 21.3
Mask feature (w/ FT) | 42.0 | 57.4 | 50.5 | 24.1
CLIP+Mask | 44.8 | 59.7 | 51.7 | 28.5

Table 5: Comparisons of CLIP and Mask features of FC-CLIP on ScanNet. “FT”
denotes fine-tuning.

Method | 2D Mask | 3D Mask
---|---|---
w/o FT | 36.6 | 40.1
Full Parameter | 40.4 | 42.0
LoRA [18] | 39.0 | 41.3
Adapter [17] | 37.9 | 40.9

Table 6: Comparisons of different fine-tuning methods.

CLIP Features _vs_. Mask Features.  In addition to OpenSeg [14], we employ FC-
CLIP [56] to extract 2D features due to its effectiveness. As show in Tab. 6,
we compare the performance by using CLIP and Mask features as supervision.
‘In’ and ‘Out’ denote in-vocabulary and out-vocabulary classes, respectively.
FC-CLIP contains an in-vocabulary classifier and an out-vocabulary classifier,
which correspond to the seen and unseen categories in the training process,
respectively. As can be seen, the fixed CLIP feature is more advantageous in
segmenting unseen categories, which outperforms mask feature by 10.5% in terms
of mIoU(Out). This demonstrates that the fixed CLIP visual encoder could
maintain the strong generalization ability on novel classes. While for in-
vocabulary classes, mask features outperform the CLIP feature by 11.6%. By
combining these features, we can simultaneously achieve competitive results on
in- and out-vocabulary categories, attaining 44.8% mIoU over all classes.

Comparisons of Different Fine-Tuning Methods. We fine-tune the mask head of
FC-CLIP with different strategies for incorporating 3D structural priors into
mask features. As can be seen in Tab. 6, by fully fine-tuning the mask head,
the performances of 2D and 3D masks are improved by 3.8% and 1.9%,
respectively. LoRA [18] and Adapter [17] can also achieve obvious improvements
by tuning a small amount of parameters.

Figure 7: Open-vocabulary segmentation results on rare categories and
different forms of queries. The same color corresponds to the same
query/category.

Open-Vocabulary Segmentation for Different Text Queries. We finally
investigate the ability of our method to segment rare categories. As shown in
Fig. 7, our method can accurately segment the corresponding regions for the
given texts/queries in 3D scenes, even for unseen categories. For instance,
our well-trained model can quickly locate the position of new categories such
as “Snoopy”, or functional areas such as “kitchen”, _etc_. On one hand, we
align with 2D CLIP features that have been trained with a vast corpus of text.
On the other hand, we construct a comprehensive and scalable textual modality
by using VLMs, further enhancing the understanding ability.

##  5 Conclusion

We presented a dense multimodal alignment (DMA) framework for open-vocabulary
3D scene understanding by establishing dense correspondences between 3D
points, 2D images and 1D texts, and leveraging their synergistic benefits to
learn robust and generalizable 3D representations. To build a scalable
language modality, we utilized powerful vision-language models to extract
comprehensive scene descriptions and category information. Furthermore, we
preserved the open-vocabulary recognition ability of the image modality by
combining frozen CLIP features with trainable mask features. Extensive
experiments demonstrate the promising performance of our method in open-
vocabulary segmentation tasks across various indoor and outdoor scenarios.

Limitations. Our method relies on the quality of generated text descriptions
and image features. In addition, collecting a larger 3D scene dataset is
crucial for improving the generalization ability to unseen categories and
variations.

## References

  * [1] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35, 23716–23736 (2022)
  * [2] Armeni, I., Sener, O., Zamir, A.R., Jiang, H., Brilakis, I., Fischer, M., Savarese, S.: 3d semantic parsing of large-scale indoor spaces. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1534–1543 (2016)
  * [3] Cadena, C., Dick, A.R., Reid, I.D.: Multi-modal auto-encoders as joint estimators for robotics scene understanding. In: Robotics: Science and systems. vol. 5 (2016)
  * [4] Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11621–11631 (2020)
  * [5] Cao, Y., Yihan, Z., Xu, H., Xu, D.: Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. Advances in Neural Information Processing Systems 36 (2024)
  * [6] Chang, A., Dai, A., Funkhouser, T., Halber, M., Niebner, M., Savva, M., Song, S., Zeng, A., Zhang, Y.: Matterport3d: Learning from rgb-d data in indoor environments. In: 2017 International Conference on 3D Vision (3DV). pp. 667–676. IEEE Computer Society (2017)
  * [7] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al.: Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)
  * [8] Chen, D.Z., Chang, A.X., Nießner, M.: Scanrefer: 3d object localization in rgb-d scans using natural language. In: European conference on computer vision. pp. 202–221. Springer (2020)
  * [9] Chen, L., Lei, C., Li, R., Li, S., Zhang, Z., Zhang, L.: Fpr: False positive rectification for weakly supervised semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1108–1118 (2023)
  * [10] Choy, C., Gwak, J., Savarese, S.: 4d spatio-temporal convnets: Minkowski convolutional neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3075–3084 (2019)
  * [11] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: Scannet: Richly-annotated 3d reconstructions of indoor scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5828–5839 (2017)
  * [12] Dai, A., Ritchie, D., Bokeloh, M., Reed, S., Sturm, J., Nießner, M.: Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4578–4587 (2018)
  * [13] Ding, R., Yang, J., Xue, C., Zhang, W., Bai, S., Qi, X.: Pla: Language-driven open-vocabulary 3d scene understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7010–7019 (2023)
  * [14] Ghiasi, G., Gu, X., Cui, Y., Lin, T.Y.: Scaling open-vocabulary image segmentation with image-level labels. In: European Conference on Computer Vision. pp. 540–557. Springer (2022)
  * [15] Han, L., Zheng, T., Xu, L., Fang, L.: Occuseg: Occupancy-aware 3d instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2940–2949 (2020)
  * [16] Hou, J., Graham, B., Nießner, M., Xie, S.: Exploring data-efficient 3d scene understanding with contrastive scene contexts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15587–15597 (2021)
  * [17] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: International Conference on Machine Learning. pp. 2790–2799. PMLR (2019)
  * [18] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
  * [19] Hu, W., Zhao, H., Jiang, L., Jia, J., Wong, T.T.: Bidirectional projection network for cross dimension scene understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14373–14382 (2021)
  * [20] Hu, Z., Bai, X., Shang, J., Zhang, R., Dong, J., Wang, X., Sun, G., Fu, H., Tai, C.L.: Vmnet: Voxel-mesh network for geodesic-aware 3d semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15488–15498 (2021)
  * [21] Huang, J., Zhang, H., Yi, L., Funkhouser, T., Nießner, M., Guibas, L.J.: Texturenet: Consistent local parametrizations for learning from high-resolution signals on meshes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4440–4449 (2019)
  * [22] Huang, X., Huang, Y.J., Zhang, Y., Tian, W., Feng, R., Zhang, Y., Xie, Y., Li, Y., Zhang, L.: Open-set image tagging with multi-grained text supervision. arXiv e-prints pp. arXiv–2310 (2023)
  * [23] Huang, X., Zhang, Y., Ma, J., Tian, W., Feng, R., Zhang, Y., Li, Y., Guo, Y., Zhang, L.: Tag2text: Guiding vision-language model via image tagging. arXiv preprint arXiv:2303.05657 (2023)
  * [24] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International conference on machine learning. pp. 4904–4916. PMLR (2021)
  * [25] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [26] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollar, P., Girshick, R.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 4015–4026 (October 2023)
  * [27] Li, B., Weinberger, K.Q., Belongie, S., Koltun, V., Ranftl, R.: Language-driven semantic segmentation. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=RriDjddCLN
  * [28] Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)
  * [29] Li, R., He, C., Li, S., Zhang, Y., Zhang, L.: Dynamask: dynamic mask selection for instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11279–11288 (2023)
  * [30] Li, R., He, C., Zhang, Y., Li, S., Chen, L., Zhang, L.: Sim: Semantic-aware instance mask generation for box-supervised instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7193–7203 (2023)
  * [31] Li, R., Li, S., He, C., Zhang, Y., Jia, X., Zhang, L.: Class-balanced pixel-level self-labeling for domain adaptive semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11593–11603 (2022)
  * [32] Li, Y., Yu, A.W., Meng, T., Caine, B., Ngiam, J., Peng, D., Shen, J., Lu, Y., Zhou, D., Le, Q.V., et al.: Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17182–17191 (2022)
  * [33] Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., Marculescu, D.: Open-vocabulary semantic segmentation with mask-adapted clip. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7061–7070 (2023)
  * [34] Liong, V.E., Nguyen, T.N.T., Widjaja, S., Sharma, D., Chong, Z.J.: Amvnet: Assertion-based multi-view fusion network for lidar semantic segmentation. arXiv preprint arXiv:2012.04934 (2020)
  * [35] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023)
  * [36] Liu, K., Zhan, F., Zhang, J., Xu, M., Yu, Y., Saddik, A.E., Theobalt, C., Xing, E., Lu, S.: Weakly supervised 3d open-vocabulary segmentation. arXiv preprint arXiv:2305.14093 (2023)
  * [37] Liu, M., Zhou, Y., Qi, C.R., Gong, B., Su, H., Anguelov, D.: Less: Label-efficient semantic segmentation for lidar point clouds. In: European Conference on Computer Vision. pp. 70–89. Springer (2022)
  * [38] Lu, Y., Xu, C., Wei, X., Xie, X., Tomizuka, M., Keutzer, K., Zhang, S.: Open-vocabulary point-cloud object detection without 3d annotation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1190–1199 (2023)
  * [39] Milioto, A., Vizzo, I., Behley, J., Stachniss, C.: Rangenet++: Fast and accurate lidar semantic segmentation. In: 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS). pp. 4213–4220. IEEE (2019)
  * [40] Misra, I., Girdhar, R., Joulin, A.: An end-to-end transformer model for 3d object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2906–2917 (2021)
  * [41] Nekrasov, A., Schult, J., Litany, O., Leibe, B., Engelmann, F.: Mix3d: Out-of-context data augmentation for 3d scenes. In: 2021 International Conference on 3D Vision (3DV). pp. 116–125. IEEE (2021)
  * [42] Peng, S., Genova, K., Jiang, C., Tagliasacchi, A., Pollefeys, M., Funkhouser, T., et al.: Openscene: 3d scene understanding with open vocabularies. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 815–824 (2023)
  * [43] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [44] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [45] Rozenberszki, D., Litany, O., Dai, A.: Language-grounded indoor 3d semantic segmentation in the wild. In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)
  * [46] Schult, J., Engelmann, F., Kontogianni, T., Leibe, B.: Dualconvmesh-net: Joint geodesic and euclidean convolutions on 3d meshes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8612–8622 (2020)
  * [47] Takmaz, A., Fedele, E., Sumner, R.W., Pollefeys, M., Tombari, F., Engelmann, F.: Openmask3d: Open-vocabulary 3d instance segmentation. arXiv preprint arXiv:2306.13631 (2023)
  * [48] Tang, H., Liu, Z., Zhao, S., Lin, Y., Lin, J., Wang, H., Han, S.: Searching efficient 3d architectures with sparse point-voxel convolution. In: European conference on computer vision. pp. 685–702. Springer (2020)
  * [49] Tatarchenko, M., Park, J., Koltun, V., Zhou, Q.Y.: Tangent convolutions for dense prediction in 3d. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3887–3896 (2018)
  * [50] Vu, T., Kim, K., Luu, T.M., Nguyen, T., Yoo, C.D.: Softgroup for 3d instance segmentation on point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2708–2717 (2022)
  * [51] Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J.: 3d shapenets: A deep representation for volumetric shapes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1912–1920 (2015)
  * [52] Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., De Mello, S.: Open-vocabulary panoptic segmentation with text-to-image diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2955–2966 (2023)
  * [53] Xue, L., Gao, M., Xing, C., Martín-Martín, R., Wu, J., Xiong, C., Xu, R., Niebles, J.C., Savarese, S.: Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1179–1189 (2023)
  * [54] Xue, L., Yu, N., Zhang, S., Li, J., Martín-Martín, R., Wu, J., Xiong, C., Xu, R., Niebles, J.C., Savarese, S.: Ulip-2: Towards scalable multimodal pre-training for 3d understanding. arXiv preprint arXiv:2305.08275 (2023)
  * [55] Yang, J., Ding, R., Wang, Z., Qi, X.: Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding. arXiv preprint arXiv:2304.00962 (2023)
  * [56] Yu, Q., He, J., Deng, X., Shen, X., Chen, L.C.: Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. In: NeurIPS (2023)
  * [57] Zhang, Y., Huang, X., Ma, J., Li, Z., Luo, Z., Xie, Y., Qin, Y., Luo, T., Li, Y., Liu, S., et al.: Recognize anything: A strong image tagging model. arXiv preprint arXiv:2306.03514 (2023)
  * [58] Zhu, X., Zhou, H., Wang, T., Hong, F., Ma, Y., Li, W., Li, H., Lin, D.: Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9939–9948 (2021)

In this supplemental file, we provide the following materials:

  * $\bullet$

The instructions to reduce noisy tags via GPT and the corresponding
visualizations of 3D label map by using denoised tags (referring to “Sec.
3.1-Comprehensive Text Modality Generation-Reliable GPT-based Denoising” in
the main paper);

  * $\bullet$

Examples of scene-level captions extracted by MLLMs and the corresponding
visualizations of 3D label map by using the captions as queries (referring to
“Sec. 3.1-Comprehensive Text Modality Generation-Scalable Scene Descriptions”
in the main paper);

  * $\bullet$

The impacts of tagging models and MLLMs on different datasets (referring to
“Sec. 3.1-Comprehensive Text Modality Generation” and “Sec. 4.3-Ablation
Study-tagging models _vs_. MLLMs” in the main paper);

  * $\bullet$

Effect of Mutually Inclusive Loss (MIL) (referring to “Sec. 3.4-Dense
Multimodal Alignment-Mutually Inclusive Loss” in the main paper);

  * $\bullet$

More qualitative results on indoor and outdoor datasets (referring to “Sec.
4.2-Comparison with State-of-The-Arts” in the main paper).

##  6 Reducing Noisy Tags

Instruction. We use GPT to reduce noisy tags generated by RAM [57]. The
corresponding instructions and examples are given in Fig. 8. Given the input
list of tags, we instruct GPT to evaluate each word individually to determine
if it meets the given rules, and we ask it to output the chain of thought and
the final boolean list.

Figure 8: Instruction and examples to reduce noisy tags.

Denoised tagging results. We give some examples of the denoised scene tags and
the corresponding visualizations in Fig. 9. We can observe that many noisy
tags can be filtered out in terms of the rules given in Fig. 8. We also
visualize the text-to-3D label map $M^{3D}_{tag}$ as defined in Eq. 2 of the
main paper. Label errors in this 3D label map are consequently reduced based
on the denoised tags.

##  7 Scene Captions

In addition to category names, we also leverage MLLMss to generate
comprehensive descriptions for each view. Fig. 10 shows some examples of the
scene captions extracted by LLaVA [35] and the visualizations of 3D label map
$M^{3D}_{llm}$ generated by employing the captions as queries. We can build
dense text(caption)-to-3D correspondences, thus achieving more accurate and
fine-grained alignment.

##  8 Tagging Models _vs_. MLLMs

In Tab. 7 we compare the results of aligning 3D points to image tags and
captions on different datasets. The former involves complete category
information, while the latter contains comprehensive scene descriptions and
more contextual information. As can be seen, tagging model plays a crucial
role in the overall performance, as it encompasses more extensive categories
and semantics. MLLM further enhances the final performance by incorporating
rich contextual semantics and diversified descriptions. It is worth noting
that MLLM only contributes to a marginal performance improvement on the
nuScenes [4] dataset since this dataset contains a limited number of image
views.

| ScanNet | Matterport3D | nuScenes
---|---|---|---
Tagging+GPT | 46.3 | 37.7 | 43.3
MLLM | 24.2 | 19.4 | 11.3
Both | 50.5 | 39.8 | 43.8

Table 7: Performance comparison of tagging model and MLLM.

##  9 Mutually Inclusive Loss

As discussed in Sec. 3.4 of the main paper, we use mutually inclusive loss to
align 3D representations to text modality. In Tab. 8, we compare the
performance by using mutually inclusive (BCE loss) and exclusive (CE loss)
losses. Compared to binary cross-entropy loss, the cross-entropy loss leads to
a performance degradation of 6.4% mIoU because it causes conflicts between
text labels with similar semantics, making it difficult for the model to
converge and limiting the performance improvement.

| Cross-entropy | Binary cross-entropy
---|---|---
mIoU | 46.9 | 53.3
Table 8: Comparison of mutually inclusive and exclusive losses on ScanNet
dataset.

##  10 Qualitative Results

We give more qualitative results on both indoor and outdoor datasets in Fig.
11 and Fig. 12. Our method consistently produces better results on ScanNet
[11] and nuScenes [4] datasets, demonstrating the robustness and better
generalization ability of the proposed method on different data distributions
and open-set categories.

Figure 9: Examples of scene tagging results by RAM [57] and visualizations of
3D label map $M^{3D}_{tag}$ by employing the tags as queries. The red words
denote the noisy tags which are filtered out by GPT and multi-view voting.
Figure 10: Examples of scene captions extracted by LLaVA [35] and
visualizations of 3D label map $M^{3D}_{llm}$ by employing the captions as
queries. We can build dense text-to-3D correspondences based on the obtained
captions.
Figure 11: Qualitative results of different methods on ScanNet dataset.
Figure 12: Qualitative results of different methods on nuScenes dataset.


# LGMCTS: Language-Guided Monte-Carlo Tree Search
for Executable Semantic Object Rearrangement

Haonan Chang, Kai Gao, Kowndinya Boyalakuntla, Alex Lee, Baichuan Huang,
Harish Udhayakumar,
Jinjin Yu, Abdeslam Boularias The authors are with the Department of Computer
Science, Rutgers University, 08854 New Brunswick, USA. This work is supported
by NSF awards 1846043 and 2132972.

###### Abstract

We introduce a novel approach to the executable semantic object rearrangement
problem. In this challenge, a robot seeks to create an actionable plan that
rearranges objects within a scene according to a pattern dictated by a natural
language description. Unlike existing methods such as StructFormer and
StructDiffusion, which tackle the issue in two steps by first generating poses
and then leveraging a task planner for action plan formulation, our method
concurrently addresses pose generation and action planning. We achieve this
integration using a Language-Guided Monte-Carlo Tree Search (LGMCTS).
Quantitative evaluations are provided on two simulation datasets, and
complemented by qualitative tests with a real robot. Our code and
supplementary materials are accessible at https://github.com/changhaonan/LG-
MCTS.

##  I INTRODUCTION

In daily life, tasks like “Set up the kitchen” require arranging objects
according to language cues, an intuitive process for humans but a complex
challenge for robots. The semantic rearrangement problem aims to enable robots
to reconstruct scenes based on linguistic descriptions. This necessitates the
seamless integration of scene understanding, linguistic reasoning, and action
planning, involving multiple disciplines such as robotics and natural language
processing.

Consider the task: “Set the dinnerware for dinner and place a candle in front
of a plate.” A robot must identify the items constituting ‘dinnerware’ and
their correct arrangement, while also handling real-world constraints like
initial object coverings and spatial obstacles. The command introduces two key
constraints: 1) ‘dinnerware’ must be arranged appropriately for dinner, and 2)
a candle must be placed in front of a plate. This example highlights the
problem’s complexity.

One approach employs a multi-modality transformer [1, 2, 3] to learn the
mapping between language and object poses through simulated arrangements and
rule-based language descriptions. However, this method has limitations. It
assumes that language descriptions map to precise ground-truth poses, which is
often unrealistic. It is also less adaptable to free-form linguistic inputs,
as it performs best with descriptions similar to its training data.

Recent research employs diffusion models to capture how language tokens map to
spatial distributions. DALL-E-Bot [4] uses text-to-image to generate target
images and derive poses, while StructDiffusion [5] employs a diffusion model
conditioned on language and point-cloud embeddings. Both approaches show
promise but have limitations: DALL-E-Bot can be unstable and distracted by
extraneous objects, whereas StructDiffusion is constrained to known training
patterns and lacks zero-shot adaptability.

Moreover, recent developments emphasize direct robot control through Large
Language Models (LLMs) [6, 7, 8] and prompt techniques such as Chain of
Thoughts (COT) [9]. The ‘Code-as-policies’ [10] approach shows remarkable
zero-shot capabilities and could theoretically address the semantic
rearrangement problem, leveraging LLMs’ strength in pattern comprehension and
object selection.

Figure 1: Robot Setup. We use a UR5e robot equipped with a RealSense D455
camera. Figure 2:  Illustration of infeasible start and goal configuration. In
this scene, block 2 is in an infeasible start configuration. And the goals of
blocks 3 and 2 are infeasible without an excessive number of intermediate
steps.

Current methods such as StructFormer and Code-as-policies often fail to
produce collision-free arrangements, whereas StructDiffusion incorporates a
learning-based collision checker. However, collision-free does not mean
executable, as shown in Fig. 2. For example, block 3 obstructs block 2’s
target position, complicating block 1’s motion. This highlights the challenge
of distinguishing between collision-free and executable goals, as some may be
the former but not the latter, or may require excessive steps to execute.

This study presents the Language-Guided Monte-Carlo Tree Search (LGMCTS)
technique, designed specifically for executable semantic object rearrangement.
LGMCTS leverages LLMs to interpret free-form language and consider object
placements as probability distributions, not exact points. We frame the
challenge as a sequential sampling problem, where each object’s pose is drawn
from a distribution influenced by language and the current scene state. The
approach incorporates distracting objects, enabling plans that meet language
criteria while also being executable.

The primary contributions of this study are: 1) Introducing a novel approach
that concurrently addresses semantic rearrangement goal generation and action
planning 2) Presenting a unique method that facilitates zero-shot multi-
pattern composition for semantic object rearrangement 3) Establishing a new
benchmark tailored for executable semantic object rearrangement.

##  II RELATED WORKS

###  II-A Semantic Rearrangement

The semantic rearrangement problem consists of devising a rearrangement plan
that is both semantically congruent with a given language description and
physically feasible. In recent years, this has gained increased traction,
particularly as a pivotal application in language-driven robotics. CLIPort [1]
took the initial step in this direction by merging CLIP features with a
Transporter network. Yet, its design is limited to basic pick-and-place tasks.
StructFormer [2] advanced the field using a transformer model, simulating
rearrangements with hand-crafted rules and connecting language tokens to
object poses. Leveraging StructFormer’s dataset, StructDiffusion [5]
introduced a pose diffusion model to predict poses from language, enhancing
performance. Nonetheless, a common shortcoming amongst these methodologies is
their limitation to a single structure or pattern that they have been trained
on, making composite patterns a persistent challenge. Meanwhile, the
rearrangement goals generated by these methods might be inexecutable.

###  II-B LLM-driven Robot Control

Recent advancements in large language models (LLMs) [6, 7, 8] have showcased
stellar performance across a broad spectrum of tasks. This has led to a
growing interest in LLM-driven robotics. SayCan [11] was among the first to
integrate LLMs into robotic task planning, which resulted in impressive
context comprehension and behavioral decision-making. Subsequently, Code-as-
policies [10] merges LLM code generation with API planning, demonstrating
remarkable zero-shot capabilities across various robotic tasks. Nonetheless,
LLM-driven robotics faces several challenges. LLMs, while semantically adept,
seem to lack a true physical scene understanding, leading to plans that,
although meaningful, can be unfeasible.

###  II-C Rearrangement Planning

In rearrangement problems, determining the sequence of tasks presents a
significant challenge. Several studies have tackled this by encoding
collisions between initial and target arrangements into a graph[12, 13, 14].
These representations then transform the problem into established graph
problems. Additionally, recent advances[15, 16] have adopted the Monte Carlo
Tree Search (MCTS) for long-horizon planning. A common attribute among most
prehensile planners is the need for specific goal states[17]. However,
dictating a goal state in semantic rearrangement can restrict the solution
space, leading to potential planning failures. To address this, our proposed
LGMCTS planner calls for goal state distributions from a language model, which
act as constraints for individual goal poses.

##  III Preliminaries

###  III-A Problem Formulation

The task of semantic rearrangement can be succinctly defined as follows. Given
a scene with objects represented by $O_{S}=\\{o_{1},o_{2},\ldots,o_{N}\\}$ and
a command $L$, where $L$ is a pure natural language command that implies a
desired distribution list $D=\\{d_{i}:p(o_{i})\sim f_{d_{i}}|o_{i}\in
O_{R}\\}$, $p(o_{i})$ refers to the position of object $o_{i}$. Here,
$O_{R}\subset O_{S}$ denotes the objects designated for rearrangement, and
$d_{i}$ indicates the desired pose distribution for each object. The objective
is to identify an optimal action sequence, $A=(a_{t})_{t=1}^{H}$, where each
action $a_{t}$ corresponds to moving an object $o_{i}$ to a sampled position
$p(o_{i})$, with the goal to achieve a goal arrangement aligning $L$, i.e.
$\prod_{o_{i}\in O_{R}}f_{d_{i}}(p_{i})>0$ and minimizing the action steps
$H$. Noticeably, $A$ not only includes movement of objects $o\in O_{R}$, but
also distracting objects, denoted as $O_{d}$, with $O_{d}\subset O_{S}$.

###  III-B Monte Carlo Tree Search

A typical MCTS algorithm iteratively performs the following four operations:

  1. 1.

Selection. On a fully expanded node (all the children nodes have been
visited), MCTS selects a branch to explore with an Upper Confidence Bound
(UCB) formula:

$argmax_{a}(\dfrac{w(f(s,a))}{n(f(s,a))}+C\sqrt{\dfrac{\log(n(s))}{n(f(s,a))}})$
(1)

where $f(s,a)$ is the child node of state $s$ after action $a$, $w(\cdot)$ and
$n(\cdot)$ are cumulative rewards and the number of visits to a state.

  2. 2.

Expansion. On a node that is not fully expanded, MCTS selects an action that
has not been attempted yet.

  3. 3.

Simulation. Given a node and the selected action, MCTS simulates and gets
rewards.

  4. 4.

Back-Propagation. MCTS passes the acquired reward to ancestor nodes to update
the quality evaluation of the branch.

In each iteration, MCTS starts from the root node. When all the child nodes of
the current node are visited, MCTS selects a child node with the UCB formula.
When some child nodes of the current node are unvisited, MCTS expands by
randomly selecting a new action and doing a simulation to reach a new child
node. The new node returns a reward, which is be back-propagated to all the
ancestor nodes.

##  IV Method

###  IV-A Notation Vocabulary

TABLE I: Notation and Definitions Symbol | Meaning | Definition
---|---|---
$L$ | Language input |
$D$ | Parsed distribution set |
$o_{i}$ | Individual object |
$d_{i}$ |  Pose distribution for $o_{i}$ |
$O_{R}$ | Objects to be rearranged |
$L_{i}$ |  Language guidance for sampling $o_{i}$ |
$O_{R_{i}}$ |  Subset of objects associated with $L_{i}$ |
$w(f(s,a))$ |  Total reward accumulated for action $a$ in state $s$ | Eq. (1)
$n(f(s,a))$ |  Number of times action $a$ in state $s$ is taken | Eq. (1)
$C$ | Exploration constant | Eq. (1)
$\log(n(s))$ |  Logarithm of the number of times state $s$ has been visited | Eq. (1)
$f_{d_{i}}(p_{i}|P_{S},L_{i})$ |  Conditional distribution for $o_{i}$ | Eq. (2)
$p_{i}$ |  Pose of object $o_{i}$ |
$P_{S}$ | Set of current poses for all objects |
$f_{\text{prior}}$ | Pattern Prior function | Eq. (3)
$f_{\text{free}}$ | Free space function |
$F_{k}$ |  $k^{th}$ Key in database | Eq. (4)
$F_{L^{i}}$ |  Sentence-BERT embedding of $L^{i}$ |
$\gamma$ | Parametric curve function |
$\kappa$ | Parameters related with a parametric curve |
$P_{R^{i}}$ |  Poses for all objects in $O_{R^{i}}$ |
$P^{s}_{R_{i}}$ |  Poses of sampled objects in $O_{R^{i}}$ |
$N$ | Total number of objects associated with a pattern |
$K$ | Number of sampled objects |
$S_{A}$ | MCTS action set |  Algorithm 1

###  IV-B Language Parsing & Object Selection

Since the input $L$ is a pure natural language command, we must first parse
$L$ into $D=\\{o_{i}:d_{i}|o_{i}\in O_{R}\\}.$ Here, we utilize a large
language model (LLM) to parse the language into some structured
representation. Similar to previous methods [10], we conduct a prompt
engineering to guide the LLM to parse $L$ into $D$. The listing below
showcases how prompt engineering is implemented. Essentially, the language
model translates user requirements into structured goal configurations and
constraints that guide task execution.

Figure 3: An example of Language Parsing. We are using GPT-4 [6] in this work.

Consider the example depicted in Fig. 3. Here, the system prompt provides
guidelines for the LLM to follow when interpreting user queries. For producing
the descriptor labels such as name, color, and ID for the objects in the
scene, we use the Recognize Anything Model (RAM) [18, 19] for producing the
semantic labels, and a color detector for associating colors to the objects in
the scene.

Following the relay of user instructions to the LLM, our requirements for the
model can be summarized in two main steps: 1) Identifying an ensemble of
pertinent objects ($O_{R}$) and 2) Correlating them with their respective
sampling pattern descriptions and relevant objects to form these patterns.
More precisely, for each object $o_{i}\in O_{R}$, there is a corresponding
sampling pattern description $L_{i}$ and a subset of objects $O_{R_{i}}\subset
O_{R}$ associated with $L_{i}$.

Through extensive testing, we observed that instead of directly instructing
the LLM to prescribe a sampling pattern for individual objects, it is more
robust to guide it to discern the number of patterns present in the
instruction and to identify the respective objects within these patterns (Fig.
3).

###  IV-C Distribution Generation

Figure 4: Visualization of $(x,y)$ prior for ‘line’ pattern. From left to
right: $K=0$, $K=1$, $K=2$, $K=3$. White star marks are sampled poses. When
$K=0$, the pose can be sampled anywhere. When $K=1$, it needed to sampled
outside a circle region. After that all poses will be sampled along the line
defined by the first two poses.

In this section, our aim is to construct a conditional distribution
$f_{d_{i}}(p_{i}|P_{S},L_{i})$ for each object $o_{i}$ in the set $O_{R}$.
Here, $p_{i}$ denotes the pose $(x,y,\theta)$ of an object $o_{i}$, $P_{S}$
represents the current set of poses for all objects, and $L_{i}$ serves as the
language guidance for sampling poses for $o_{i}$.

To facilitate this, we compute $f_{d_{i}}(p_{i}|P_{S},L_{i})$ as an element-
wise product of two components: a pattern prior function
$f_{\text{prior}}(P_{R^{i}},L_{i})$ and a boolean function
$f_{\text{free}}(P_{S})$ of the workspace,

$f_{d_{i}}(p_{i}|P_{S},L_{i})=f_{\text{prior}}(P_{R^{i}},L_{i})\times
f_{\text{free}}(P_{S})$ (2)

In this equation, $P_{R^{i}}$ stands for the set of poses for objects $o_{j}$
belonging to $O_{R^{i}}$. $f_{\text{free}}(P_{S})$, indicating the free space,
can be determined by running a 2D collision simulation using point cloud
observations for each object $o_{i}$.

Our primary focus is to determine $f_{\text{prior}}(P_{R^{i}},L_{i})$. To this
end, we employ an approach akin to the one described in [11]. We maintain a
database comprising a collection of predefined prior functions. Each of these
functions is linked with one or more Sentence-BERT embeddings, acting as keys.
The function corresponding to the most closely matching key is selected, as
follows,

$f_{\text{prior}}(P_{R^{i}},L_{i})=f_{\text{prior}}^{k}(P_{R^{i}});\quad
k=\underset{k\in\text{DB}}{\text{argmax}}(F_{k}\cdot F_{L^{i}})$ (3)

Here, $F_{k}$ is the $k^{th}$ key in the database (DB), and $F_{L^{i}}$ is the
Sentence-BERT embedding generated from the language instruction $L^{i}$. In
summary, the most suitable prior function from the database is selected based
on the Sentence-BERT embedding of the given language instruction $L_{i}$.

We now delve into the definition of $f_{prior}^{k}(P_{R^{i}})$ in the context
of LGMCTS. We aim to provide a unified approach to defining patterns by
employing different parametric curves. A parametric curve can be expressed as
$(x,y)=\gamma(t,\kappa)$, where $t$ ranges from 0 to 1 and $\kappa$ is a set
of curve-defining parameters. In LGMCTS, $\kappa$ is modeled as a function of
two 2D positions, denoted as $\kappa(p_{0},p_{1})$. For each pattern, we
define two functions: $\gamma$ and $\kappa$.

Given that pattern prior $f_{prior}$ is used inside a sequential sampling
process (check Section IV-D for more details), we want distribution to capture
the history of sampling. Consequently, we further categorize $O_{R^{i}}$ based
on whether the objects have been sampled in MCTS-Planner. The sets
$O_{R_{i}}^{s}$ and $O_{R_{i}}^{n}$ denote the sampled and non-sampled
objects, respectively.

A predefined sampling function $f$ for $o_{i}$ takes three parameters into
account: 1) $N=|O_{R_{i}}|$ represents the total number of objects forming
this pattern 2) $K=|O_{R_{i}}^{s}|$ indicates the number of objects already
sampled 3) ${P^{s}_{R_{i}}}$ refers to the poses of the sampled objects
$O_{R_{i}}^{s}$.

The sampling function $f$ has three distinct cases based on $K$:

  1. 1.

When $K=0$, $(x_{i},y_{i},\theta_{i})\sim U$, suggesting that the first object
can be placed arbitrarily.

  2. 2.

For $K=1$, $(x_{i},y_{i},\theta_{i})\sim U$ and
$\sqrt{(x_{i}-x_{0})^{2}+(y_{i}-y_{0})^{2}}\geq\delta$, imposing that the
second object must be distanced from the first by at least $\delta$.

  3. 3.

When $K\geq 2$,
$(x_{i},y_{i})=\gamma\left(\frac{K}{N},\kappa(p_{0},p_{1})\right)+\epsilon$
where $\epsilon\sim G(0,\sigma)$, here $G$ represents an Gaussian distribution
of variance $\sigma$. $\theta=atan2(1,\gamma^{\prime}(\frac{K}{N}))$, we use
the angle of gradient represented as the rotation angle of the object.

In our current implementation, we have defined patterns such as “line,”
“circle,” “rectangle,” “tower,” “spatial:left,” “spatial:right,” and so on.
Due to space constraints, we refrain from elaborating on the definitions of
$\gamma$ and $\kappa$ for all these predefined patterns. However, to enhance
clarity for the readers, we offer illustrative figures of the ‘lines’ pattern
in Fig. 4 to shed light on the process of the prior generation.

Noticeably, we divide patterns into ‘ordered’ and ‘unordered’ based on if the
pattern requires an execution sequence.

###  IV-D Monte-Carlo Tree Search (MCTS) for Task Planning

We propose a task planner based on the Monte Carlo Tree Search (MCTS)
algorithm to move objects to desired collision-free poses. The distribution
list $D$ indicates the preference distributions of the goal poses. We define
pose $p_{i}$ as a desired pose of the object $o_{i}$ if its probability
$f_{d_{i}}(p_{i})$ is higher than a threshold $\epsilon$. Our MCTS-Planner
seeks an action sequence by maintaining a search tree. In MCTS-Planner, each
state in the tree represents an arrangement of $O$:
$\\{p_{1},p_{2},...,p_{N}\\}$, and a list of remaining pose distributions $D$.
The MCTS action set $S_{A}$ of a state defines a finite number of branches
that we expand the state with, which is computed as shown in Algorithm 1. Each
MCTS action is labeled as $(d_{i},j)$, representing the $j^{th}$ attempt to
make progress in sampling $d_{i}$. $k$ is the number of attempts we try for
each $d_{i}$. Note that in ordered patterns, we will not consider sampling an
object $o_{i}$ into $d_{i}$ if there is an object $o_{j}$ in the constraints
of $d_{i}$ still away from the goal pose (Line 3). For example, in Fig. 3,
there is a pattern “A is on the right and behind B”. $A$ will not be sampled
to its goal distribution if $B$ is still away from its goal.

Input : $s$: An MCTS state,

$k$: The number of attempts for each sampler.

Output : $S_{A}$: Action set

1 $S_{A}\leftarrow\emptyset$;

2 for  _ $d_{i}\in s.D$_ do

3 if  _availableToSample( $d_{i}$)_ then

4 for  _$j\leftarrow 0$ to $k-1$_ do $S_{A}$.add($d_{i},j$);

5

return $S_{A}$

Algorithm 1 Get Action Set

The simulation stage of MCTS-Planner is presented in Algorithm 2. The
algorithm consumes the MCTS state $s$ and the attempted action $(d_{i},j)$. If
the corresponding object $o_{i}$ is not graspable, we randomly choose a
graspable obstacle on top and uniformly sample a free space to place it (Line
2-6). If $o_{i}$ is graspable, we try to sample it in $d_{i}$. (Line 8) If the
sampled position is not preferred: $f_{d_{i}}(p)<\epsilon$, we find an
obstacle $o$ in $d_{i}$ and uniformly sample a free space to place it (Line
10-14). Similar to the formulation in [15], in LGMCTS, the reward of a node
$s$ is the number of finished samplers. That is, $|root.D|-|s.D|$. While MCTS
is an anytime search algorithm, in our implementation, MCTS-Planner returns
the first found solution. Finally, we prove that MCTS-Planner is probabilistic
complete under our setting.

Input : $s$: An MCTS state,

$(d_{i},j)$: The action for this simulation.

Output : $(o,p)$: a rearrangement action.

1 $o_{i}\leftarrow$ The sampling object in $d_{i}$;

2 if  _$o_{i}$ not graspable_ then

3 $o\leftarrow$ Randomly choose a graspable object on top of $o_{i}$;

4 $p\leftarrow$uniformSampling($o,s$);

5 if  _$p$_ then return ($o$,$p$);

6 else return None;

7

8else

9 $p\leftarrow$sampling($o_{i},d_{i},s$);

10 if  _$f_{d_{i}}(p)\geq\epsilon$_ then return ($o_{i}$,$p$);

11 else

12 $o\leftarrow$ An obstacle in $d_{i}$;

13 $p\leftarrow$uniformSampling($o,s$);

14 if  _$p$_ then return ($o$,$p$);

15 else return None;

16

Algorithm 2 Simulation Figure 5:  Results with a real UR5 robot. The language
instructions for the five scenes are: (a). “Move all blocks into a circle;
while put the white bottle in front of one block;” (b). “Put all boxes into a
rectangle; and move the white bottle to the left of one box;” (c) “Move
bottles into a line; and formulate all phones into another line;” (d)
“Formulate all yellow objects into a line;” (e) “Set all phones into a line;”.
Dotted lines imply a shape pattern and red arrows indicate a spatial pattern
(left, right, front, back). These real robot experiments show that LGMCTS can
parse complex language instructions and also deal with infeasible start
configurations as well as pattern composition.

###### Proposition IV.1

MCTS-Planner is probabilistic complete.

###### Proof:

For a semantic rearrangement task and the distribution list $D$, assume that
there is a feasible action sequence $A^{*}$ moving objects to a final
arrangement $A_{f}$, such that $f_{d_{i}}(A_{f}[o_{i}])\geq\epsilon\ \forall
o_{i}\in O_{R}$. Denote p as the probability that MCTS can find an action
sequence satisfying the goal state criteria. We prove that as the number of
samples k increases, p approaches 1.

First, we prove that there is an action sequence $A^{*}_{0}$ whose actions can
all be generated by Algorithm 2. Note that in Algorithm 2, a MCTS action
$(d_{i},j)$ satisfies two rules: R1: If $o_{i}$ is not graspable, we move away
obstacles of $o_{i}$(Line 3); R2: If $o_{i}$ is graspable, we move $o_{i}$
into $d_{i}$ or remove some obstacle in $d_{i}$ for $o_{i}$ (Line 11). We
construct $A^{*}_{0}$ by reordering and deleting actions in $A^{*}$ as
follows: 1) If an action satisfies R1 and R2, we add the action to
$A^{*}_{0}$. Otherwise, we delay the addition until it satisfies the rules; 2)
If the action still cannot satisfy the requirements before we examine the next
action in $A^{*}$ for the same object, we delete the action. In this way, all
the actions in $A^{*}_{0}$ can be generated by Algorithm 2, and the final
arrangement is also $A_{f}$. Let the rearrangement action $(o,p)$ that
$A^{*}_{0}$ chooses at state $s$ be $(A^{*}_{0}[s].o,A^{*}_{0}[s].p)$.

Let $r=\dfrac{min(C_{1},C_{2})}{|A^{*}_{0}|}$, where $C_{1}$ is the minimum
distance between objects and their nearest obstacles in $A^{*}_{0}$, $C_{2}$
is the minimum distance between each pose $A_{f}[o_{i}]$ and their nearest
position $p$, s.t. $d_{i}(p)<\epsilon$. As $k$ increases, the probability that
$A^{*}_{0}[s].o$ is moved to the $r-$neighborhood of $A^{*}_{0}[s].p$ at state
$s$ approaches $1$. Then, given a tolerance of pose offset $r$, the
probability that all the intermediate states of $A^{*}_{0}$ are in the MCTS
tree approaches $1$. When $A_{f}$ is in the MCTS tree, the MCTS-Planner can
find a solution after enough iterations.∎

##  V EXPERIMENTS

In this part, we present a comprehensive evaluation of LGMCTS on: 1) Its
capability to produce collision-free and semantically correct goal poses 2)
The advantages of concurrently addressing pose generation and action planning
3) The stability of the LLM parse module, and 4) LGMCTS’s performance in
actual robotic systems.

| Line (4295) | Circle (3416) | Tower (1335) | Dinner (2440)
---|---|---|---|---
StructFormer | 47.24% | 62.64% | 99.10% | 28.36%
StructDiffusion | 61.49% | 81.41% | 98.95% | 69.38%
LGMCTS-T (ours) | 95.99% | 95.25% | 100% | 100%

TABLE II: Efficacy of StructFormer, StructDiffusion, and LGMCTS across diverse
rearrangement tasks (task counts indicated) from the StructFormer dataset

###  V-A Baselines

We compare our approach with the following baselines and LGMCTS variants:

StructFormer [2]: StructFormer is a multi-modal transformer architecture
specifically designed for language-guided rearrangement tasks.

StructDiffusion [5]: Recognized as the state-of-the-art, StructDiffusion
employs a diffusion model combined with a learning-based collision checker for
pattern pose generation.

Pose+MCTS: The Pose+MCTS (PMCTS) approach assumes that a collision-free and
semantically aligned goal pose is provided. However, direct execution of this
pose might be hindered if the target space is already occupied. To address
this, we utilize MCTS to search for a viable plan to place objects in their
predetermined goal poses.

LGMCTS-T: This is a variation of LGMCTS that uses ground-truth data for object
and pattern language selection. Given that $O_{R}$ and ${L_{i}}$ are directly
provided, LGMCTS-T functions as LGMCTS without the LLM parser.

LGMCTS-L: This represents the full LGMCTS system. It employs the LLM to
interpret input from natural language and subsequently produce an action plan.

###  V-B Datasets

StructFormer Dataset [2]: We use the test set from the StructFormer dataset to
evaluate the goal pose generation ability. This dataset is composed of
approximately $11,500$ rearrangement tasks, categorized into four patterns:
line, circle, tower, and dinner.

Figure 6: Compared to StructFormer and StructDiffusion, LGMCTS ensures a
collision-free goal arrangement in a qualitative comparison.

A rearrangement plan is regarded as successful if and only if it meets the
language constraints while containing no collision. It is worth noting we do
not apply collision checking for the ‘tower’ task, for collision is not
avoidable in that specific task. We solve the ‘dinner’ pattern by treating it
as a pattern composition. It involves rearranging objects such as plates,
bowls, forks, spoons, cups, and knives. Within this pattern, the plate and
bowl are arranged to form a ‘tower,’ while the remaining objects are
positioned adjacent to this tower forming a ‘line’.

LGR-Benchmark Existing datasets for semantic object rearrangement tasks, like
StructFormer, have limitations. They typically support only a single pattern
per scene and lack crowded scenarios. Moreover, they often overlook the
feasibility challenge, especially scenarios like infeasible starting
configurations where one object might be placed under another from the outset.
Addressing these gaps, we introduce LGR-Bench (Language-Guided Rearrangement
Benchmark). This new benchmark presents a novel task termed the “multi-pattern
task”, which requires multiple pattern goals to be satisfied during the
rearrangement process. We also incorporate scenarios with infeasible starting
configurations, where objects may initially be stacked. In each scene, we
randomly select two patterns from “line”, “circle”, and “spatial”. This LGR-
Benchmark is modified from VIMA-Benchmark [3].

###  V-C Semantic Pattern Pose Generation

In this assessment, we draw comparisons between StructFormer, StructDiffusion,
and LGMCTS-T. We decided against using LGMCTS-L because, in the StructFormer
Dataset, object selection demands an insight into the object’s shape and size,
which is not the focus of LGMCTS. To ensure an equitable comparison, we
provide ground-truth object selections to both StructFormer and
StructDiffusion. In this context, LGMCTS showcased exemplary performance
across all four rearrangement task categories. As evidenced in TABLE II,
LGMCTS posted outstanding success rates: 95.99% for the ‘line’ pattern, 95.25%
for ‘circle,’ and a perfect 100% for both the ‘tower’ and ‘dinner’ patterns.
On the other hand, although StructDiffusion improved upon StructFormer’s
results, it did not rival the success of our approach. A unique characteristic
of StructDiffusion is its employment of a collision checker that filters out
samples with collisions, providing some clarity to its enhanced performance
over StructFormer. For a more tangible comparison, Fig. 6 displays a scene
from the circle task, illuminating the distinctions between LGMCTS and the
existing benchmarks.

| $SR_{p}$ | $SR_{e}$ | $SR_{a}$ | Steps |
---|---|---|---|---|---
PMCTS | 82.9% | 86.2% | 74.1% | 6.15 |
LGMCTS-T (ours) | 97.3% | 93.4% | 92.8% | 5.99 |
TABLE III: $SR_{p}$ represents the success rate of planning. Both PMCTS and LGMTCS-T were set with an identical planning step limit, specifically, 10,000 steps. $SR_{e}$ denotes the success rate of execution, assessed post-execution of the plan to determine if the outcome aligns with the language-derived constraints. $SR_{e}$ is the overall success rate. The term ‘steps’ here refers to the average number of steps in the plan returned by each planner. Smaller Steps mean the action can executed faster. | $SR_{p}$ | $SR_{e}$ | $SR_{a}$ | $Acc_{LLM}$ |
---|---|---|---|---|---
LGMCTS-L | 90.9% | 83.1% | 79.2% | 89.3% |
TABLE IV: Performance of LGMTC-L on LGR-Benchmark. For $Acc_{LLM}$, we compare
the object selection and pattern selection, if they are the same, we return
true.

###  V-D Benefit of Joint Modeling

Contrary to previous methodologies like StructFormer and StructDiffusion,
which viewed pose generation and action planning as separate challenges,
LGMCTS concurrently addresses both pattern generation and action planning. We
posit that this integrated approach will render our goal poses more executable
than other ‘correct’ alternatives. To substantiate this idea, we juxtapose the
performance of LGMCTS-T and PMCTS using the LGR-Benchmark, providing PMCTS
with ground-truth goal poses. The comparative data is presented in TABLE III.
Beyond assessing the success rates of planning and execution, we also evaluate
the overall success rate and the number of actions suggested by each planning
mechanism. Our findings highlight the superiority of LGMCTS over the two-step
solutions, even when they are provided with a semantically accurate and
collision-free goal pose.

###  V-E Language Parse Stability

We provide an evaluation of our whole pipeline, i.e. LGMCTS-L on LGR-
Benchmark. Apart from the success rate, we also present the accuracy of LLM
instruction parse. The result is shown in TABLE IV. From the result, we can
find the LLM parsing module of LGMTS has high stability. While the performance
of LGMCTS-L lags behind that of LGMCTS-T, it still surpasses that of PMCTS.

###  V-F Physical Robot Experiment

We qualitatively evaluated our system using a UR5e robot outfitted with a D455
depth camera as shown in Fig. 1. We employed the Recognize-Anything-Model
(RAM) [19, 18] and an HSV-based color detector to detect object semantics and
colors. Selected queries and their corresponding execution outcomes can be
viewed in Fig. 5. These real-world robot experiments highlight the
capabilities of LGMCTS in intricate real-world settings.

##  VI CONCLUSION

We introduced LGMCTS, a new framework for tabletop, semantic object
rearrangement tasks. LGMCTS stands out by accepting free-form natural language
input, accommodating multiple pattern requirements, and jointly solving goal
pose generation and action planning. However, its main drawback is the
extended execution time for complex scenes. Improving its Monte-Carlo tree
search efficiency is a key research direction. Currently tailored for tabletop
pick-place setups, future work should explore LGMCTS’s adaptability to more
complex rearrangement contexts.

## References

  * [1] M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” _ArXiv_ , vol. abs/2109.12098, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:237396838
  * [2] W. Liu, C. Paxton, T. Hermans, and D. Fox, “Structformer: Learning spatial structure for language-guided semantic rearrangement of novel objects,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 6322–6329.
  * [3] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan, “Vima: General robot manipulation with multimodal prompts,” _arXiv preprint arXiv:2210.03094_ , 2022.
  * [4] I. Kapelyukh, V. Vosylius, and E. Johns, “Dall-e-bot: Introducing web-scale diffusion models to robotics,” _IEEE Robotics and Automation Letters_ , 2023\.
  * [5] W. Liu, T. Hermans, S. Chernova, and C. Paxton, “Structdiffusion: Object-centric diffusion for semantic rearrangement of novel objects,” _arXiv preprint arXiv:2211.04604_ , 2022.
  * [6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [7] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar _et al._ , “Llama: Open and efficient foundation language models,” _arXiv preprint arXiv:2302.13971_ , 2023.
  * [8] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale _et al._ , “Llama 2: Open foundation and fine-tuned chat models,” _arXiv preprint arXiv:2307.09288_ , 2023.
  * [9] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou _et al._ , “Chain-of-thought prompting elicits reasoning in large language models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 24 824–24 837, 2022.
  * [10] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 9493–9500.
  * [11] K. E. Ferguson and P. Turnbull, _Oh, Say, Can You See?: The Semiotics of the Military in Hawaii_. U of Minnesota Press, 1999, vol. 10.
  * [12] S. D. Han, N. M. Stiffler, A. Krontiris, K. E. Bekris, and J. Yu, “Complexity results and fast methods for optimal tabletop rearrangement with overhand grasps,” _The International Journal of Robotics Research_ , vol. 37, no. 13-14, pp. 1775–1795, 2018.
  * [13] K. Gao, S. W. Feng, B. Huang, and J. Yu, “Minimizing running buffers for tabletop object rearrangement: Complexity, fast algorithms, and applications,” _The International Journal of Robotics Research_ , p. 02783649231178565, 2023.
  * [14] K. Gao, D. Lau, B. Huang, K. E. Bekris, and J. Yu, “Fast high-quality tabletop rearrangement in bounded workspace,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 1961–1967.
  * [15] Y. Labbé, S. Zagoruyko, I. Kalevatykh, I. Laptev, J. Carpentier, M. Aubry, and J. Sivic, “Monte-carlo tree search for efficient visually guided rearrangement planning,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 3715–3722, 2020.
  * [16] H. Song, J. A. Haustein, W. Yuan, K. Hang, M. Y. Wang, D. Kragic, and J. A. Stork, “Multi-object rearrangement with monte carlo tree search: A case study on planar nonprehensile sorting,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 9433–9440.
  * [17] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez, “Integrated task and motion planning,” _Annual review of control, robotics, and autonomous systems_ , vol. 4, pp. 265–293, 2021\.
  * [18] X. Huang, Y. Zhang, J. Ma, W. Tian, R. Feng, Y. Zhang, Y. Li, Y. Guo, and L. Zhang, “Tag2text: Guiding vision-language model via image tagging,” _arXiv preprint arXiv:2303.05657_ , 2023.
  * [19] Y. Zhang, X. Huang, J. Ma, Z. Li, Z. Luo, Y. Xie, Y. Qin, T. Luo, Y. Li, S. Liu _et al._ , “Recognize anything: A strong image tagging model,” _arXiv preprint arXiv:2306.03514_ , 2023.