#  SceneVerse:
Scaling 3D Vision-Language Learning for Grounded Scene Understanding

Baoxiong Jia∗ Yixin Chen∗ Huanyue Yu Yan Wang Xuesong Niu
Tengyu Liu Qing Li Siyuan Huang
Beijing Institute for General Artificial Intelligence (BIGAI)
https://scene-verse.github.io

###### Abstract

3D vision-language grounding, which focuses on aligning language with the 3D
physical environment, stands as a cornerstone in the development of embodied
agents. In comparison to recent advancements in the 2D domain, grounding
language in 3D scenes faces several significant challenges: (i) the inherent
complexity of 3D scenes due to the diverse object configurations, their rich
attributes, and intricate relationships; (ii) the scarcity of paired 3D
vision-language data to support grounded learning; and (iii) the absence of a
unified learning framework to distill knowledge from grounded 3D data. In this
work, we aim to address these three major challenges in 3D vision-language by
examining the potential of systematically upscaling 3D vision-language
learning in indoor environments. We introduce the first million-scale 3D
vision-language dataset, SceneVerse, encompassing about $68$K 3D indoor scenes
and comprising $2.5$M vision-language pairs derived from both human
annotations and our scalable scene-graph-based generation approach. We
demonstrate that this scaling allows for a unified pre-training framework,
Grounded Pre-training for Scenes (GPS), for 3D vision-language learning.
Through extensive experiments, we showcase the effectiveness of GPS by
achieving state-of-the-art performance on all existing 3D visual grounding
benchmarks. The vast potential of SceneVerse and GPS is unveiled through zero-
shot transfer experiments in the challenging 3D vision-language tasks.

Figure 1:  Overview of SceneVerse. A million-scale 3D vision-language dataset
that comprises over $68$K various 3D indoor scenes and $2.5$M aligned scene-
language pairs in the form of scene caption, object caption, and object
referral.

∗ indicates equal contribution.

##  1 Introduction

The foundation of human cognitive development lies in the grounding of
language within the physical world [73, 46, 97]. Recent progress in Large
Language Models (LLMs) [11, 75], often referred to as “foundation models”
[10], has markedly promoted the alignment between vision and language [66, 3,
51] through utilizing billion-scale vision-language datasets [71, 96].
Nonetheless, with these advancements predominantly focusing on the 2D domain,
the grounded understanding of 3D physical environments remains in an incipient
stage [16, 1, 5]. Recognizing the pivotal role of grounded 3D experiences in
shaping human cognition [7, 8] and the delayed research development in this
area, there is a compelling need to intensify the exploration into the vision-
language learning challenge, specifically in the context of 3D scenes.

Seeking insights from the 2D vision-language (2D-VL) achievements, a major
factor to the success was the notable scale-up of paired vision-language data
[45, 15, 71]. However, applying these principles directly from 2D to 3D is
fraught with challenges. Primarily, 3D data collection heavily relies on the
scanning device, making it inherently much more complex and expensive than
gathering 2D images. Despite steady efforts to increase the volume of 3D scene
data [23, 87, 58, 9], most datasets remain limited to thousands of scenes,
substantially lagging behind the scale of existing 2D datasets. This gap is
further widened by the inherent complexities of 3D scenes, which feature a
multitude of object instances with diverse attributes, varying arrangements,
and intricate inter-object relationships. These unique aspects of 3D scenes
not only make the accurate description of objects and their relations more
challenging but also considerably increase the number of language descriptions
required for thorough scene depiction. Consequently, this presents a
significant challenge in obtaining a sufficient supply of high-quality paired
scene-language data crucial for grounded scene understanding.

To confront these challenges, we propose consolidating current efforts to
build up SceneVerse, the first million-scale dataset aimed at advancing 3D
vision-language (3D-VL) learning for grounded scene understanding. At the
scene level, we unify 3D scene data from existing datasets [23, 78, 9, 67, 40]
and supplement the collection with synthetic scenes [95, 27]. This compilation
represents the most extensive 3D scene data gathered to date, amounting to
$68,406$ scenes for grounding. Additionally, we propose an automated
generation pipeline utilizing 3D scene graphs [4, 79] and LLMs to create
comprehensive, high-quality scene-language pairs. This refined collection,
including $190,836$ human annotated pairs and totaling $2.5$M scene-language
pairs, provides detailed and comprehensive portrayals of both object-level and
scene-level descriptions within the 3D scene.

We thoroughly investigate the potential offered by the data scale-up in
SceneVerse with large-scale pre-training. Specifically, we present a novel and
unified pre-training framework, Grounded Pre-training for Scenes (GPS), which
is designed with scene-level and object-level alignment objectives and devoid
of auxiliary losses and designs. Through multi-level contrastive alignment, we
observe significant performance improvements across all existing 3D visual
grounding benchmarks, achieving new state-of-the-art results through a simple
and effective pre-training process. Additionally, we unveil the vast
possibilities offered by SceneVerse and GPS in 3D-VL tasks in a zero-shot
transfer setting. At last, we provide a more comprehensive understanding of
the data-scaling effects in SceneVerse through extensive ablative experiments
to point out future directions.

Our main contributions can be summarized as follows:

  * •

We introduce SceneVerse, the first million-scale 3D-VL dataset for grounded
scene understanding. SceneVerse encompasses $68$K 3D scenes coupled with
$2.5$M scene-language pairs, sourced through a combination of human annotation
and automated generation methods. This represents a significant improvement in
terms of data diversity and scale compared to prior datasets.

  * •

We propose GPS, an efficient transformer-based model trained with multi-level
scene-text alignment that achieves state-of-the-art results on all existing
3D-VL grounding benchmarks, benefiting from pre-training on multi-level scene-
language pairs in SceneVerse.

  * •

We demonstrate that with the data scale-up and model design, our pre-trained
models exhibit emerging zero-shot generalization capabilities in grounded
scene understanding, paralleling the successes seen in 2D-VL models.

##  2 Related Work

##### Datasets for Grounded 3D Understanding

Obtaining aligned 3D-language data is an inherently difficult task. In 3D
object modeling, pioneering works like ShapeNet [14] sourced 3D assets from
online repositories, leading to follow-up proliferation of high-quality 3D
object datasets [60, 22, 81]. Notably, recent developments include internet-
scale data collection with Objaverse [26, 25], accompanied by the integration
of object-level captions [83] for 3D-language alignment. Models trained on
these datasets demonstrate an enhanced understanding of objects, evident in
classification [52], generation [53], and captioning tasks [55].

In contrast, developing datasets for grounded 3D scene understanding is even
more challenging due to the extensive requirements for scene acquisition and
annotation. Existing works curate RGB-D and scanned indoor scene datasets [23,
13, 78, 9, 67, 58] initially used for benchmarking classical grounding tasks
like 3D object detection and segmentation [30, 42, 59, 77, 72]. These
semantically labeled scenes are subsequently applied in fine-grained scene
grounding tasks like object referral [16, 1, 93], captioning [20, 88, 17, 19],
vision-language-navigation [56, 80, 63, 38] and reasoning [5, 57, 37]. Recent
work exploits the representation of 3D scene graphs (3DSGs) [4, 79, 69], which
concisely describes scenes with hierarchical structures. This representation
is notably advantageous for planning [2, 68] and captioning [33], owing to its
compatibility with LLMs. Nevertheless, as shown in Tab. 1, these datasets are
significantly constrained in both scene and language scales, underscoring the
need for scaling up fine-grained scene-language-aligned data to enhance
grounded scene understanding.

##### Vision-Language Learning

Recent years have witnessed tremendous progress in 2D vision-language learning
[76, 66, 70, 3, 49, 51, 24], empowered by transformer-based pre-training
models [28, 11, 62] and large-scale image-language datasets [15, 71]. A
central theme across language and 2D-VL domains is the effectiveness of data
scaling [43], as demonstrated by improved alignment and expanded capabilities
in open-vocabulary understanding [50, 47, 32, 44] through a simplified
contrastive pre-training pipeline [66].

However, in grounded scene understanding, the primary challenge for models has
been the limited availability of paired 3D scene-language data, which
restricts the application of insights gained from 2D-VL. Current models for 3D
scene grounding [35, 94, 86, 6, 54, 40, 41, 18, 82] heavily rely on task-
specific knowledge in both model and loss designs or advanced optimization
strategies [98]. To bridge this gap, there has been a growing emphasis on
employing pre-trained 2D-VL models for 3D-VL [91, 34, 83, 36, 92, 64, 74].
Nonetheless, these models predominantly draw on information available from
2D-VL models (_e.g_., object attribute, affordance, _etc_.), falling short on
capturing crucial information like object spatial relationships, which are
only attainable through 3D data. This urges the need for a multi-level
alignment between language and 3D scenes, particularly regarding 3D-specific
information. Considering the nascent stage of existing 3D pre-training methods
[29, 84, 98], we believe SceneVerse and GPS have the potential to spearhead
new avenues in 3D-VL research.

Figure 2: SceneVerse collection and statistics. Given a 3D scene (a), our
automated pipeline (c) generates three types of description including scene
caption, object caption and object referral. (b) The comparison of different
language sources and data composition. Table 1: Comparison of SceneVerse with
existing 3DVL Datasets. SceneVerse expands the data scale of prior work by an
order of magnitude. Anno.: human annotations. Syn.: template or LLM generated
descriptions.

Dataset | 3D Data | Language | Total
---|---|---|---
Scene | Object | Anno. | Syn.
ScanRefer[16] | $|$ | $|$ | 52K | - | 52K
ReferIt3D[1] | 42K | 200K | 242K
ScanQA[5] | 1.5K | 33K | 27K | - | 27K
SQA3D[57] | $|$ | $|$ | - | 33K | 33K
Multi3DRefer[93] | 52K | 10K | 62K
Cap3D[55] | - | 666K | 58K | 666K | 724K
ScanScribe[98] | 3K | 56K | 94K | 184K | 278K
SceneVerse | 68K | 1.5M | 190K | 2.3M | 2.5M

##  3 SceneVerse

SceneVerse is the first million-scale dataset designed for grounded scene
understanding. Our 3D scenes are curated from diverse existing datasets of
both real and synthetic environments. Harnessing the power of 3D scene graphs
and LLMs, we introduce an automated pipeline to generate comprehensive and
high-quality language for both object-level and scene-level descriptions. We
additionally incorporate the most extensive human-annotated object referrals
to date, providing new training sources and benchmarks in this field.

###  3.1 Scene Curation

To address the scarcity of available 3D scene data, we construct SceneVerse by
unifying 3D scene data from various existing datasets. We use real-world scene
datasets, including ScanNet [23], ARKitScenes [9], HM3D [67], 3RScan [78] and
MultiScan [58], alongside synthetic environments from Structured3D [95] and
ProcTHOR [27]. The inclusion of these synthetic datasets is mainly motivated
by their potential as scalable data sources for 3D-VL alignment. To ensure
cohesion across various sources, we conduct preprocessing steps such as room
segmentation, point subsampling, axis alignment, normalization, and semantic
label alignment. Each scan is represented by a point cloud
$\mathrm{P}\in\mathbb{R}^{N\times 8}$, wherein each point is defined by its 3D
coordinates, RGB color, instance id and semantic label. In total, we curate
$68,406$ 3D scenes in SceneVerse.

###  3.2 3D Scene Graph Construction

Our 3D scene graph is defined as a set of tuples
$\mathcal{G}=(\mathcal{V},\mathcal{E})$, where the nodes $\mathcal{V}$
comprises
$\mathcal{V}_{1}\bigcup\mathcal{V}_{2}\bigcup\dots\bigcup\mathcal{V}_{K}$,
with $\mathcal{V}_{k}$ representing the set of nodes at a particular
hierarchical level. Each node $v$ represents one distinct 3D object instance,
parameterized by its centroid ${\bm{p}}_{i}\in\mathbb{R}^{3}$ and bounding box
size of ${\bm{b}}_{i}=(b_{x},b_{y},b_{z})\in\mathbb{R}^{3}$. The edges
$\mathcal{E}$ represent spatial relationships between nodes.

To construct the scene graph $\mathcal{G}$, we first instantiate the nodes
with the instance annotation from the point clouds and assign object classes
with their corresponding semantic labels. Following prior work[1, 79], we
consider the following spatial relations.

##### Vertical proximity

This encompasses both in-contact relationships (_e.g_., support, inside,
embed), and non-contact ones (_e.g_., above, below).

##### Horizontal proximity

Horizontal relationships describe the proximity relations like in front of,
next to, behind, _etc_. Relationships like left, right are contextually
dependent on a reference view, where another anchor object is utilized to
establish the view direction. The distance between the two objects is also
calculated to describe whether the objects are far or near in space.

##### Multi-object Relationships

This models the spatial arrangement of multiple objects, _e.g_., align and
between.

The node hierarchy is decided by the support relationship. We traverse all the
object nodes to calculate spatial relationships, which undergo an automatic
verification procedure to rectify incorrect ones. For a more detailed
description of the scene graph construction and relationship determination,
please refer to Sec. A.2.

###  3.3 Language Generation with LLMs

The scene-language pairs in SceneVerse aim to capture varying aspects of the
3D scene, which include detailed object attribute descriptions in object
captioning, spatial relationships between objects in object referral, and
global scene descriptions in scene captioning. Based on the 3D scene graph, we
utilize both templates and LLMs to automatically generate descriptions on
these three granularities.

##### Object Captioning

Object captions aim to provide detailed descriptions of an object’s visual and
physical properties, facilitating object-level grounding with its distinctive
features. Given the multi-view images, we utilize the point cloud of the
object $v\in\mathcal{V}$ to identify its occurrence in the images through
rendering. The images are then cropped with the rendered bounding boxes and
processed through BLIP2 [48] to generate initial object captions. To refine
the captions, we select the top 10 sentences with the highest CLIP [66]
similarity score and minimal occlusion. The selected sentences are fed into a
LLM to obtain a coherent summary of the object captions. In this process, we
explicitly instruct the language model to identify and correct the potential
errors. The detailed object captioning pipeline is illustrated in Sec. A.3.

##### Object Referral

Object relationship captions refer to objects by articulating their spatial
relationships in the scene. Spatial relationship triplets
$(v_{i},v_{j},e_{ij})$ are first extracted from the constructed 3D scene
graph. We design various templates to generate descriptions for each
relationship type, assigning the entities in the form of
$(\text{{\color[rgb]{0,0.88,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.88,0}\pgfsys@color@cmyk@stroke{0.91}{0}{0.88}{0.12}\pgfsys@color<EMAIL_ADDRESS>object}},\text{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}spatial-
relation}},\text{{\color[rgb]{1,0.58,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.58,0}\pgfsys@color@cmyk@stroke{0}{0.42}{1}{0}\pgfsys@color<EMAIL_ADDRESS>object(s)}})$. This results in examples like “the chair is next to the
armchair”, “facing the sofa, there is a suitcase far to the right of the
shoes”, and “the fridge is between cabinet and sofa”. To add complexity to the
template-based descriptions, we design "star-reference" templates, where the
reference to the target object is generated by describing its relationship to
3 randomly chosen adjacent objects in the scene graph. Our designed templates
span passive and active tenses, as well as inversion clauses, contributing to
the richness of the generated text. To enhance the naturalness of the
descriptions, we employ LLM for sentence rephrasing. Fig. 2 presents
statistics for the descriptions before and after rephrasing.

##### Scene Captioning

The scene-level captions emphasize global information, portraying the key
objects in the scene along with their attributes and functionalities. We
leverage the constructed 3D scene graph and prompt LLMs to generate these
captions. To enhance the diversity of scene captions, we utilize a subgraph
sampling strategy, where a subset of edges and nodes are randomly sampled as
the scene context. The object counts are also provided as LLM prompts,
together with the room type and object attributes if such annotations are
available in the dataset.

###  3.4 Referral Annotation by Humans

In addition to automatically generated scene-text pairs, SceneVerse includes
the most comprehensive set of human-annotated, context-rich object referrals
to date, serving as a valuable benchmark for assessing grounded scene
understanding capabilities. The human annotations contain $96,863$
descriptions in ARKitScenes [9], HM3D [67] and MultiScan [58]. During the
annotation process, one human annotator was assigned to write at least 20
words to distinctly refer to a single 3D object within a 3D scene. Each
referral text then undergoes independent verification by two additional
reviewers, both mandated to accurately locate the referenced object based on
the 3D scene and the annotated referral text. Any object referrals that do not
pass the verification by either reviewer are flagged for re-annotation.

###  3.5 Details and Statistics

In total, SceneVerse comprises a total of $68,406$ room-level 3D scans, with
the source composition shown in Fig. 2 (b). The dataset contains $1.5$M object
instances, comprising $21$ types of relationships following prior work [1,
79]. For the language descriptions, we generate $1$M template-based texts and
$1$M sentences by LLM rephrased by Llama [75] and GPT-3.5 [61]. All the
rephrasing and summary prompts, along with the complete set of relationships,
are detailed in Sec. A.3. To verify the efficacy of our automated language
generation pipeline, we conduct a quality check (QC) where 12K generated
object-level descriptions are randomly selected for human verification,
achieving a $96.93\%$ pass rate. This shows the capability of our proposed
scene-graph-based generation approach to produce high-quality language
descriptions, laying a robust foundation for future scalability.

##  4 Grounded Pre-training for Scenes

Figure 3: Overview of our proposed GPS model. We leverage contrastive
alignment in three levels $\mathcal{L}_{\text{obj}}$,
$\mathcal{L}_{\text{scene}}$, and $\mathcal{L}_{\text{ref}}$ as well as a
masked language modeling objective $\mathcal{L}_{\text{MLM}}$ for model
learning.

In this section, we introduce GPS, an efficient transformer-based model
trained with multi-level contrastive losses for aligning 3D scenes and texts.
As shown in Fig. 3, we echo the language descriptions collected at different
levels to form scene-language pairs at both object-level, referral-object-
level, and scene-level for contrastive objectives in GPS. We describe the
design of each level in the following sections.

###  4.1 Object-level Grounding

Given a 3D scene point cloud $\mathcal{S}$, we use an off-the-shelf 3D object
segmentation model to decompose it into a bag of $N$ objects
$\mathcal{S}=\left\\{{\bm{o}}_{1},{\bm{o}}_{2},\cdots,{\bm{o}}_{n}\right\\}_{i=1}^{N}$.
We extract object features $\\{{\bm{f}}^{O}_{i}\\}$ with an object point cloud
encoder and text features $\\{{\bm{f}}^{T}_{i}\\}$ by feeding object-captions
$\\{{\bm{T}}^{\text{obj}}_{i}\\}$ into a frozen language model. Following
[83], we perform cross-modal alignment on the object features and text
features via:

$\displaystyle\mathcal{L}_{\text{obj}}=-\frac{1}{2}\sum_{(p,q)}$
$\displaystyle\left(\log\frac{\exp{\left(D^{\text{obj}}(p,q)\right)}}{\sum_{r}\exp{\left(D^{\text{obj}}(p,r)\right)}}+\right.$
(1) $\displaystyle\left.\ \ \
\log\frac{\exp{\left(D^{\text{obj}}(p,q)\right)}}{\sum_{r}\exp{\left(D^{\text{obj}}(r,q)\right)}}\right),$

where $D^{\text{obj}}(p,q)=({\bm{f}}^{O}_{p}{\bm{f}}^{T}_{q}/\tau)$ denotes
the dot product between object and text features and $(p,q)$ denotes a pair of
aligned object-text pair in the training batch and $r$ iterates over all
object-text pairs in the training batch. Similar to CLIP [66], we use a
learnable temperature parameter $\tau$ to facilitate model learning.

###  4.2 Scene-level Grounding

With aligned object features, we encode the scene by incorporating object
spatial locations into the extracted object features. Specifically, we use a
spatial transformer model to encode extracted object features
$\\{{\bm{f}}^{O}_{i}\\}$ with their spatial location features
$\\{{\bm{l}}_{i}\\}$ following [18, 98]:

${\bm{f}}^{S}=\mathrm{SpatialAttn}(\\{{\bm{f}}_{i}^{O}\\},\\{{\bm{l}}_{i}\\})$

where $\\{{\bm{f}}_{i}^{S}\\}$ denotes the feature of object ${\bm{o}}_{i}$
after encoding with spatial location features. To perform scene-level
alignment, we operate on these scene-level object features
$\\{{\bm{f}}_{i}^{S}\\}$ and align it with the scene caption
${\bm{T}}^{\text{scene}}$. Specifically, we feed the object features into a
projection layer and use max-pooling over all object features to obtain the
scene feature ${\bm{g}}^{S}$. Similar to object-level grounding, we pass the
scene caption through a tunable language model to obtain text feature
${\bm{g}}^{T}$ and perform scene-level contrastive alignment through:

$\displaystyle\mathcal{L}_{\text{scene}}=-\frac{1}{2}\sum_{(p,q)}$
$\displaystyle\left(\log\frac{\exp{\left(D^{\text{scene}}(p,q)\right)}}{\sum_{r}\exp{\left(D^{\text{scene}}(p,r)\right)}}+\right.$
(2) $\displaystyle\left.\ \ \
\log\frac{\exp{\left(D^{\text{scene}}(p,q)\right)}}{\sum_{r}\exp{\left(D^{\text{scene}}(r,q)\right)}}\right),$

where $D^{\text{scene}}(p,q)=({\bm{g}}_{p}^{S}{\bm{g}}_{q}^{T}/\tau)$ denotes
the dot product between scene feature ${\bm{g}}_{p}^{S}$ and scene caption
feature ${\bm{g}}_{q}^{T}$ for each pair of aligned scene-text pairs in the
training batch and $r$ iterates over all scene-text pairs in the training
batch.

###  4.3 Referral-object-level Grounding

To model the relationships revealed in referring expressions, we employ a
self-attention-based reasoning transformer for grounding object referrals in
scenes. This transformer takes in scene-object features
$\\{{\bm{f}}^{S}_{i}\\}$ and an object referral ${\bm{T}}^{\text{ref}}$ and
performs self-attention to learn relationships between text descriptions and
object relationships. We use the same tunable language encoder as in scene-
level grounding for extracting per-object referral features. We pass this text
feature together with scene-object features into the self-attention
transformer to obtain the aligned object features ${\bm{h}}^{S}_{i}$ and the
sentence-level referral feature ${\bm{h}}^{T}$. We then perform the referral-
object-level contrastive alignment following:

$\displaystyle\mathcal{L}_{\text{ref}}=-\log\frac{\exp{\left(\bar{{\bm{h}}}^{S}{\bm{h}}^{T}/\tau\right)}}{\sum_{p}\exp{\left({\bm{h}}_{p}^{S}{\bm{h}}^{T}/\tau\right)}},$
(3)

where $\bar{{\bm{h}}}^{S}$ denotes the feature of the referred object, $p$
iterates over all objects within the same scene. Notably, in contrast to
inter-scene contrast that was done in object- and scene-level alignment, we
force the selection of positive pairs to be within the same scene to provide
intra-scene contrast for fine-grained object grounding. This mimics the
success of intra-image and inter-image contrasts commonly used for region-word
alignment in 2D-VL models [90].

To learn the multi-level alignment between 3D scenes and language, we first
train the point cloud encoder with object-level grounding objects to obtain a
good feature initialization for grounding objects in scenes. During the scene
grounding stage, we train our inter- and intra-scene objectives together with
a mask language modeling loss $\mathcal{L}_{\text{MLM}}$ over the inputted
object-referral texts to tune the parameters within the language encoder and
self-attention transformer. Above all, the learning of GPS could be summarized
as optimizing the following objective:

$\mathcal{L}=\mathcal{L}_{\text{obj}}+\mathcal{L}_{\text{scene}}+\mathcal{L}_{\text{ref}}+\mathcal{L}_{\text{MLM}}.$

##  5 Experiments

In this section, we present the evaluation results addressing the following
questions:

  * •

How effective is the data scaling in SceneVerse for 3D visual grounding? Does
the scale-up work for general pre-training based 3D-VL models?

  * •

How well is the GPS pre-training pipeline? Does it exhibit similar properties
of 2D-VL models in 3D-VL tasks?

  * •

What potentials are offered by SceneVerse and GPS for future research? What is
missing?

In the following sections, we describe in detail about model performance
regarding these key topics. Due to the page limit, we direct readers to the
Appendices B and C for implementation details and more experimental analyses.

###  5.1 3D Visual Grounding

Table 2: 3D Visual Grounding results on Nr3D, Sr3D, and ScanRefer. We use
“direct” for our model trained on SceneVerse with no additional fine-tune
head, and “fine-tune” for the data-specific fine-tuned version of our model.
We highlight the best results in bold.

Method | Nr3D | Sr3D | ScanRefer<EMAIL_ADDRESS>
---|---|---|---
Overall | Easy | Hard | V-Dep. | V-Indep. | Overall | Easy | Hard | V-Dep. | V-Indep. | Overall | Unique | Multiple
3DVG-Trans [94] | 40.8 | 48.5 | 34.8 | 34.8 | 43.7 | 51.4 | 54.2 | 44.9 | 44.6 | 51.7 | 34.7 | 60.6 | 28.4
TGNN [39] | 37.3 | 44.2 | 30.6 | 35.8 | 38.0 | 45.0 | 48.5 | 36.9 | 45.8 | 45.0 | 29.7 | 56.8 | 23.2
TransRefer3D [35] | 48.0 | 56.7 | 39.6 | 42.5 | 50.7 | 57.4 | 60.5 | 50.2 | 49.9 | 57.7 | - | - | -
InstanceRefer [89] | 38.8 | 46.0 | 31.8 | 34.5 | 41.9 | 48.0 | 51.1 | 40.5 | 45.8 | 48.1 | 32.9 | 66.8 | 24.7
FFL-3DOG [31] | 41.7 | 48.2 | 35.0 | 37.1 | 44.7 | - | - | - | - | - | 34.0 | 67.9 | 25.7
LAR [6] | 48.9 | 58.4 | 42.3 | 47.4 | 52.1 | 59.4 | 63.0 | 51.2 | 50.0 | 59.1 | - | - | -
SAT [86] | 56.5 | 64.9 | 48.4 | 54.4 | 57.6 | 57.9 | 61.2 | 50.0 | 49.2 | 58.3 | 30.1 | 50.8 | 25.2
3D-SPS [54] | 51.5 | 58.1 | 45.1 | 48.0 | 53.2 | 62.6 | 56.2 | 65.4 | 49.2 | 63.2 | 37.0 | 66.7 | 29.8
3DJCG [12] | - | - | - | - | - | - | - | - | - | - | 37.3 | 64.3 | 30.8
BUTD-DETR [41] | 54.6 | 60.7 | 48.4 | 46.0 | 58.0 | 67.0 | 68.6 | 63.2 | 53.0 | 67.6 | 39.8 | 66.3 | 35.1
MVT [40] | 59.5 | 67.4 | 52.7 | 59.1 | 60.3 | 64.5 | 66.9 | 58.8 | 58.4 | 64.7 | 33.3 | 66.5 | 25.3
ViL3DRel [18] | 64.4 | 70.2 | 57.4 | 62.0 | 64.5 | 72.8 | 74.9 | 67.9 | 63.8 | 73.2 | 37.7 | 68.6 | 30.7
EDA [82] | 52.1 | 58.2 | 46.1 | 50.2 | 53.1 | 68.1 | 70.3 | 62.9 | 54.1 | 68.7 | 42.3 | 68.6 | 37.6
3D-VisTA (scratch) [98] | 57.5 | 65.9 | 49.4 | 53.7 | 59.4 | 69.6 | 72.1 | 63.6 | 57.9 | 70.1 | 41.5 | 70.9 | 34.8
3D-VisTA [98] | 64.2 | 72.1 | 56.7 | 61.5 | 65.1 | 76.4 | 78.8 | 71.3 | 58.9 | 77.3 | 45.8 | 75.1 | 39.1
Ours (scratch) | 58.7 | 67.0 | 50.9 | 55.8 | 59.8 | 68.4 | 70.5 | 63.4 | 53.1 | 69.0 | 40.4 | 71.3 | 34.7
Ours (pre-train) | 55.2 | 62.8 | 48.0 | 45.5 | 58.8 | 74.1 | 76.4 | 68.5 | 54.1 | 75.0 | 47.1 | 77.4 | 41.6
Ours (fine-tuned) | 64.9 | 72.5 | 57.8 | 56.9 | 67.9 | 77.5 | 80.1 | 71.6 | 62.8 | 78.2 | 48.1 | 77.9 | 42.7

##### Settings

We evaluate our model on three commonly-used datasets for 3D visual grounding:
ScanRefer [16], Nr3D, and Sr3D [1]. For Nr3D and Sr3D, we follow Achlioptas
_et al_. [1] and report the grounding accuracies of models using ground-truth
object masks. For ScanRefer, we follow Zhu _et al_. [98] and use Mask3D [72]
to generate object proposals. Results are reported as<EMAIL_ADDRESS>to
evaluate the correctness of predictions whose object bounding boxes overlap
the ground truth with IoU $>0.5$. For comparisons, we compare with existing
baselines by providing the results of pre-trained GPS and dataset-specific
fine-tuned GPS. Please see more details in the Appendix C.

##### Results and Analyses

As shown in Tab. 2, GPS trained on SceneVerse achieves state-of-the-art
results on all existing 3D-VL grounding benchmarks. Initially, when GPS is
trained directly on the training sets of benchmark datasets, labeled as Ours
(scratch), it underperforms compared to existing models that employ more
complex structures or loss designs. This result underscores the data-intensive
nature of the contrastive alignment paradigm. However, when presented with
extensive training data in SceneVerse, the results of our model without
additional fine-tuning, _i.e_., Ours (pre-train), significantly improves and
already achieves state-of-the-art results on benchmarks like ScanRefer.
Moreover, the dataset-specific fine-tuned model, _i.e_., Ours (fine-tuned),
consistently outperforms existing baselines with only a simple projection MLP
added on top of the pre-trained model, jointly optimized during fine-tuning
without any other auxiliary architecture or loss objective. These results
underscore the strong potential of both the SceneVerse and GPS for 3D-VL
tasks.

###  5.2 Zero-Shot Transfer

##### Settings

To better evaluate the effectiveness of both the SceneVerse data and the GPS
model, we further perform zero-shot transfer experiments to test the models’
capability in 4 benchmarks, ScanRefer, Sr3D, Nr3D, and SceneVerse-val. We
create SceneVerse-val using $8.5K$ annotated object referrals of $271$ scenes
in MultiScan, and randomly split the scenes following a 4:1 train / test split
for creating the held-out test set. We mainly consider 2 specific transfer
settings in our experiments: (i) zero-shot: models trained by removing all the
scenes from the target dataset, tested on held-out unseen scenes, and (ii)
zero-shot text: Models trained on data that include the 3D scenes from
training set of the target dataset, yet tested exclusively with unseen scene-
text distribution. Specifically, for the zero-shot text setting, we use the
generated texts in SceneVerse as fine-tuning sources for the zero-shot model.
We mainly compare our model against a recent pre-training-based model
3D-VisTA. See more details on experimental setting and implementation in the
Appendix C.

Table 3: Zero-shot transfer results on established benchmarks.

Method | Nr3D | Sr3D |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---|---
3D-VisTA (scratch) | 57.5 | 69.6 | 45.9 | 41.5
3D-VisTA (zero-shot) | 35.2 | 31.2 | 33.2 | 29.6
3D-VisTA (zero-shot text) | 43.1 | 36.1 | 41.1 | 36.4
Ours (scratch) | 58.7 | 68.4 | 44.5 | 40.4
Ours (zero-shot) | 32.4 | 33.3 | 35.2 | 31.1
Ours (zero-shot text) | 41.9 | 38.1 | 40.7 | 35.8

Table 4: Zero-shot transfer on SceneVerse-val. We evaluate models following
settings in Nr3D/Sr3D using GT object proposals.

Method | Overall | Easy | Hard | V-Dep. | V-Indep.
---|---|---|---|---|---
3D-VisTA (scratch) | 40.7 | 53.1 | 21.6 | 37.3 | 44.3
3D-VisTA (zero-shot) | 52.9 | 59.6 | 35.4 | 53.7 | 52.2
3D-VisTA (zero-shot text) | 58.1 | 70.0 | 39.6 | 52.5 | 64.1
Ours (scratch) | 38.5 | 50.2 | 20.8 | 33.7 | 43.9
Ours (zero-shot) | 59.2 | 69.4 | 44.0 | 53.1 | 66.3
Ours (zero-shot text) | 60.6 | 70.9 | 45.1 | 54.8 | 67.3

##### Results and Analyses

We present the results of zero-shot transfer experiments in Tab. 3 and Tab. 4
with the following key observations:

  * •

Our GPS model demonstrates superior generalization to unseen scenes compared
to the 3D-VisTA model. In zero-shot transfer scenarios, our model consistently
outperforms 3D-VisTA across established benchmarks and SceneVerse-val. This
indicates the effectiveness of contrastive alignment over traditional
classification objectives, aligning with the advancements seen in 2D-VL models
for open-vocabulary grounding and transfer capabilities

  * •

SceneVerse dataset substantially enhances 3D-VL grounding capabilities through
zero-shot transfer, especially when provided with relatively limited training
data, _i.e_., SceneVerse-val. As demonstrated in Tab. 4, there is a
significantly improved performance when comparing models trained on SceneVerse
in a zero-shot manner to those trained from scratch. This indicates that
SceneVerse can effectively capture knowledge for general 3D scene grounding.
Consequently, this underscores its potential as a go-to pre-training dataset
for 3D-VL tasks.

  * •

The impact of our extensive collection and scalable generation of scene-text
pairs is further evidenced by the results in the zero-shot text setting.
Notably, as shown in Tab. 3, these automatically generated scene-text pairs
supply ample knowledge for comprehending the scene distribution. This
contributes significantly to the substantial improvement over the zero-shot
performance.

Figure 4: Model performance v.s. data scale. Models consistently improve in
both the pre-train and zero-shot transfer settings on ScanRefer and
SceneVerse-val with data scaling-up.

###  5.3 Ablative Studies and Discussions

In this section, we present ablative studies focused primarily on the data
collected in SceneVerse. Our aim is to elucidate the effects of data scaling
more clearly. For discussions regarding ablation studies on model
architecture, readers are referred to the Appendix D. The following points are
specifically discussed in this section.

##### How important is data-scaling?

We conduct ablation studies over the amount of data used while pre-training
GPS. We consider the model trained with $\frac{1}{8}$, $\frac{1}{4}$,
$\frac{1}{2}$ of SceneVerse to show the effectiveness of data-scaling on model
performance in the pre-train and zero-shot transfer settings in ScanRefer and
SceneVerse-val. As shown in Fig. 4, we observe consistent performance
improvement over the increase of data scale for both settings. We provide
additional experiments in the Appendix D to show that such scaling effect is
not only beneficial for 3D-VL grounding but also for other 3D tasks like
semantic segmentation [72, 85].

##### How is the generated data compared with human-annotated data?

We assess the performance of models trained using various scene-text sources,
specifically focusing on their performance in the ScanRefer dataset without
additional fine-tuning. As shown in Tab. 5, models trained with our template-
based generated texts and Large Language Model (LLM)-refined texts show
significant improvements over models trained solely on ScanRefer. More
importantly, these variants of our model already achieve state-of-the-art
results compared with previous baselines. This indicates the effectiveness of
our text-generation pipeline. Finally, we observe that adding human-annotated
data is still beneficial for model performance. However, the improvement is
relatively marginal over models trained on our generated data.

##### What is the role of the synthetic scenes in this scale-up process?

With synthetic data providing large-scale and diverse scene data for 3D-VL
tasks, we evaluate the models’ domain transfer (Sim2Real) capability.
Specifically, we compare models trained on all real scenes in SceneVerse
against models trained exclusively on two synthetic subsets of SceneVerse,
_i.e_., Structured3D and ProcTHOR. As shown in Tab. 6, models trained on
synthetic subsets demonstrate remarkable performance on their corresponding
test sets while suffering when transferred to real or other synthetic scenes.
In contrast, the model trained on real scene-text pairs exhibits less severe
performance drops when generalizing to synthetic scenes. This result affirms
the domain gap between real and synthetic scenes in 3D-VL grounding and shows
that a simple scale-up in the amount of scenes is insufficient when the scene
naturalness can not be guaranteed. Considering the scalability of our quality-
ensured language generation and also the scaling effect shown in our
experiments, the rate-determining step for further scaling-up 3D-VL comes to
the collection of diverse, high-quality, and realistic scenes that capture
natural 3D scene distributions.

Table 5: Ablation on scene-text pair types used in training. We report model
results on ScanRefer with no additional finetuning.

Template | LLM | Anno. |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---|---
✗ | ✗ | ✗ | 43.5 | 38.4
✓ | ✗ | ✗ | 50.9 | 46.1
✓ | ✓ | ✗ | 51.1 | 46.3
✓ | ✓ | ✓ | 52.0 | 47.1

Table 6: Cross domain transfer results of models learned in real and synthetic
datasets without additional fine-tuning. “S3D” stands for Structured3D.

Real | Synthetic |  SceneVerse-val | S3D | ProcTHOR
---|---|---|---|---
All | ✗ | 64.8 | 37.1 | 43.4
✗ | S3D | 7.0 | 85.1 | 16.1
✗ | ProcTHOR | 4.2 | 16.3 | 91.0

##  6 Conclusion

In this work, we scale up 3D-VL in the context of grounded scene
understanding. We introduce SceneVerse, a million-scale 3D-VL dataset
encompassing various scenes and multi-level scene descriptions sourced from
both human annotation and our proposed scene-text generation approach.
Utilizing SceneVerse, we propose Grounded Pre-training for Scenes, a model
trained with multi-level scene-language contrastive alignment over the data
collected. Through extensive experiments, we show that GPS achieves state-of-
the-art results on all existing 3D-VL grounding tasks. We further conduct
zero-shot transfer experiments to show the improved generalization
performances of GPS trained on SceneVerse compared with previous baselines. We
hope our efforts and successful scale-up attempts in SceneVerse could pave the
way for a new research paradigm in 3D-VL.

##  7 Acknowledgement

The authors thank Yaowei Zhang from BIGAI for designing the result
visualization framework, Jiangyong Huang and Xiongkun Linghu from BIGAI for
suggestions on data generation and refinement, and colleagues from BIGAI for
their helpful discussions and suggestions.

## References

  * [1] Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas.  Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes.  In Proceedings of European Conference on Computer Vision (ECCV), 2020.
  * [2] Christopher Agia, Krishna Murthy Jatavallabhula, Mohamed Khodeir, Ondrej Miksik, Vibhav Vineet, Mustafa Mukadam, Liam Paull, and Florian Shkurti.  Taskography: Evaluating robot task planning over large 3d scene graphs.  In Proceedings of Conference on Robot Learning (CoRL), 2022.
  * [3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
  * [4] Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese.  3d scene graph: A structure for unified semantics, 3d space, and camera.  In Proceedings of International Conference on Computer Vision (ICCV), 2019.
  * [5] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe.  Scanqa: 3d question answering for spatial scene understanding.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [6] Eslam Bakr, Yasmeen Alsaedy, and Mohamed Elhoseiny.  Look around and refer: 2d synthetic semantics knowledge distillation for 3d visual grounding.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
  * [7] Lawrence W Barsalou.  Perceptual symbol systems.  Behavioral and brain sciences, 22(4):577–660, 1999.
  * [8] Lawrence W Barsalou.  Grounded cognition.  Annu. Rev. Psychol., 59:617–645, 2008.
  * [9] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al.  Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data.  In Proceedings of Advances in Neural Information Processing Systems Datasets and Benchmarks (NeurIPS Datasets and Benchmarks Track), 2021\.
  * [10] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  arXiv preprint arXiv:2108.07258, 2021.
  * [11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
  * [12] Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu.  3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [13] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  Proceedings of International Conference on 3D Vision (3DV), 2017\.
  * [14] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [15] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [16] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner.  Scanrefer: 3d object localization in rgb-d scans using natural language.  In Proceedings of European Conference on Computer Vision (ECCV), 2020.
  * [17] Dave Zhenyu Chen, Qirui Wu, Matthias Nießner, and Angel X Chang.  D3net: a speaker-listener architecture for semi-supervised dense captioning and visual grounding in rgb-d scans.  In Proceedings of European Conference on Computer Vision (ECCV), 2022.
  * [18] Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev.  Language conditioned spatial relation reasoning for 3d object grounding.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
  * [19] Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen.  End-to-end 3d dense captioning with vote2cap-detr.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [20] Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang.  Scan2cap: Context-aware dense captioning in rgb-d scans.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [21] Zhenyu Chen, Ronghang Hu, Xinlei Chen, Matthias Nießner, and Angel X Chang.  Unit3d: A unified transformer for 3d dense captioning and visual grounding.  In Proceedings of International Conference on Computer Vision (ICCV), 2023.
  * [22] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al.  Abo: Dataset and benchmarks for real-world 3d object understanding.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [23] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [24] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  arXiv preprint arXiv:2305.06500, 2023.
  * [25] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al.  Objaverse-xl: A universe of 10m+ 3d objects.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2023.
  * [26] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [27] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi.  Procthor: Large-scale embodied ai using procedural generation.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
  * [28] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2018.
  * [29] Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi.  Pla: Language-driven open-vocabulary 3d scene understanding.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [30] Zhipeng Ding, Xu Han, and Marc Niethammer.  Votenet: A deep learning label fusion method for multi-atlas segmentation.  In Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2019.
  * [31] Mingtao Feng, Zhen Li, Qi Li, Liang Zhang, XiangDong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang, and Ajmal Mian.  Free-form description guided 3d visual graph network for object grounding in point cloud.  In Proceedings of International Conference on Computer Vision (ICCV), 2021.
  * [32] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin.  Scaling open-vocabulary image segmentation with image-level labels.  In Proceedings of European Conference on Computer Vision (ECCV), 2022.
  * [33] Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al.  Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning.  arXiv preprint arXiv:2309.16650, 2023.
  * [34] Huy Ha and Shuran Song.  Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models.  In Proceedings of Conference on Robot Learning (CoRL), 2022.
  * [35] Dailan He, Yusheng Zhao, Junyu Luo, Tianrui Hui, Shaofei Huang, Aixi Zhang, and Si Liu.  Transrefer3d: Entity-and-relation aware transformer for fine-grained 3d visual grounding.  In Proceedings of ACM International Conference on Multimedia (MM), 2021.
  * [36] Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal Patel.  Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition.  In Proceedings of International Conference on Computer Vision (ICCV), 2023.
  * [37] Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan.  3d concept learning and reasoning from multi-view images.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [38] Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould.  Vln bert: A recurrent vision-and-language bert for navigation.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [39] Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, and Tyng-Luh Liu.  Text-guided graph neural networks for referring 3d instance segmentation.  In Proceedings of AAAI Conference on Artificial Intelligence (AAAI), 2021.
  * [40] Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang.  Multi-view transformer for 3d visual grounding.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [41] Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki.  Bottom up top down detection transformers for language grounding in images and point clouds.  In Proceedings of European Conference on Computer Vision (ECCV), 2022.
  * [42] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia.  Pointgroup: Dual-set point grouping for 3d instance segmentation.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [43] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  arXiv preprint arXiv:2001.08361, 2020.
  * [44] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  In Proceedings of International Conference on Computer Vision (ICCV), 2023.
  * [45] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  In International Journal of Computer Vision (IJCV), 2017.
  * [46] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman.  Building machines that learn and think like people.  Behavioral and brain sciences, 40:e253, 2017.
  * [47] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl.  Language-driven semantic segmentation.  In Proceedings of International Conference on Learning Representations (ICLR), 2022.
  * [48] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In ICML, 2023.
  * [49] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In Proceedings of International Conference on Machine Learning (ICML), 2022.
  * [50] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.  Grounded language-image pre-training.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [51] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2023.
  * [52] Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su.  Openshape: Scaling up 3d shape representation towards open-world understanding.  arXiv preprint arXiv:2305.10764, 2023.
  * [53] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In Proceedings of International Conference on Computer Vision (ICCV), 2023.
  * [54] Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu.  3d-sps: Single-stage 3d visual grounding via referred point progressive selection.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [55] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson.  Scalable 3d captioning with pretrained models.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2023.
  * [56] Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong.  Self-monitoring navigation agent via auxiliary progress estimation.  In Proceedings of International Conference on Learning Representations (ICLR), 2019.
  * [57] Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang.  Sqa3d: Situated question answering in 3d scenes.  In Proceedings of International Conference on Learning Representations (ICLR), 2023.
  * [58] Yongsen Mao, Yiming Zhang, Hanxiao Jiang, Angel Chang, and Manolis Savva.  Multiscan: Scalable rgbd scanning for 3d environments with articulated objects.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
  * [59] Ishan Misra, Rohit Girdhar, and Armand Joulin.  An end-to-end transformer model for 3d object detection.  In Proceedings of International Conference on Computer Vision (ICCV), 2021.
  * [60] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su.  Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [61] OpenAI.  Introducing chatgpt.  https://openai.com/blog/chatgpt, 2022.
  * [62] OpenAI.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [63] Alexander Pashevich, Cordelia Schmid, and Chen Sun.  Episodic transformer for vision-and-language navigation.  In Proceedings of International Conference on Computer Vision (ICCV), 2021.
  * [64] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al.  Openscene: 3d scene understanding with open vocabularies.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [65] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2017.
  * [66] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [67] Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al.  Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.  In Proceedings of Advances in Neural Information Processing Systems Datasets and Benchmarks (NeurIPS Datasets and Benchmarks Track), 2021\.
  * [68] Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf.  Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning.  In Proceedings of Conference on Robot Learning (CoRL), 2023.
  * [69] Antoni Rosinol, Andrew Violette, Marcus Abate, Nathan Hughes, Yun Chang, Jingnan Shi, Arjun Gupta, and Luca Carlone.  Kimera: From slam to spatial perception with 3d dynamic scene graphs.  International Journal of Robotics Research (IJRR), 2021.
  * [70] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
  * [71] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
  * [72] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe.  Mask3d: Mask transformer for 3d semantic instance segmentation.  In Proceedings of International Conference on Robotics and Automation (ICRA), 2023.
  * [73] Linda Smith and Michael Gasser.  The development of embodied cognition: Six lessons from babies.  Artificial life, 11(1-2):13–29, 2005.
  * [74] Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann.  Openmask3d: Open-vocabulary 3d instance segmentation.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2023.
  * [75] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [76] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill.  Multimodal few-shot learning with frozen language models.  Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021.
  * [77] Thang Vu, Kookhoi Kim, Tung M Luu, Thanh Nguyen, and Chang D Yoo.  Softgroup for 3d instance segmentation on point clouds.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [78] Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Nießner.  Rio: 3d object instance re-localization in changing indoor environments.  In Proceedings of International Conference on Computer Vision (ICCV), 2019.
  * [79] Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari.  Learning 3d semantic scene graphs from 3d indoor reconstructions.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [80] Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang.  Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [81] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al.  Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [82] Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang.  Eda: Explicit text-decoupling and dense alignment for 3d visual grounding.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [83] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese.  Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [84] Jihan Yang, Runyu Ding, Zhe Wang, and Xiaojuan Qi.  Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding.  arXiv preprint arXiv:2304.00962, 2023.
  * [85] Yu-Qi Yang, Yu-Xiao Guo, Jian-Yu Xiong, Yang Liu, Hao Pan, Peng-Shuai Wang, Xin Tong, and Baining Guo.  Swin3d: A pretrained transformer backbone for 3d indoor scene understanding.  arXiv preprint arXiv:2304.06906, 2023.
  * [86] Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo.  Sat: 2d semantics assisted training for 3d visual grounding.  In Proceedings of International Conference on Computer Vision (ICCV), 2021.
  * [87] Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai.  Scannet++: A high-fidelity dataset of 3d indoor scenes.  In Proceedings of International Conference on Computer Vision (ICCV), 2023.
  * [88] Zhihao Yuan, Xu Yan, Yinghong Liao, Yao Guo, Guanbin Li, Shuguang Cui, and Zhen Li.  X-trans2cap: Cross-modal knowledge transfer using transformer for 3d dense captioning.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [89] Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui.  Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring.  In Proceedings of International Conference on Computer Vision (ICCV), 2021.
  * [90] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao.  Glipv2: Unifying localization and vision-language understanding.  In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
  * [91] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li.  Pointclip: Point cloud understanding by clip.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [92] Renrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao, and Hongsheng Li.  Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders.  In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [93] Yiming Zhang, ZeMing Gong, and Angel X Chang.  Multi3drefer: Grounding text description to multiple 3d objects.  In Proceedings of International Conference on Computer Vision (ICCV), 2023.
  * [94] Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu.  3dvg-transformer: Relation modeling for visual grounding on point clouds.  In Proceedings of International Conference on Computer Vision (ICCV), 2021.
  * [95] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou.  Structured3d: A large photo-realistic dataset for structured 3d modeling.  In Proceedings of European Conference on Computer Vision (ECCV), 2020.
  * [96] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi.  Multimodal c4: An open, billion-scale corpus of images interleaved with text.  arXiv preprint arXiv:2304.06939, 2023.
  * [97] Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, et al.  Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense.  Engineering, 6(3):310–345, 2020.
  * [98] Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li.  3d-vista: Pre-trained transformer for 3d vision and text alignment.  In Proceedings of International Conference on Computer Vision (ICCV), 2023.

Supplementary Material

In Appendix A, we introduce more details of SceneVerse, including the 3D scene
preprocessing, scene graph construction and automatic language generation.
Appendix B presents more model and implementation details. Appendix C include
a more in-depth summary of the settings and implementations for the
experiments in the main paper, as well as the ablative study and additional
experiments on semantic segmentation to demonstrate the benefits of
SceneVerse.

##  Appendix A SceneVerse

###  A.1 3D Scenes

To address the scarcity of available 3D scene data, we construct SceneVerse by
unifying 3D scene data from various existing datasets. The curation involves
utilizing real-world scene datasets such as ScanNet [23], ARKitScenes [9],
HM3D [67], 3RScan [78] and MultiScan [58],in conjunction with synthetic
environments from Structured3D [95] and ProcTHOR [27]. The incorporation of
these synthetic datasets is primarily driven by their potential as scalable
data sources for 3D-VL alignment. To facilitate the training process, we
conduct the following preprocessing steps.

##### Room Segmentation

The 3D scenes in HM3D and ProcTHOR are released at the building level,
encompassing multiple rooms and sometimes spanning over 50 meters. To align
with existing benchmarks [16, 1], we leverage the associated metadata to
segment the 3D point cloud at the room level, facilitating subsequent
operations in scene graph construction and language description generation.
Additionally, we implement a filtering process to exclude extremely large
rooms and those with fewer than 4 objects in the scene.

##### Point Cloud Normalization

In order to mitigate the data disparities arising from diverse capture devices
across various data sources, we subsample each point cloud to a maximum of
$240,000$ points. Each point cloud then undergoes a transformation centered on
the central point on the floor, followed by rotation to align the room layout
with the axis following the approach by Chen _et al_. [18].

##### Semantic Label Alignment

Given the divergence in semantic label sets across different datasets, we
undertake a comprehensive effort to map all the object class labels to the
$607$ semantic labels in ScanNet [23] to facilitate close-vocabulary object
classification [65] in the existing model framework [98]. We construct the
mapping in each dataset through LLM and manual verification. Note that the
object-level grounding in GPS can directly deal with open-set object labels or
captions, similar to CLIP [36].

After the preprocessing, each scan is represented by a point cloud
$\mathrm{P}\in\mathbb{R}^{N\times 8}$, wherein each point is defined by its 3D
coordinates, RGB color, instance id and semantic label. In total, we curate
$68,406$ 3D scenes in SceneVerse.

###  A.2 3D Scene Graph Construction

In Sec. 3.2, we introduce an automated pipeline to construct 3D scene graphs
from point clouds. Here, we provide more implementation details and the
relationship definition.

####  A.2.1 Relationships

Table A.1: Relationships in SceneVerse. The 3D scene graph captures 21 types of relationships ranging in 4 categories. Category | Relation
---|---
$|$ | supported by
In-contact vertical | embedded into
placed in
$|$ | inside
| hanging on
affixed on
mounted on
Non-contact vertical | above
| higher than
below
lower than
| near(far) to the left of
near(far) to the right of
is behind
Horizontal | is in front of
close to
| adjacent to
besides
next to
Multi-object | between
aligned

Our 3D scene graph captures $21$ types of relations as shown in Tab. A.1. We
provide illustrations of how these relations are defined in the 3D space, as
can be seen in Fig. A.1.

####  A.2.2 Scene Graph Construction

Due to the inherent noise and incompleteness in the point cloud
representation, automatically extracting precise and comprehensive
relationships from the point clouds is a non-trivial task. Below we detail our
3D scene graph construction process, as outlined in Alg. 1.

We first instantiate the graph nodes with the instance annotation from the
point cloud and parameterize each node with object centroid
$\mathrm{p_{i}}\in\mathbb{R}^{3}$ and size of the axis-aligned bounding box
$\mathrm{b_{i}}=(b_{x},b_{y},b_{z})\in\mathbb{R}^{3}$ (Line 1-3). Next, we
traverse all the nodes to determine their spatial relationships (Line 4-22).
Notably, in cases where an object node lacks any in-contact vertical
relationships with other objects in the scene, we designate such objects as
"hangable" and calculate their non-contact vertical relationships (Line 9-13).
Examples of such objects include paintings, curtains, _etc_. Finally, we
establish relationships between multiple objects (Line 23): i) When a target
object is connected with two edges labeled left and right, the target object,
along with the two neighboring nodes, forms a between relationship triplets.
ii) If the offset of the center point coordinates of a group of objects in
either the X-axis or Y-axis direction is smaller than a specified offset
threshold $\delta$, then this group of objects forms an align relationship.
The offset threshold $\delta$ will be adjusted based on the size of the scene.
In additional, we utilize an automatic verification procedure to validate the
scene graph, further improving the quality of the scene graph we constructed
(line 24). One of the verification operations involves manually maintaining a
mapping between objects and relationship descriptions based on common sense.
For example, people usually use “mounted on” to describe the relation between
TV and wall, rather than “hanging on”. Therefore, we would automatically
refined ( TV, hanging on, wall) to ( TV, mounted on, wall).

In our constructed 3D scene graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, the
nodes $\mathcal{V}$ comprises
$\mathcal{V}_{1}\bigcup\mathcal{V}_{2}\bigcup\dots\bigcup\mathcal{V}_{K}$,
with $\mathcal{V}_{k}$ representing the set of nodes at a particular
hierarchical level. The hierarchies are determined by the support
relationship; for instance, objects supported by the floor constitute
$\mathcal{V}_{0}$, while objects supported by the table will form
$\mathcal{V}_{1}$, _etc_. Note that edges originating from one node
$v\in\mathcal{V}_{k}$ may only terminate in nearby hierarchies
$\mathcal{V}_{k}\cup\mathcal{V}_{k+1}\cup\mathcal{V}_{k+1}$. In other words,
edges in the scene graph exclusively connect nodes within the same
hierarchical level, or one level higher or lower.

Input : $M$ object point clouds $\\{P_{1},P_{2},\dots,P_{m}\\}$

Output : 3D scene graph $\mathcal{G}(\mathcal{V},\mathcal{E})$

1: for $i$ from $1$ to $M$ do

2: Create node $v_{i}\in\mathcal{V}$ using the centroid $p_{i}$ and bounding
box size $b_{i}$ of object point cloud $P_{i}$

3: end for

4: for $i$ from $1$ to $M$ do

5: for $j$ from $i+1$ to $M$ do

6: $\mathrm{RelsType}_{v}\leftarrow\mathrm{VerticalInContact}(v_{i},v_{j})$

7: Add in-contact vertical relationship triplets $(v_{i},v_{j},e_{i,j})$ with
$\mathrm{RelsType}_{v}$ to $\mathcal{G}$

8: end for

9: if No objects horizontally related to $v_{i}$ then

10: for $k$ from $1$ to $M$ and $i\neq k$ do

11: $\mathrm{RelsType}_{v}\leftarrow\mathrm{VerticalNonContact}(v_{i},v_{k})$

12: Add non-contact vertical relationship triplets $(v_{i},v_{k},e_{i,k})$
with $\mathrm{RelsType}_{v}$ to $\mathcal{G}$

13: end for

14: end if

15: end for

16: for $v_{i}\in\mathcal{V}$ do

17: let ${\\{v_{i_{1}},v_{i_{2}},...,v_{i_{N}}\\}}$ be the $N$ different nodes
with the same in-contact vertical parent node $v_{i}$

18: for $j$ from $1$ to $N$ do

19: $\mathrm{RelsType}_{h}\leftarrow\mathrm{Horizontal}(v_{i},v_{i_{j}})$

20: Add horizontal relationship triplets $(v_{i},v_{i_{j}},e_{i,i_{j}})$ with
$\mathrm{RelsType}_{h}$ to $\mathcal{G}$

21: end for

22: end for

23: Update $\mathcal{G}\leftarrow\mathrm{MultiObjects}(\mathcal{G})$

24: Update $\mathcal{G}$ with automatic verification procedure

Algorithm 1 Scene Graph Construction Pipeline Figure A.1: Overview of the
relationships in SceneVerse. The target object is colored in blue. Figure A.2:
Examples of object captioning. We color the target object in bold. Figure A.3:
Examples of scene captioning. Figure A.4: Examples of object referral. Note
that the green bounding box indicates the target object and yellow bounding
box indicates the anchor object(s).

###  A.3 Language Generation Details

In Sec. 3.3, we adopt both templates and LLM to automatically generate scene-
language pairs in SceneVerse. More technical details and examples are provided
in this section.

####  A.3.1 Object Captioning Pipeline

Object captions aim to provide detailed descriptions of an object’s visual and
physical properties, facilitating object-level grounding with its distinctive
features. The detailed object captioning pipeline is outlined in Alg. 2. Given
the multi-view images $\\{I_{1},I_{2},\dots,I_{n}\\}$, we utilize the point
cloud $P_{o}$ of the object $o$ to get the visible points $P_{o,v}^{vis}$ in
the images $v$ through rendering. The occlusion score $s_{o,v}^{occ}$ is
calculated as the ratio between the number of visible points and the object
point cloud. The image is then cropped with the rendered bounding box and
processed through BLIP2 [48] to generate the initial object caption $C_{o,v}$.
For each initial caption, we calculate its CLIP [66] similarity score between
the text and the cropped image, denoted by $s_{o,v}^{clip}$. To get a refined
object caption, we select the top $10$ initial captions with the highest CLIP
score and minimal occlusion. The selected sentences are fed into a LLM to
obtain a coherent summary of the object captions. In this process, we
explicitly instruct the language model to identify and correct the potential
errors.

Input : $M$ object point clouds $\\{P_{1},P_{2},\dots,P_{m}\\}$; $N$ multiview
images $\\{I_{1},I_{2},\dots,I_{n}\\}$

Output : Captions for each object in the scene $\\{C_{1},C_{2},\dots,C_{m}\\}$

1: for $o=1,2,\dots,M$ do

2: for $v=1,2,\ldots,N$ do

3: Project $P_{o}$ on $I_{v}$ to get visible points $P^{vis}_{o,v}$

4: Crop $I_{v}$ with the bounding box of $P^{vis}_{o,v}$ to get
$I^{crop}_{o,v}$

5: Get the image caption $C_{o,v}$ for $I^{crop}_{o,v}$ using BLIP2 [48]

6: Calculate the similarity score $s^{clip}_{o,v}$ between $C_{o,v}$ and
$I^{crop}_{o,v}$ with CLIP [66]

7: Calculate the occlusion score
$s^{occ}_{o,v}=\frac{\\#P^{vis}_{o,v}}{\\#P_{o}}$

8: end for

9: Select the top-10 $\\{C_{o,v}\\}$ with highest
$s^{clip}_{o,v}*s^{occ}_{o,v}$

10: Summary selected $\\{C_{o,v}\\}$ with GPT-3.5 to get $C_{o}$

11: end for

Algorithm 2 Object Captioning Pipeline

####  A.3.2 Automatic Language Generation

##### Template-based

We create diverse templates to generate descriptions for each type of
relationship. We categorized the templates into three types based on the
number of objects involved and their spatial relationships.

  * •

Pair-wise: The pair-wise templates are used to describe the positional
relationship between the target object and the anchor object in the scene. We
design various templates to enrich the templated-based descriptions, spanning
active and passive tense, as well as inversion clauses. Typical examples are
shown below:

    * -

The target-object (is) spatial-relation the anchor-object.

    * -

It is a target-object that (is) spatial-relation the anchor-object.

    * -

There is a target-object that (is) spatial-relation the anchor-object.

    * -

Spatial-relation the anchor-object is the target-object.

    * -

Spatial-relation the anchor-object, a target-object is placed.

  * •

Multi-objects: This is utilized when the target object forms a between or
align relationship with multiple anchor objects in the scene. The templates
follow the same construction rules as the Pair-wise templates.

  * •

Star-reference: To increase complexity in templated-based descriptions, we
design “star-reference” to describe the target object and its relationship
with 3 randomly selected anchor objects in the scene graph. In particular, we
perform cluster analysis on the selected relationship triplets. Based on the
diversity of the analysis, different templates will be chosen to generate
descriptions. For example, when the relations between 3 anchor objects and the
target object is the same, we prefer to use the template like: “The target-
object (is) spatial-relation the anchor-object-1, anchor-object-2 and anchor-
object-3”. If 2 out of the 3 anchor objects have the same relations with the
target object, we would use a template like: “The target-object (is) spatial-
relation-1 the anchor-object-1 and anchor-object-2, and (is) spatial-
relation-2 the anchor-object-3.”

Table A.2: Prompts used in SceneVerse.

Description type |  Prompt
---|---
Object caption |  Summarize caption below. The summary should be a description of the target-object. Focus on the target-object’s attribute, like color, shape and material, _etc_. Identify and correct the potential errors.
caption: A bed in a hotel room. A white comforter on a bed. A bed with a
striped comforter…
target-object: Bed
Object referral |  Rewrite the following caption using one random sentence structure. You should give me only one rewritten sentence without explanation. caption: The bed is between desk and nightstand.
|  Rewrite the following caption. You should give me only one rewritten
sentence about target-object without explanation. Make sure target-object is
the subject of the sentence, not anchor-object(s). If the sentence is in full
inversion, keep the inversion. caption: The armchair is next to the sofa.
target-object: Armchair anchor-object(s): Sofa
|  Rewrite the following caption using one random sentence structure. You need
to focus on the location and relations of the target-object that appears in
the sentence. If multiple target-object appear in the sentence, you need to
focus on the first target-object that appears. You can also add the target-
object’s function and comfort level based on the sentence, e.g., how the
objects can be used by humans and human activities in the scene. You should
give me only one rewritten sentence without explanation. caption: Far from the
bowl and peppershaker, the vase is to the left, it is also on the top of
countertop. target-object: Vase
Scene captioning |  Your task is to provide a summary for a scene from a given scene graph. The scene contains some objects, which compose a scene graph in json format.
There are 3 types of descriptions in scene graph: “scene type” denotes the
type of the scene. “objects count” then listed the objects in the scene and
their quantity, it should be noted that the actual objects in the room may be
more than listed. “objects relations” describe the spatial relations with
objects.
Also describe the scene concerning commonsense, e.g., how the objects can be
used by human and human activity in the scene. The description should conform
to the given scene graph. The spatial relations between objects can only be
inferred from the “objects relations“ in scene graph. Don’t describe each
object in the scene, pick some objects of the scene for summary. Don’t
describe each relations in the scene, pick some relations of the scene for
summary. You can also summarize the room’s function, style, and comfort level
based on the arrangement and count of objects within the room. The summary
should be about the object types, object attributes, relative positions
between objects. Your summary must not exceed 80 words. You must write using
one random sentence structure.
scene graph: { ’scene_type’: ’Bedroom’, ’object_count’: {’nightstand’:2, ...},
’relation’: {’nightstand’, ’on’, ’floor’}, {’backback’, ’in front of’, bed},
...}

#####  LLM-rephrasing

To increase description diversity we use the GPT-3.5[61] and Llama[75] for
description rephrasing. This improves the diversity and naturalness of the
template-based descriptions, as is shown in Fig. 2. The detailed prompts are
provided in Tab. A.2.

More examples of the scene-language pairs in SceneVerse are shown in Fig. A.2,
Fig. A.3 and Fig. A.4.

##  Appendix B Model Details

###  B.1 Spatial-Attention Transformer

In Sec. 4.2, we leveraged and spatial-attention based transformer architecture
to aggregate object-level point cloud features with spatial location
information. In this section, we provide the detailed design of this proposed
module.

Formally, given object features $\\{{\bm{f}}_{i}^{O}\\}_{i=1}^{N}$ and their
locations $\\{{\bm{l}}_{i}\\}_{i=1}^{N}$, we first construct pair-wise spatial
relationship feature via:

${\bm{m}}_{ij}=\left[d_{ij},\sin(\theta_{h}),\cos(\theta_{h}),\sin(\theta_{v}),\cos(\theta_{v})\right],$

where $d_{ij}$ denotes the Euclidean distance between objects and
$\theta_{h},\theta_{v}$ are the horizontal and vertical angles of the line
connecting the centers of objects $i$, $j$. We then use this pair-wise
relationship feature $M=[{\bm{m}}_{ij}]\in\mathbb{R}^{N\times N\times 5}$ to
modulate the attention weights of the self-attention layer in the transformer
when aggregating object features as shown below:

$\small\mathrm{Attn}(Q,K,V,M)=\mathrm{softmax}\left(\frac{QK^{T}}{\sqrt{d_{h}}}+\log\sigma(M\omega)\right)V,$

where $\omega\in\mathbb{R}^{5}$ is a projection layer mapping spatial pair-
wise features to the attention scores and $\sigma$ denotes the sigmoid
function. This process could be equivalently interpreted as using the spatial
location of objects to adjust the self-attention feature aggregation between
objects, making spatially related objects have more attention weights.

###  B.2 Pre-training Details

For training our model GPS, we conduct a two-stage training approach. As
described in Sec. 4.3, we first pre-train the object point cloud encoder with
the object-level grounding objective. Next, we freeze the object point cloud
encoder during the second pre-training stage for scene-level pre-training that
includes model training with scene-level grounding and referral object
grounding objectives. This design is inspired by recent works like [98, 21]
that demonstrated a well-grounded initialization of object representations is
beneficial for 3D scene grounding.

##### Object-level pre-training

To correctly align objects in scenes with their captions, we utilize the
ground-truth object bounding boxes provided with the datasets to segment all
objects in the scenes. Next, we utilize a PointNet++ [65] encoder to encode
and align these object point clouds with object captions provided in
SceneVerse following Sec. 4.1. For object instances with no object captions
synthesized, we follow [66] and construct captions with their semantic class
labels like “the point cloud of <CLASS>”. Notably, as our model design sets no
constraints on object point cloud encoders, the choice of object encoder
mainly depends on the computing resources available.

##### Scene-level pre-training

With pre-trained object feature extractors, we further use both scene captions
and object-referring expressions for scene-level pre-training. We use a
4-layer BERT encoder for encoding both scene captions and object referrals. As
discussed in Sec. 4.2, we apply a 4-layer spatial transformer to encode object
features with their locations. For scene-level grounding, we adopt a max-
pooling layer to aggregate features from the spatial transformer and align
with the [CLS] feature of the scene caption. For referral-object-level
grounding, we further pass the obtained object features as well as the
referral language features into a 4-layer self-attention transformer and use
the grounding objective described in Sec. 4.3 to match the referred object’s
feature and the [CLS] feature of the referring expression.

##### Training

For object-level pre-training, we utilize an AdamW optimizer with a learning
rate of $1\times 10^{-2}$ for 1500 epochs and no warm-up periods. During
training, we use a batch size of 512 and leverage a cosine annealing scheme
for learning rate scheduling with a minimum learning rate of $1\times
10^{-3}$. For scene-level pre-training, we use an AdamW optimizer with a
learning rate of $1\times 10^{-5}$ for the language encoder, a learning rate
of $1\times 10^{-4}$ for the spatial transformer, a learning rate of $1\times
10^{-4}$ for the self-attention transformer, and a learning rate of $5\times
10^{-4}$ for all remaining learnable parameters (_e.g_., projections). For all
experiments, we train the model for 150 epochs with a warm-up period of 500
and also a cosine annealing scheme for learning rate with a minimum learning
rate ratio of $0.1$. All pre-training experiments are run on 8 NVIDIA-A100
GPUs with the longest pre-training on SceneVerse taking about 2 days.

##  Appendix C Experimental Details

In this section, we provide details on experimental settings, model
implementation, and additional results.

###  C.1 3D Visual Grounding

##### Setting

For all datasets, we evaluate all models with only the training sets provided.
Following previous works [98], we report model performance on the validation
set of all datasets in Tab. 2. Notably, we used an off-the-shelf Mask3D
segmentation model for generating object proposals with no optimization.

##### Implementation

As briefly mentioned in Sec. 5.1, we mainly considered three model settings in
3D visual grounding experiments, namely scratch, pre-train, and fine-tuned.
For the pre-train setting, we follow the same setting mentioned in Sec. B.2.
In the scratch and fine-tuned settings, to fairly compare with other dataset-
specific fine-tuned models, we add an additional 2-layer MLP over the object
features from the referral grounding self-attention transformer. During
training, we fine-tune this grounding head together with all model weights for
100 epochs with a learning rate of $1\times 10^{-4}$ for the added projection
layer and set all other settings the same as the implementation described in
Sec. B.2.

###  C.2 Zero-shot Transfer

##### Setting

In the zero-shot experiments, we first construct the held-out test set by
aggregating scene-text pairs in SceneVerse from scenes in ScanNet and
MultiScan. Specifically, we use the validation set of ScanRefer, Nr3D, and
Sr3D. For scene-text pairs in the SceneVerse-val, we construct the test set by
randomly sampling $\frac{1}{5}$ of human-annotated object referrals in the
MultiScan dataset. This results in a test set with around 1.7K object
referrals randomly drawn from 8.5k human-annotated object referrals in the
MultiScan dataset. In the zero-shot settings, we use all scene-text pairs from
datasets in SceneVerse except for ScanNet and MultiScan. This includes both
human-annotated and generated texts in ARKitScenes, 3RScan, and HM3D. This
setting serves to test models’ generalization capability in grounding objects
with both unseen scenes and unseen texts. In the zero-shot text setting, we
add generated scene-text pairs in ScanNet and MultiScan into the data used in
the zero-shot setting, thereby making the held-out test containing mainly
unseen object referrals.

##### Implementation

In the zero-shot experiments, we mainly considered three model settings
scratch, zero-shot, and zero-shot text. For the zero-shot setting, we pre-
train the model following Sec. B.2 without additional grounding heads
considering there is no additional training data available in the zero-shot
transfer setting. In the scratch and zero-shot text setting, we follow the
model implementation described in Sec. C.1 and add an additional 2-layer MLP
over the object features from the self-attention transformer. We follow the
same fine-tuning setting described in Sec. C.1.

##  Appendix D Additional Results

In this section, we provide additional experimental results. Specifically, we
leverage our collected SceneVerse as the pre-training data source for a
traditional 3D semantic segmentation task. Next, we provide ablative analyses
of our model design.

###  D.1 Semantic Segmentation

##### Setting

To test if the scaling effect of SceneVerse is universally beneficial for 3D
understanding tasks, we use 3D semantic segmentation as a signature task to
illustrate the effectiveness of SceneVerse. Notably, a recent work that
introduced the Swin3D model [85] has identified the importance of pre-training
for 3D semantic segmentation [85]. Following the same setting, we test if the
proposed Swin3D model could be further improved by substituting the pre-
training data to SceneVerse. Specifically, we test models’ performance on the
ScanNet semantic segmentation task with 20 semantic categories and report the
mean IoU and mean Acc on the validation set of ScanNet. As the original
implementation of Swin3D pre-training requires surface normals as additional
inputs, we reimplement the model and pre-train all models with only point
coordinates and colors.

Table A.3: Semantic segmentation results on ScanNet validation set. $\dagger$
denotes model trained with surface normals as an additional input. S3D
indicates models initialized with the original Swin3D model weights pre-
trained on Structured3D provided by Yang _et al_. [85].

Methods | Init. |  SceneVerse Pre. | mIoU | mAcc
---|---|---|---|---
Swin3Dn-S$\dagger$ | ✗ | ✗ | 75.2 | -
Swin3Dn-S$\dagger$ | S3D | ✗ | 75.6 | -
Swin3D-S | ✗ | ✗ | 63.2 | 72.8
Swin3D-S | S3D | ✗ | 64.1 | 75.1
Swin3D-S (pre-train) | ✗ | ✓ | 67.7 | 78.0
Swin3D-S (pre-train) | S3D | ✓ | 69.5 | 80.1
Swin3D-S (fine-tuned) | S3D | ✓ | 70.6 | 80.2

##### Comparison

As shown in Tab. A.3, we observe a significant model performance improvement
($\sim$6%) by training Swin3D-S model on our SceneVerse dataset. Comparing our
pre-training set to Structured 3D, we also observe consistent model
performance improvement, showcasing the benefit of scaling-effect in
SceneVerse. Moreover, we fine-tune the model on ScanNet after pre-training on
SceneVerse. This process further brings improvement in model performance on
semantic segmentation. We believe these results serve as strong pieces of
evidence validating the effectiveness of data scaling in SceneVerse and also
its potential benefit for all 3D tasks in addition to 3D visual grounding.

####  D.1.1 Model Ablation

In this section, we provide ablative analyses of our multi-level contrastive
alignment design. We mainly consider removing objectives in our model as
ablations. We choose the referral-object-level alignment objective as the
default setting and consider removing: (i) object-level alignment objective,
(ii) masked language modeling objective, and (iii) scene-level alignment
objective. For removing the object-level alignment objective, we remove the
first stage pre-training of the object point cloud encoder and jointly learn
this module within the referral-object-level alignment. As shown in Tab. A.4,
we test different model settings on the SceneVerse-val without additional
fine-tuning. First, we show that the scene-level alignment objective is
crucial for referral object grounding in SceneVerse-val with the $\sim$5%
performance drop. Similar observations could be made for the model trained
without object-level alignment ($\sim$2% drop) and masked language modeling
objective ($\sim$1.5% drop). We believe these results affirm the effectiveness
of our overall model design.

Table A.4: Model ablation of our model on SceneVerse-val. Obj-lvl | MLM | Scene-lvl | Overall | Easy | Hard
---|---|---|---|---|---
✗ | ✗ | ✗ | 64.8 | 75.4 | 48.7
✓ | ✗ | ✗ | 65.2 | 77.1 | 47.4
✓ | ✓ | ✗ | 62.4 | 73.4 | 45.8
✓ | ✓ | ✓ | 66.9 | 77.8 | 50.3
  *[LLMs]: Large Language Models
  *[2D-VL]: 2D vision-language
  *[3D-VL]: 3D vision-language
  *[GPS]: Grounded Pre-training for Scenes
  *[LLM]: Large Language Model