11institutetext: Visual Geometry Group, University of Oxford 11email:
<EMAIL_ADDRESS>
# N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fields

Yash Bhalgat Iro Laina João F. Henriques
Andrew Zisserman Andrea Vedaldi

###### Abstract

Understanding complex scenes at multiple levels of abstraction remains a
formidable challenge in computer vision. To address this, we introduce Nested
Neural Feature Fields (N2F2), a novel approach that employs hierarchical
supervision to learn a single feature field, wherein different dimensions
within the same high-dimensional feature encode scene properties at varying
granularities. Our method allows for a flexible definition of hierarchies,
tailored to either the physical dimensions or semantics or both, thereby
enabling a comprehensive and nuanced understanding of scenes. We leverage a 2D
class-agnostic segmentation model to provide semantically meaningful pixel
groupings at arbitrary scales in the image space, and query the CLIP vision-
encoder to obtain language-aligned embeddings for each of these segments. Our
proposed hierarchical supervision method then assigns different nested
dimensions of the feature field to distill the CLIP embeddings using deferred
volumetric rendering at varying physical scales, creating a coarse-to-fine
representation. Extensive experiments show that our approach outperforms the
state-of-the-art feature field distillation methods on tasks such as open-
vocabulary 3D segmentation and localization, demonstrating the effectiveness
of the learned nested feature field.

###### Keywords:

Hierarchical Scene Understanding Feature Field Distillation Open-Vocabulary 3D
Segmentation

##  1 Introduction

Figure 1: Nested Neural Feature Fields (N2F2). We present N2F2, wherein
different dimensions of the same feature field encode scene properties at
varying granularities. The illustration captures the essence of hierarchical
scene understanding, depicting how our model differentiates between coarse and
fine scales to accurately interpret complex semantic queries, such as “donuts
in a white box” and “chocolate donut”, showcasing the model’s versatility in
handling detailed object descriptions within 3D environments.

3D scene understanding is an important problem in computer vision which still
presents several challenges. One of them is that scene understanding is
inherently hierarchical, as it requires reasoning about the scene at varying
levels of geometric and semantic granularity. Models must simultaneously
understand the high-level structure and composition of the scene as well as
fine-grained object details. This is important for applications like robotics
and augmented reality.

Recent progress in radiance fields has played a pivotal role in advancing 3D
scene understanding. Methods such as Neural Radiance Fields (NeRF) [38] and 3D
Gaussian Splatting [21] can extract the shape and appearance of 3D scenes
without 3D supervision or specialized sensors, as they are optimized directly
from several RGB images using differentiable renderers. Furthermore, this
optimization process can be extended to distill and fuse 2D information into
the 3D representation well beyond RGB values. Several authors have in fact
proposed _fusion approaches_ , where 2D labels [61, 1, 23, 46] or 2D features
[52, 25, 22] are extracted from multiple views of the scene and fused into a
single 3D model. The fused features not only augment the 3D reconstruction
with a semantic interpretation, but can also remove noise from the 2D labels,
improving their quality.

In this work, we are particularly interested in the problem of 3D distillation
of vision-language representations, such as CLIP [44], which, in turn, enables
open-vocabulary 3D segmentation and localization of objects or scene elements
based on natural language descriptions. Existing methods like LERF [22] have
demonstrated the potential of embedding language features into NeRFs, allowing
users to query 3D scenes with arbitrary text inputs. However, these approaches
face two key limitations.

First, while vision-language models exhibit remarkable few-shot transfer
capabilities, their performance often degrades for more complex linguistic
constructs like compound nouns (“paper napkin”) or partitive phrases (“bag of
cookies”). This is because individual components may be interpreted separately
(_e.g_., causing to detect “paper” and “napkin” as separate concepts instead
of a “paper napkin”). This stems from the inherent challenge of compositional
generalization that plagues such models and causes them to behave like bags-
of-words [58, 51, 34, 30]. This behavior makes it difficult to use vision-
language models to capture compositions of objects and their attributes or
relations and is often attributed to the text encoder bottleneck [20] and the
contrastive formulation that is used to train such models. In practice, this
means that querying CLIP with the aforementioned prompts would falsely
localize both paper _and_ napkins, bags _and_ cookies. Existing 3D feature
distillation methods directly inherit these shortcomings, failing to
accurately localize or segment objects described by compound expressions.

Second, a more practical limitation of methods like LERF is their inefficiency
during inference. To produce relevance maps for a given text query, these
approaches need to densely evaluate the feature field at multiple spatial
scales. This imposes a noticeable computational burden that scales linearly
with the number of scales processed.

This work aims to address both limitations. Our key insight is to impose a
_hierarchical structure_ on the 3D feature field during training.
Specifically, we propose Nested Neural Feature Fields (N2F2), wherein
different subsets of dimensions within a single high-dimensional feature field
are tasked with encoding scene properties at varying granularities. This
design choice enables N2F2 to simultaneously capture multi-scale scene
representations in a coherent and parameter-efficient manner. To address the
compositionality challenge, we further contribute a novel composite embedding
approach that, during inference, combines the features across all hierarchy
levels in a weighted manner, effectively aggregating relevance scores across
multiple scales given a text query and eliminating the need for explicit scale
selection.

Through extensive experimentation on challenging datasets, we demonstrate that
N2F2 significantly outperforms prior work on open-vocabulary 3D segmentation
and localization, including those involving complex compound queries.
Moreover, our composite embedding strategy yields considerable speedups during
inference, making N2F2 1.7$\times$ faster than the current leading approach,
LangSplat [43], while achieving superior results and a significant increase in
granularity.

In summary, our core contributions are: (1) A hierarchical supervision
framework for distilling multi-scale semantic representations into a unified
feature field; (2) a composite embedding strategy that enables efficient open-
vocabulary querying without explicit scale selection; (3) state-of-the-art
performance on challenging 3D segmentation and localization benchmarks, with
particular gains on complex compound queries.

##  2 Related Work

####  2.0.1 Radiance Fields.

Radiance fields have emerged as a powerful representation for capturing and
rendering complex 3D scenes from a set of multi-view posed images. Neural
Radiance Fields (NeRFs) [38] pioneered this approach by representing a scene
as a continuous 5D function, mapping spatial coordinates and viewing
directions to colors and densities. These representations can be optimized by
leveraging differentiable volume rendering. This seminal work has inspired a
plethora of variants and extensions, such as deformable or dynamic NeRFs [42],
NeRFs for unconstrained images [35], and variants for improved efficiency [6,
5, 47]. Recent work on 3D Gaussian Splatting (3DGS) [21] offers an alternative
by representing scenes as a mixture of 3D Gaussians, which has shown
impressive quality and speed in 3D reconstruction. The real-time rendering of
3DGS has enabled various downstream applications in dynamic scene
reconstruction [55], editing [12, 8] as well as generation [50].

####  2.0.2 3D Scene Segmentation.

Motivated by the success of radiance fields, several studies have successfully
extended such representations to 3D segmentation models. Examples include
semantic segmentation [61, 54], panoptic segmentation [1, 46, 14, 32] and part
segmentation [59]. A common characteristic of these works is that, given
multi-view images of a scene and corresponding 2D labels, these labels are
fused into the 3D space as part of the radiance field optimization process.
While NeRF has gained significant attention, it is worth noting that the
concept of integrating multiple viewpoints and semantic information for 3D
reconstruction predates NeRF and has been explored in various methods [17, 37,
48, 33, 36, 53].

More recently, several authors have turned to the Segment Anything model (SAM)
[24] to obtain class-agnostic 3D segmentations [23, 4, 3, 18]. SAM offers a
promptable approach to segmentation with multiple levels of granularity, which
has led most of these studies to adopt an interactive procedure, based on
user-provided object seeds. A notable exception is GARField [23], which
operates independently of user input. Instead, it leverages masks produced by
SAM, associating each mask with its corresponding 3D scale, and optimizes a
scale-conditioned affinity field. GARField can either be interactively queried
at specific scales or produce a hierarchy of groupings automatically. However,
it does not offer an automatic scale selection mechanism.

####  2.0.3 3D Feature Distillation.

Another related line of work focuses on lifting 2D image features to the 3D
space [52, 25, 62, 15]. Similarly to the segmentation case, distilling
features from a 2D teacher model, such as DINO [2], amounts to optimizing a 3D
feature field (alongside the radiance field) to reconstruct the teacher
features. In such cases, 3D segmentation involves computing the similarity of
the features embedded in the 3D feature field to some query.

Closest to our work are methods that construct 3D language fields from image-
text features [25, 22, 43, 31, 7, 11], which then enables querying the 3D
representation with open-vocabulary text descriptions, thus achieving text-
guided segmentation. In particular, LERF [22] is a scaled-conditioned approach
that distills multi-scale CLIP [44] features into a NeRF model. However, its
segmentation quality is limited since it does not use mask supervision and
relies mainly on the CLIP encoder which produces global features that lack
precise localization. LangSplat [43] addresses this issue by combining CLIP
features and SAM masks in a 3DGS representation. Unlike LERF or GARfield, it
only comprises three distinct scales (the same as SAM), which improves the
efficiency of the method but reduces the granularity of the semantic
hierarchy.

For each query, both LERF and LangSplat need to densely evaluate the feature
field at multiple spatial scales to select the best scale, which results in a
noticeable overhead, especially as the number of scales increases (_e.g_., as
in LERF). Our N2F2 approach makes use of the same underlying 2D models while
being simultaneously more granular than LangSplat and more efficient than both
methods while avoiding the need for explicit scale selection.

Beyond radiance fields, several works [41, 10, 16, 60, 19] operate on point-
cloud data and align dense point-features with text and/or image pixels using
pre-trained 2D vision-language models.

####  2.0.4 Hierarchical Representation Learning

focuses on developing multi-layered data abstractions to enhance model
adaptability and interpretability across tasks. Matryoshka representation
learning (MRL) [27] aims to embed multiple levels of granularity into a single
representation allowing the learned embeddings to adapt to the varying
computational constraints of the task at hand. MERU [9] presents a contrastive
approach that yields hyperbolic representations of images and text, resulting
in interpretable and structured representations while being competitive with
CLIP embeddings on various downstream tasks. In the domain of NLP, TreeFormers
[40] introduce a general-purpose text encoder that learns a composition
operator and pooling function to construct hierarchical encodings for natural
language expressions leading to improvements in compositional generalization.
Our approach, inspired by MRL, aims to learn a single feature field wherein
different subsets of the feature field encode varying levels of scene
properties based on both physical scale and the underlying semantics.

##  3 Method

In this section, we describe Nested Neural Feature Fields (N2F2), a
hierarchical supervision approach to learning multi-scale semantic
representations for 3D scenes. We first describe our underlying feature field
architecture (Sec. 3.1). Then, we introduce our proposed hierarchical
supervision method (Sec. 3.2) which encodes varying levels of scene
granularity into different subsets of the same feature field, leading to the
idea of nested feature fields. Finally, in Sec. 3.3, we formulate a novel
composite embedding approach that enables N2F2 to handle compound open-
vocabulary queries during inference. Figure 2 provides a high-level overview
of N2F2.

Figure 2: N2F2 Overview. (Left) N2F2 employs a 3D Gaussian Splatting (3DGS)
model for scene representation, augmented with a feature field that captures
scene properties across different scales and semantic granularities. (Middle)
Different subsets of the same feature field encode scene properties at varying
scales. This unified feature field is optimized using a hierarchical
supervision loss applied to the scale-aware features. (Right) We extract a
pool of segments using SAM and precompute a CLIP embedding for each. Each
segment is assigned a physical scale computed using the 3DGS model which is
then used to compute the scale-aware feature.

###  3.1 Feature Field Architecture

####  3.1.1 3D Scene Representation.

Given a set of images $\mathcal{I}$ of a scene with a corresponding camera
pose $\pi\in SE(3)$ for every image $I\in\mathcal{I}$, we aim to construct a
language-aware 3D model of the scene. We use 3D Gaussian Splatting (3DGS) [21]
as the scene representation, where the scene is modeled by a mixture of 3D
Gaussians. For any point $x\in\mathbb{R}^{3}$ in the scene, the influence
function of the $i^{th}$ Gaussian is parametrized as:

$g_{i}(x)=\exp\left(-\frac{1}{2}(x-\mu_{i})^{\top}\Sigma_{i}^{-1}(x-\mu_{i})\right),$

where $\mu_{i}\in\mathbb{R}^{3}$ is the Gaussian mean or center and
$\Sigma_{i}\in\mathbb{R}^{3\times 3}$ is its covariance, modeled using a
scaling matrix $S_{i}$ and rotation matrix $R_{i}$ as
$\Sigma_{i}=R_{i}S_{i}S_{i}^{\top}R_{i}^{\top}$. To model the 3D radiance
field, each Gaussian also has an opacity $\sigma_{i}\in\mathbb{R}_{+}$ and a
view($\nu$)-dependent colour $c_{i}(\nu)$ represented by spherical harmonic
coefficients $\mathcal{C}_{i}\in\mathbb{R}^{k}$ (upto the $3^{rd}$ order).
Overall, the 3DGS model is given by
$G=\\{(\mu_{i},\Sigma_{i},\sigma_{i},\mathcal{C}_{i})\\}_{i}$. Given the model
$G$, the 3D radiance field is defined as:

$\sigma(x)=\sum_{i}\sigma_{i}g_{i}(x),~{}~{}~{}c(x,\nu)=\frac{\sum_{i}c_{i}(\nu)\sigma_{i}g_{i}(x)}{\sum_{j}\sigma_{j}g_{j}(x)}.$
(1)

Given $G$ and a camera viewpoint $\pi$, the differentiable Gaussian Splatting
renderer [21] produces an image
$\hat{I}=\mathcal{R}(G,\pi)\in\mathbb{R}^{H\times W\times 3}$. The key reason
for choosing 3DGS as the scene representation rather than NeRF [38] or its
faster variants [57, 39, 6, 47] is that 3DGS provides a very efficient
renderer (both in speed and memory) resulting in real-time rendering of full-
sized images as well as feature maps, which allows us to perform rapid
querying during inference-time.

####  3.1.2 Gaussian Feature Field.

In this work, we propose to augment the 3DGS model with a feature field
$\Theta:\mathbb{R}^{3}\rightarrow\mathbb{R}^{D}$. A natural choice for doing
this would be to associate each Gaussian with a learnable feature vector and
optimize these features in the same manner as the other parameters (_i.e_.,
$\mu_{i},\Sigma_{i},\sigma_{i},\mathcal{C}_{i}$). However, in practice, the
size $D$ tends to very high (_e.g_., $D=512$ for CLIP embeddings) which
drastically increases the number of parameters in the augmented 3DGS model. To
address this issue, we model the feature field $\Theta$ with a memory-
efficient TriPlane representation [13] followed by a 3-layer MLP. During
rendering, we query this hybrid representation at the Gaussian centers to get
the associated feature for each Gaussian, _i.e_.,
$\Theta_{i}=\Theta(\mu_{i})$.111We use the same variable name $\Theta_{i}$ for
simplicity. Thus, given a camera viewpoint $\pi$, this results in a rendered
pixel-level feature map:

$\theta=\mathcal{R}(\Theta,G,\pi)\in\mathbb{R}^{H\times W\times D}$ (2)

In addition, to speed up training, we first render the TriPlane features, then
use them as input to the 3-layer MLP to obtain the pixel/ray feature. This
deferred rendering is performed only during training and leads to a
significantly faster optimization process without any loss in performance.

###  3.2 Scale-aware Hierarchical Supervision

Our main goal is to learn a unified representation that captures the meaning
of the scene across different scales and semantic granularities. We do so by
modeling N2F2 with a single Gaussian feature field, where different feature
dimensions represent varying levels of detail, bridging the gap between the
overall scene structure and fine-grained object particulars. To achieve this,
we train our feature field model using a scale-aware hierarchical supervision
method described below.

####  3.2.1 Extracting Training Data.

We start by using SAM [24] to extract class-agnostic but semantically
meaningful segments from every image $I\in\mathcal{I}$, resulting in a pool of
segments $\mathcal{S}$ spanning all the images. For each segment
$S\in\mathcal{S}$, we use the expected depth values from the 3DGS model to
lift the pixels in the segment to a corresponding 3D point cloud. We obtain
the scale of this point cloud as the largest eigenvalue of the covariance
matrix of the 3D point positions. We then quantize the extracted scales into
$D$ bins using a quantile transformation, where $D$ is the dimension of the
feature vectors in our field.222This transforms the scales to follow a uniform
distribution and also reduces the impact of marginal outliers. We denote the
quantized scale corresponding to segment $S$ as $s\in[0,1]$. Next, for every
segment $S\in\mathcal{S}$ and its corresponding image $I$, we use the CLIP
[45] vision encoder to obtain a language-aligned embedding,
$\phi^{gt}\in\mathbb{R}^{D}$, such that:

$\phi^{\operatorname{gt}}=E(I\odot S),$ (3)

where $E$ is the CLIP vision encoder and $\odot$ is the Hadamard product.
Below, we use this vector as “ground-truth” for the hierarchical supervision.

We use this data to fit our N2F2 model. To do this, we associate each
dimension of our feature field $\Theta\in\mathbb{R}^{D}$ to a quantized scale
value (note that the dimension $D$ is also the number of quantized values),
such that the lower dimensions are mapped to larger (coarser) scales and
higher dimensions to finer scales. This is predicated on the observation that
objects or scene components at larger scales can be differentiated with fewer
dimensions, using less specific features. Conversely, as one zooms into finer
scales, more dimensions become necessary to match the increased specificity
and diversity of the features. Mathematically, for a quantized scale
$s\in[0,1]$, we associate the dimension $M(s)$ defined by the mapping
$M(s)=\lfloor D\cdot(1-s)\rfloor$.

####  3.2.2 Scale-aware Feature.

Consider a pixel $u$ sampled from a segment $S$ with a corresponding scale
$s$. The rendered feature $\theta(u)$ can be obtained from our feature field
using Eq. 2. Now, given the mapping $M(\cdot)$, we obtain the scale-aware
feature as:

$\phi(u,s)=\mathbf{W}_{1:M(s)}\theta(u)_{1:M(s)}$ (4)

Here, $\mathbf{W}\in\mathbb{R}^{D\times D}$ is a learnable projection matrix.
$\mathbf{W}_{1:M(s)}\in\mathbb{R}^{D\times M(s)}$ is formed by the first
$M(s)$ columns of $\mathbf{W}$ and $\theta(u)_{1:M(s)}$ are the first $M(s)$
elements of the feature $\theta(u)$. Hence, the resulting
$\phi(u,s)\in\mathbb{R}^{D}$. We don’t explicitly normalize $\phi$ or $\theta$
so that the former remains a linear function of the latter.

Intuitively, the projection matrix $\mathbf{W}_{1:M(s)}$ dynamically adjusts
to the variable dimensionality of the feature vector $\theta(u)_{1:M(s)}$
corresponding to pixel $u$ at scale $s$. It ensures that this vector is mapped
to a fixed dimensionality $D$ that matches that of the features
$\phi^{\operatorname{gt}}$. This enables the use of single-scale features to
supervise the training of a nested multi-scale feature field.

####  3.2.3 Hierarchical Loss Supervision.

During training, we sample pixels $u$ uniformly across the image set
$\mathcal{I}$. Since there can be multiple segments associated with the same
pixel, we sample a segment $S$ with a probability inversely proportional to
the $\log$ of the area of $S$. With the rendered features $\theta(u)$ and the
language-aligned teacher embeddings $\phi^{\operatorname{gt}}$ from Eq. 3, we
minimize the following loss:

$\mathcal{L}_{\text{hier}}=\sum_{u,s}\mathcal{L}(\phi(u,s),\phi^{\operatorname{gt}})$
(5)

where $\mathcal{L}=\mathcal{L}_{\text{2}}+\lambda\mathcal{L}_{\text{cos}}$ is
a weighted combination of the L2 and cosine distance functions.

###  3.3 Composite Embedding for Open-Vocabulary Querying

Once the model has been optimized with the hierarchical loss function, it can
be queried with language prompts encoded using the CLIP text encoder. The
querying can be done either in 3D with the point features at the Gaussian
centers or in 2D with the rendered 2D features. We discuss the latter in this
section, but our method is compatible with both scenarios.

Previous methods [22, 43], compute relevancy maps at multiple scales (_e.g_.,
LERF uses 30 scales) for each text query and then choose the scale with the
highest relevancy score. We notice that this leads to a noticeable overhead
for every text input, especially for a large number of scales. Hence, we
propose a composite embedding method that allows us to compute only one
relevancy map per text query and use that as the output.

We define the composite embedding at a pixel $u$ to be a weighted sum of all
possible scale-aware features at that pixel. The idea of using a linear
combination of features across scales arises from the necessity to handle
compound queries effectively. Consider the query “lid of a cup”. The scale-
aware features ideally trigger twice for pixels that are both on a “lid” and
on a “cup”, akin to an intersection of sets. This is because the composite
embedding aggregates features from the scales corresponding to “a cup” and “a
lid”, effectively highlighting the summed response of these features.
Conversely, querying “a cup” would trigger once for all pixels on a cup,
reflecting the broader, single-scale interpretation.

The weights in the linear combination are query-agnostic and account for the
relevance of each scale, ensuring that our model can adaptively focus on the
most pertinent scales for any given query. They are pre-computed in 3D space
using canonical phrases such as “object”, “stuff”, “thing”, “part” and
“texture”. Mathematically, given a point-feature for a Gaussian, $\Theta_{k}$,
the weight $\gamma^{3D}_{k}$ is:

$\gamma^{3D}_{k}=\underset{i}{\text{Softmax}}\left(\max_{j}(\mathbf{W}_{1:i}\Theta_{k,1:i})^{\top}\phi^{\text{canon}}_{j}\right).$
(6)

where $\phi^{\text{canon}}_{j}$ are the set of CLIP embeddings of the
predefined canonical phrases listed above. The $\gamma^{3D}$ tensor only needs
to be computed once when training is over. For every new viewpoint,
$\gamma^{3D}$ is rendered to obtain the pixel-level tensor
$\gamma\in\mathbb{R}^{H\times W\times D}$.

Since there are exactly $D$ quantized scales, each mapped to a dimension of
the feature field, the composite embedding
$\phi_{\text{comp}}(u)=\sum_{i=1}^{D}\gamma_{i}(u)\phi(u,s_{i})$. Expanding
this using Eq. 4, we get:

$\displaystyle\phi_{\text{comp}}(u)$
$\displaystyle=\sum_{i=1}^{D}\gamma_{i}(u)\mathbf{W}_{1:i}\theta(u)_{1:i}=\sum_{i=1}^{D}\gamma_{i}(u)\left(\sum_{j=1}^{i}\theta(u)_{j}W_{j}\right)$
$\displaystyle=\sum_{j=1}^{D}\left(\sum_{i=j}^{D}\gamma_{i}(u)\right)\theta(u)_{j}W_{j}$

We can rewrite this as $\phi_{\text{comp}}(u)=\mathbf{W}\tilde{\theta}(u)$,
where

$\tilde{\theta}(u)=\begin{bmatrix}\sum_{i=1}^{D}\gamma_{i}(u)\\\
\sum_{i=2}^{D}\gamma_{i}(u)\\\ \vdots\\\
\gamma_{D}(u)\end{bmatrix}\odot\theta(u),\quad\text{or
equivalently,}\quad\tilde{\Theta}_{i}=\begin{bmatrix}\sum_{i=1}^{D}\gamma_{i}^{3D}\\\
\sum_{i=2}^{D}\gamma_{i}^{3D}\\\ \vdots\\\
\gamma_{D}^{3D}\end{bmatrix}\odot\Theta_{i}$ (7)

Given the above formulation, instead of computing the point-features
$\Theta_{i}$, we pre-compute the point-features $\tilde{\Theta}_{i}$ before
the text-querying process is started.

##  4 Experiments

####  4.0.1 Tasks and Datasets.

We assess the ability of our nested feature field to perform open-vocabulary
segmentation and localization on challenging natural language queries. We
train and evaluate our method on the expanded LERF dataset made available by
Qin _et al_. [43]. This expanded version contains ground-truth segmentation
masks for textual queries required to evaluate the segmentation performance.
The expanded version also contains additional and more challenging
localization samples to evaluate the localization accuracy. We also use the
3D-OVS dataset [31] to evaluate and compare our open-vocabulary 3D
segmentation capabilities.

####  4.0.2 Metrics.

For the 3D localization task, we consider a text query to be successfully
localized if the identified highest-relevancy pixel in a view lies inside the
ground-truth bounding box, reporting the localization accuracy. We report the
mIoU scores for the 3D segmentation task.

####  4.0.3 Baselines.

We compare our method with LERF [22] and LangSplat [43], which are the current
state-of-the-art feature distillation methods for open-vocabulary 3D
segmentation and localization. We also compare with a baseline distilling the
pixel-aligned feature from LSeg [28] into 3D.

###  4.1 Results

Table 1: Localization accuracy (%) comparisons on LERF scenes, averaged across text queries. More query-specific results are provided in supplementary. Method | bouquet | ramen | figurines | teatime | waldo_kitchen | overall
---|---|---|---|---|---|---
LSeg [28] | 50.0 | 14.1 | 8.9 | 33.9 | 27.3 | 21.1
LERF [22] | 91.7 | 62.0 | 75.0 | 84.8 | 72.7 | 73.6
LangSplat [43] | - | 73.2 | 80.4 | 88.1 | 95.5 | 84.3
N2F2 (Ours) | 91.7 | 78.8 | 85.7 | 91.5 | 95.5 | 88.6
Table 2: Open-vocabulary 3D semantic segmentation performance (mIoU) on the expanded LERF dataset from Qin _et al_. [43], averaged across queries. Method | ramen | figurines | teatime | waldo_kitchen | Overall
---|---|---|---|---|---
LSeg [28] | 7.0 | 7.6 | 21.7 | 29.9 | 16.6
LERF [22] | 28.2 | 38.6 | 45.0 | 37.9 | 37.4
LangSplat [43] | 51.2 | 44.7 | 65.1 | 44.5 | 51.4
N2F2 (Ours) | 56.6 | 47.0 | 69.2 | 47.9 | 54.4
Figure 3: Qualitative comparisons with LangSplat [43] on challenging compound queries. Table 3: Open-vocabulary 3D semantic segmentation performance (mIoU) on the 3D-OVS dataset [31], averaged across queries. Method | bed | bench | room | sofa | lawn | overall
---|---|---|---|---|---|---
LSeg [28] | 56.0 | 6.0 | 19.2 | 4.5 | 17.5 | 20.6
ODISE [56] | 52.6 | 24.1 | 52.5 | 48.3 | 39.8 | 43.5
OV-Seg [29] | 79.8 | 88.9 | 71.4 | 66.1 | 81.2 | 77.5
FFD [26] | 56.6 | 6.1 | 25.1 | 3.7 | 42.9 | 26.9
LERF [22] | 73.5 | 53.2 | 46.6 | 27 | 73.7 | 54.8
3D-OVS [31] | 89.5 | 89.3 | 92.8 | 74 | 88.2 | 86.8
LangSplat [43] | 92.5 | 94.2 | 94.1 | 90.0 | 96.1 | 93.4
N2F2 (Ours) | 93.8 | 92.6 | 93.5 | 92.1 | 96.3 | 93.9

####  4.1.1 3D Localization.

Table 1 shows the 3D localization performance on the expanded LERF dataset
[43], on a diverse set of scenes, namely bouquet, ramen, figurines, teatime,
and waldo_kitchen. Notably, the bouquet scene is not a part of the extended
LERF dataset, and LangSplat [43] does not provide results for this scene.
Thus, for this scene, we evaluate and compare on the text queries from the
original LERF dataset. Our method N2F2 significantly outperforms the existing
state-of-the-art methods, LERF [22] and LangSplat [43], across most of the
evaluated scenes, achieving an overall accuracy of $88.6\%$. Our N2F2 approach
particularly outshines prior work in samples with complex compound queries,
such as “sake cup”, “bag of cookies”, etc. For a detailed breakdown of the
performance across such queries, we refer to Appendix 0.A.

####  4.1.2 3D Segmentation.

Tables 2 and 3 demonstrate the performance on the open-vocabulary 3D
segmentation task on the expanded LERF [43] and 3D-OVS [31] datasets
respectively. The expanded LERF dataset contains various compound queries such
as “bag of cookies”, “coffee mug”, “paper napkin”, _etc_. as compared to
simple queries like “cookies”, “mug”, “apple”, which makes it challenging to
segment these referred objects. LERF and also LangSplat, which directly use
the distilled CLIP embeddings, struggle with such queries producing a high
relevancy score for all objects/attributes in the compound query. An example
shown in Figure 3 shows that for “bag of cookies”, LERF highlights both the
“bag” and the “cookies”, although our method correctly highlights the “bag”
being referred to. More such examples are included in the supplementary.
Overall, our method achieves an mIoU of $54.5$ and outperforms both LERF and
LangSplat on all scenes by a significant margin.

The 3D-OVS dataset contains relatively simple scenes where all included
objects are of similar scales. Thus, the performance of previous state-of-the-
art methods is highly saturated with LangSplat achieving almost perfect
segmentations for many text queries. Nevertheless, our method outperforms
LangSplat on $4$ out of $6$ scenes achieving an overall segmentation mIoU of
$93.9$. More qualitative results are included in the supplementary material.

###  4.2 Ablation Studies

####  4.2.1 Composite Embedding vs Explicit Scale Selection.

Unlike LERF or LangSplat, our method does not explicitly perform scale
selection for every query. Rather, we design a composite embedding (and the
$\gamma$ tensor, _cf_. Eq. 7) to produce high relevancy at the appropriate
scales and generate an aggregated relevancy map implicitly. But we could, in
principle, also explicitly search the scale with the highest relevancy score
in the same manner as LERF/LangSplat. Table 4 shows a comparison of these two
methods for localization and segmentation on the expanded LERF dataset. We
observe that the composite embedding method performs better than the explicit
scale selection method while being approximately $5\times$ faster during
querying. Compared to Tables 1 and 2, we note that the N2F2 explicit scale
selection baseline _also_ outperforms LangSplat [43], which also uses explicit
selection over 3 scales. This further demonstrates the advantage of our
hierarchical representation with increased granularity (_i.e_., $D$ scales).

Additionally, we ask – what performance can N2F2 achieve if the ideal scale
was somehow provided to it? We call this the “Oracle” scale. For a given text
query, we define the orcale scale to be the scale that gives the highest
performance for the task in question. This gives an upper bound on the
performance achievable by our method, which is shown in Tab. 4.

Table 4: Comparison of composite embedding based querying vs explicit scale-selection based querying on $4$ LERF scenes. Top: 3D localization. Bottom: 3D segmentation. Method | ramen | figurines | teatime | waldo_kitchen | Overall
---|---|---|---|---|---
3D Localization |  |  |  |  |
Composite embedding | 78.8 | 85.7 | 91.5 | 95.5 | 87.9
Explicit scale selection | 78.8 | 83.9 | 89.8 | 95.5 | 87.0
Oracle scale (Upper bound) | 83.0 | 91.0 | 93.2 | 95.5 | 90.6
3D Segmentation |  |  |  |  |
Composite embedding | 56.6 | 47.0 | 69.2 | 47.9 | 54.4
Explicit scale selection | 55.7 | 45.9 | 66.8 | 46.3 | 53.7
Oracle scale (Upper bound) | 59.8 | 49.1 | 71.2 | 49.2 | 57.3

####  4.2.2 Efficiency.

For each text query, LERF computes relevancy maps at $30$ scales (from $0$ to
$2$) and LangSplat computes them at $3$ scales hardcoded to the SAM [24]
hierarchy. For N2F2, with the composite embeddings, we can get the final
output by computing only a single relevancy map per text query. This leads to
a significant speed up during querying making our method overall $1.7\times$
faster than LangSplat and $26\times$ faster than LERF, while using more
scales.

Table 5 shows a full breakdown of the querying time cost for each method. With
our method, we pre-compute the $\gamma$ tensor (_cf_. Eq. 7) for every
Gaussian in 3D. Then, for each query, we render the $512$-dimensional
composite embeddings, in chunks. Our method takes $0.1$s to produce a
$730\\!\times\\!987\\!\times\\!512$ feature map from one viewpoint, while
LangSplat takes $0.05$s to to render and decode features to the same size.
Once rendered, LangSplat takes about $0.25$s/query, while our method takes
about $0.16$s/query resulting in a $1.7$x speedup amortized over 10 queries
per viewpoint.

Table 5: Breakdown of querying speed for different methods. Method |  Rendering time |  Time (per query) |  Time (10 queries) |  Total time (1 query) |  Total time (10 queries)
---|---|---|---|---|---
LERF | 18.4s | 2.5s | 20.0s | 20.9s | 38.4s
LangSplat | 0.05s | 0.29s | 2.5s | 0.3s | 2.55s
N2F2 (Ours) | 0.1s | 0.16s | 1.4s | 0.24s | 1.5s
Table 6: Effect of step-size in the dimension-scale mapping. Evaluation on the LERF teatime scene for the open-vocabulary 3D segmentation task (mIoU). Scene | $k=1$ | $k=8$ | $k=32$ | $k=128$ | $k=170$
---|---|---|---|---|---
LERF (teatime) | 69.2 | 68.7 | 68.1 | 67.2 | 66.4

####  4.2.3 Effect of step-size in the dimension-scale mapping.

In Sec. 3.2, we quantized the object scales to $D$ values, thereby associating
each of the dimensions from $\\{1,2,\dots,D\\}$ with a quantized scale in the
hierarchical loss optimization (_cf_. Eq. 5). We investigate what happens when
we do not utilize all the dimensions for scale supervision. That is, with a
“step-size” of $k$ in the dimension-scale mapping, we quantize the scales to
$\lfloor D/k\rfloor$ values and associate them to the dimensions
$\\{k,2k,\dots,k*\lfloor D/k\rfloor\\}$. For example, with $k=3$, only the
dimensions $\\{3,6,...510\\}$ are used. We train models with
$k=\\{8,32,128,170\\}$ using the hierarchical loss and compare them with the
default model ($k=1$) on the LERF teatime scene for open-vocabulary 3D
segmentation. Table 6 shows that performance gradually decreases as the
granularity of the feature field is reduced, _i.e_., when we reduce the
utilization of its dimensions. We note that the case when $k=170$, _i.e_.,
when only $3$ different quantized scales or dimensions are used (namely
$\\{170,340,510\\}$), is the closest to LangSplat which utilizes 3 scales. In
this setting, we achieve an mIoU of $66.4$ which is $1.3$ points higher than
LangSplat’s performance (_cf_. Tab. 2).

##  5 Limitations

Despite the advancements N2F2 model brings to hierarchical scene
understanding, it faces certain challenges. The model’s ability to learn
accurate feature fields heavily depends on the quality of scene
reconstructions, necessitating a diverse array of images from varied
viewpoints. While N2F2 accurately handles compound noun phrases (like “coffee
mug”) and partitive constructions (“bag of cookies” or “lid of the cup”), it
struggles with global scene context queries, such as identifying a “wooden
desk in the corner of the room”. This limitation points to the challenge of
integrating broad scene comprehension with specific object identification.
Addressing these limitations will be crucial for future developments in the
field.

##  6 Conclusion

In this work, we introduced Nested Neural Feature Fields (N2F2), a novel
approach for hierarchical scene understanding. N2F2 employs scale-aware
hierarchical supervision to encode scene properties across multiple
granularities within a unified feature field, significantly advancing open-
vocabulary 3D segmentation and localization tasks. Through extensive
experiments, our approach demonstrated superior performance over state-of-the-
art methods, such as LERF [22] and LangSplat [43], highlighting the
effectiveness of our hierarchical supervision methodology. In particular, our
method outshines on complex compounded and partitive constructions such as
“bag of cookies”, “chair legs”, “blueberry donuts”, etc. Finally, we also
propose a novel composite embedding design that enables highly efficient
querying as well as better performance than explicit scale-selection methods
used in previous works.

####  6.0.1 Acknowledgments.

We are grateful for funding from EPSRC AIMS CDT EP/S024050/1 and AWS (Y.
Bhalgat), ERC-CoG UNION 101001212 (A. Vedaldi and I. Laina), EPSRC VisualAI
EP/T028572/1 (I. Laina, A. Vedaldi and A. Zisserman), and Royal Academy of
Engineering RF\201819\18\163 (J. Henriques).

## References

  * [1] Bhalgat, Y.S., Laina, I., Henriques, J.F., Vedaldi, A., Zisserman, A.: Contrastive lift: 3d object instance segmentation by slow-fast contrastive fusion. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS) (2023)
  * [2] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proc. ICCV (2021)
  * [3] Cen, J., Fang, J., Yang, C., Xie, L., Zhang, X., Shen, W., Tian, Q.: Segment any 3d gaussians. arXiv preprint arXiv:2312.00860 (2023)
  * [4] Cen, J., Zhou, Z., Fang, J., Shen, W., Xie, L., Jiang, D., Zhang, X., Tian, Q., et al.: Segment anything in 3d with nerfs. Advances in Neural Information Processing Systems 36 (2024)
  * [5] Chan, E.R., Lin, C.Z., Chan, M.A., Nagano, K., Pan, B., Mello, S.D., Gallo, O., Guibas, L.J., Tremblay, J., Khamis, S., Karras, T., Wetzstein, G.: Efficient geometry-aware 3D generative adversarial networks. In: Proc. CVPR (2022)
  * [6] Chen, A., Xu, Z., Geiger, A., Yu, J., Su, H.: TensoRF: Tensorial radiance fields. In: arXiv (2022)
  * [7] Chen, H., Blomqvist, K., Milano, F., Siegwart, R.: Panoptic vision-language feature fields. IEEE Robotics and Automation Letters (2024)
  * [8] Chen, Y., Chen, Z., Zhang, C., Wang, F., Yang, X., Wang, Y., Cai, Z., Yang, L., Liu, H., Lin, G.: Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. arXiv.cs abs/2311.14521 (2023)
  * [9] Desai, K., Nickel, M., Rajpurohit, T., Johnson, J., Vedantam, S.R.: Hyperbolic image-text representations. In: International Conference on Machine Learning. pp. 7694–7731. PMLR (2023)
  * [10] Ding, R., Yang, J., Xue, C., Zhang, W., Bai, S., Qi, X.: Pla: Language-driven open-vocabulary 3d scene understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7010–7019 (2023)
  * [11] Engelmann, F., Manhardt, F., Niemeyer, M., Tateno, K., Tombari, F.: Opennerf: Open set 3d neural scene segmentation with pixel-wise features and rendered novel views. In: The Twelfth International Conference on Learning Representations (2023)
  * [12] Fang, J., Wang, J., Zhang, X., Xie, L., Tian, Q.: Gaussianeditor: Editing 3d gaussians delicately with text instructions. arXiv.cs abs/2311.16037 (2023)
  * [13] Fridovich-Keil, S., Meanti, G., Warburg, F., Recht, B., Kanazawa, A.: K-planes: Explicit radiance fields in space, time, and appearance. arXiv.cs abs/2301.10241 (2023)
  * [14] Fu, X., Zhang, S., Chen, T., Lu, Y., Zhu, L., Zhou, X., Geiger, A., Liao, Y.: Panoptic NeRF: 3d-to-2d label transfer for panoptic urban scene segmentation. arXiv.cs abs/2203.15224 (2022)
  * [15] Goel, R., Sirikonda, D., Saini, S., Narayanan, P.: Interactive segmentation of radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4201–4211 (2023)
  * [16] Ha, H., Song, S.: Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. In: 6th Annual Conference on Robot Learning (2022)
  * [17] Hermans, A., Floros, G., Leibe, B.: Dense 3d semantic mapping of indoor scenes from rgb-d images. In: 2014 IEEE International Conference on Robotics and Automation (ICRA). pp. 2631–2638. IEEE (2014)
  * [18] Hu, X., Wang, Y., Fan, L., Fan, J., Peng, J., Lei, Z., Li, Q., Zhang, Z.: Semantic anything in 3d gaussians. arXiv preprint arXiv:2401.17857 (2024)
  * [19] Jatavallabhula, K.M., Kuwajerwala, A., Gu, Q., Omama, M., Chen, T., Maalouf, A., Li, S., Iyer, G., Saryazdi, S., Keetha, N., et al.: Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241 (2023)
  * [20] Kamath, A., Hessel, J., Chang, K.W.: Text encoders bottleneck compositionality in contrastive vision-language models. In: The 2023 Conference on Empirical Methods in Natural Language Processing (2023)
  * [21] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3D Gaussian Splatting for Real-Time Radiance Field Rendering. Proc. SIGGRAPH 42(4) (2023)
  * [22] Kerr, J., Kim, C.M., Goldberg, K., Kanazawa, A., Tancik, M.: Lerf: Language embedded radiance fields. In: ICCV. pp. 19729–19739 (2023)
  * [23] Kim, C.M., Wu, M., Kerr, J., Goldberg, K., Tancik, M., Kanazawa, A.: Garfield: Group anything with radiance fields. arXiv.cs abs/2401.09419 (2024)
  * [24] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: ICCV (2023)
  * [25] Kobayashi, S., Matsumoto, E., Sitzmann, V.: Decomposing NeRF for editing via feature field distillation. arXiv.cs (2022)
  * [26] Kobayashi, S., Matsumoto, E., Sitzmann, V.: Decomposing nerf for editing via feature field distillation. NeurIPS 35, 23311–23330 (2022)
  * [27] Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., et al.: Matryoshka representation learning. Advances in Neural Information Processing Systems 35, 30233–30249 (2022)
  * [28] Li, B., Weinberger, K.Q., Belongie, S., Koltun, V., Ranftl, R.: Language-driven semantic segmentation. In: ICLR (2022), https://openreview.net/forum?id=RriDjddCLN
  * [29] Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., Marculescu, D.: Open-vocabulary semantic segmentation with mask-adapted clip. In: CVPR. pp. 7061–7070 (2023)
  * [30] Lin, Z., Chen, X., Pathak, D., Zhang, P., Ramanan, D.: Visualgptscore: Visio-linguistic reasoning with multimodal generative pre-training scores. arXiv preprint arXiv:2306.01879 (2023)
  * [31] Liu, K., Zhan, F., Zhang, J., Xu, M., Yu, Y., Saddik, A.E., Theobalt, C., Xing, E., Lu, S.: Weakly supervised 3d open-vocabulary segmentation. In: NeurIPS (2023)
  * [32] Liu, Y., Hu, B., Huang, J., Tai, Y.W., Tang, C.K.: Instance neural radiance field. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 787–796 (2023)
  * [33] Ma, L., Stückler, J., Kerl, C., Cremers, D.: Multi-view deep learning for consistent semantic mapping with RGB-D cameras. In: Proc.IROS (2017)
  * [34] Ma, Z., Hong, J., Gul, M.O., Gandhi, M., Gao, I., Krishna, R.: Crepe: Can vision-language foundation models reason compositionally? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10910–10921 (2023)
  * [35] Martin-Brualla, R., Radwan, N., Sajjadi, M.S.M., Barron, J.T., Dosovitskiy, A., Duckworth, D.: NeRF in the wild: Neural radiance fields for unconstrained photo collections. In: Proc. CVPR. vol. abs/2008.02268 (2021)
  * [36] Mascaro, R., Teixeira, L., Chli, M.: Diffuser: Multi-view 2D-to-3D label diffusion for semantic scene segmentation. In: Proc. ICRA (2021)
  * [37] McCormac, J., Handa, A., Davison, A., Leutenegger, S.: Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. In: 2017 IEEE International Conference on Robotics and automation (ICRA). pp. 4628–4635. IEEE (2017)
  * [38] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: Representing scenes as neural radiance fields for view synthesis. In: Proc. ECCV (2020)
  * [39] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. In: Proc. SIGGRAPH (2022)
  * [40] Patel, N., Flanigan, J.: Forming trees with treeformers. arXiv preprint arXiv:2207.06960 (2022)
  * [41] Peng, S., Genova, K., Jiang, C., Tagliasacchi, A., Pollefeys, M., Funkhouser, T., et al.: Openscene: 3d scene understanding with open vocabularies. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 815–824 (2023)
  * [42] Pumarola, A., Corona, E., Pons-Moll, G., Moreno-Noguer, F.: D-NeRF: Neural radiance fields for dynamic scenes. arXiv.cs abs/2011.13961 (2020)
  * [43] Qin, M., Li, W., Zhou, J., Wang, H., Pfister, H.: Langsplat: 3d language gaussian splatting. arXiv preprint arXiv:2312.16084 (2023)
  * [44] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Proc. ICML. vol. 139, pp. 8748–8763 (2021)
  * [45] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [46] Siddiqui, Y., Porzi, L., Bulò, S.R., Müller, N., Nießner, M., Dai, A., Kontschieder, P.: Panoptic lifting for 3D scene understanding with neural fields. arXiv.cs abs/2212.09802 (2022)
  * [47] Sun, C., Sun, M., Chen, H.: Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In: Proc. CVPR (2022)
  * [48] Sünderhauf, N., Pham, T.T., Latif, Y., Milford, M., Reid, I.: Meaningful maps with object-oriented semantic mapping. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 5079–5085. IEEE (2017)
  * [49] Tancik, M., Weber, E., Ng, E., Li, R., Yi, B., Kerr, J., Wang, T., Kristoffersen, A., Austin, J., Salahi, K., Ahuja, A., McAllister, D., Kanazawa, A.: Nerfstudio: A modular framework for neural radiance field development. In: ACM SIGGRAPH 2023 Conference Proceedings. SIGGRAPH ’23 (2023)
  * [50] Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: DreamGaussian: Generative gaussian splatting for efficient 3D content creation. arXiv (2309.16653) (2023)
  * [51] Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., Ross, C.: Winoground: Probing vision and language models for visio-linguistic compositionality. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5238–5248 (2022)
  * [52] Tschernezki, V., Laina, I., Larlus, D., Vedaldi, A.: Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representation. In: Proceedings of the International Conference on 3D Vision (3DV) (2022)
  * [53] Vineet, V., Miksik, O., Lidegaard, M., Nießner, M., Golodetz, S., Prisacariu, V.A., Kähler, O., Murray, D.W., Izadi, S., Pérez, P., et al.: Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction. In: 2015 IEEE international conference on robotics and automation (ICRA). pp. 75–82. IEEE (2015)
  * [54] Vora, S., Radwan, N., Greff, K., Meyer, H., Genova, K., Sajjadi, M.S.M., Pot, E., Tagliasacchi, A., Duckworth, D.: NeSF: Neural semantic fields for generalizable semantic segmentation of 3D scenes. arXiv.cs abs/2111.13260 (2021)
  * [55] Wu, G., Yi, T., Fang, J., Xie, L., Zhang, X., Wei, W., Liu, W., Tian, Q., Wang, X.: 4D gaussian splatting for real-time dynamic scene rendering. arXiv (2023)
  * [56] Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., De Mello, S.: Open-vocabulary panoptic segmentation with text-to-image diffusion models. In: CVPR. pp. 2955–2966 (2023)
  * [57] Yu, A., Fridovich-Keil, S., Tancik, M., Chen, Q., Recht, B., Kanazawa, A.: Plenoxels: Radiance fields without neural networks. arXiv (2021)
  * [58] Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., Zou, J.: When and why vision-language models behave like bags-of-words, and what to do about it? In: The Eleventh International Conference on Learning Representations (2022)
  * [59] Zarzar, J., Rojas, S., Giancola, S., Ghanem, B.: Segnerf: 3d part segmentation with neural radiance fields. arXiv preprint arXiv:2211.11215 (2022)
  * [60] Zhang, J., Dong, R., Ma, K.: Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. arXiv preprint arXiv:2303.04748 (2023)
  * [61] Zhi, S., Laidlow, T., Leutenegger, S., Davison, A.J.: In-place scene labelling and understanding with implicit scene representation. In: Proc. ICCV (2021)
  * [62] Zhou, S., Chang, H., Jiang, S., Fan, Z., Zhu, Z., Xu, D., Chari, P., You, S., Wang, Z., Kadambi, A.: Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. arXiv preprint arXiv:2312.03203 (2023)

##  Appendix 0.A Performance on Compound Queries

In Sec. 4.1 of the paper, we noted that, in comparison to previous methods
(LERF [22] and LangSplat [43]), N2F2 offers a strong advantage on queries
which are compound or partitive in nature. To understand this effect more
clearly, we split the text queries from the expanded LERF dataset into two
sets: (1) simple single word queries such as “cookies”, “cup”, “waldo”,
“spatula”, (2) compound or partitive queries333For brevity, we refer to all
such queries as compound queries. such as “bag of cookies”, “frog cup”, “dark
cup”, “toy elephant”, “toy cat statue”. Table 9 provides a full list of
compound queries for each scene.

Table 7 shows that N2F2 significantly outperforms LangSplat [43], the current
state-of-the-art method, especially on compound queries where we see an
improvement as big as 7.2 mIoU points on the ramen scene.

To gain a clearer insight into which queries N2F2 excels at, we rank the
queries within each scene according to the performance disparity between N2F2
and LangSplat. We then highlight the top-3 and bottom-3 queries that exhibit
the largest and smallest differences in performance, respectively. As shown in
Tab. 8, the largest performance improvements (_cf_. the ‘top-3 queries’
column) are observed on compound queries and especially the ones which could
apply to multiple objects, if they were not specific. For example, the
figurines scene contains two similar looking rubber ducks, but only one of
them wears a hat. Hence, the query “rubber duck with hat” must only segment
the referred duck. This is successfully achieved by N2F2 but LangSplat
segments parts of both the ducks resulting in a lower segmentation IoU.

Figures 5, 6 and 7 demonstrate more qualitative comparisons between N2F2 and
LangSplat [43] on various challenging compound queries.

Table 7: Breakdown of performance on simple and compound queries for open-vocabulary 3D segmentation (mIoU). “comp.” is short for compound queries. Method | ramen | figurines | teatime | waldo_kitchen
---|---|---|---|---
simple | comp. | simple | comp. | simple | comp. | simple | comp.
LangSplat [43] | 65.4 | 35.1 | 53.9 | 32.7 | 70.1 | 47.3 | 52.6 | 31.3
N2F2 (Ours) | 66.2 | 42.3 | 54.7 | 36.1 | 72.2 | 50.9 | 53.2 | 35.5
Table 8: Queries within each scene ranked according to performance disparity ($\Delta_{\text{perf}}$) between N2F2 and LangSplat [43]. Top-3 and bottom-3 ranked queries are shown. Scene | Top-3 queries | $\Delta_{\text{perf}}$ | Bottom-3 queries | $\Delta_{\text{perf}}$
---|---|---|---|---
ramen | sake cup | $+9.8$ | chopsticks | $+0.0$
spoon handle | $+6.9$ | bowl | $+0.5$
wavy noodles | $+5.1$ | egg | $+0.5$
figurines | rubber duck with hat | $+18.1$ | pumpkin | $-0.1$
red apple | $+8.6$ | waldo | $+0.3$
porcelain hand | $+4.5$ | pikachu | $+0.4$
teatime | bag of cookies | $+11.7$ | plate | $-0.3$
stuffed bear | $+7.0$ | sheep | $+0.1$
bear nose | $+5.4$ | dall-e brand | $+0.5$
waldo_kitchen | Stainless steel pots | $+10.7$ | knife | $-0.5$
pour-over vessel | $+7.2$ | ottolenghi | $+0.2$
red cup | $+6.0$ | pot | $+0.3$

##  Appendix 0.B Open-vocabulary Retrieval Task

Next, we assess the performance of our method in the context of text-based
retrieval. The objective of the open-vocabulary retrieval task is to identify
the most relevant object within the scene based on a natural language query.
Unlike segmentation, which assesses the precision of object boundaries,
retrieval focuses more on the model’s ability to find relevant objects.

To this end, we repurpose the expanded LERF dataset from Qin _et al_. [43] for
this task. For a test view, we take the pool of segments from SAM as given,
compute a relevancy score per segment (which is the average relevancy over
segment pixels), and then rank these segments according to the predicted
relevancy scores. In this task, for a given text query, if the correct
corresponding segment is within the top $K$ retrieved segments, then the query
is said to be retrieved. We report the recall ($R@K$) metric which is the
fraction of correctly retrieved queries for a given $K$. Note that the
performance at $K=1$ is correlated with the localization accuracy of the
method, indicating its ability to directly pinpoint the exact segment most
relevant to the given text query. Figure 4 compares the performance of our
method with LangSplat [43] across a range of $K$. We can see that at $K=1$,
N2F2 consistently performs better than LangSplat across scenes (which was also
reflected in the localization performance reported in Table 1 of the main
paper). The $R@K$ of both methods increases with $K$, while the performance
gap becomes smaller.

Figure 4: Open-vocabulary Retrieval performance on the expanded LERF dataset
from Qin _et al_. [43]. $R@K$ (%) reported at $K=1,2,3,4,5$ for each scene.

##  Appendix 0.C Implementation Details

###  0.C.1 Extracting Training data

To obtain embeddings for the text queries, we leverage the OpenCLIP ViT-B/16
model with the segment masks obtained from the SAM ViT-H model. We use the
laion2b_s34b_b88k pretrained checkpoint for the OpenCLIP model. For SAM mask
generation, we sample pixels on a $32\times 32$ grid, set the minimum mask
region area to be $100$ and use a IoU threshold of $0.7$. To get an accurate
geometry model for each scene, we first optimize the radiance field related
parameters of the 3D Gaussian Splatting (3DGS) model using the RGB images for
$30$k iterations. We use the expected depth obtained from the 3DGS model to
lift the SAM masks into 3D and obtain a physical scale for each mask as the
largest eigenvalue of the covariance matrix of the lifted 3D segment-pixels.

###  0.C.2 Feature Field Optimization

Then, we optimize our feature field for another $30$k iterations while
freezing all other parameters of the model. As mentioned in the paper, the
feature field is modeled with a TriPlane + 3-layer MLP. We use a $512\times
512$ resolution for each of the planes with $64$-dimensional features. A
hidden size of $256$ is used for the MLP. We empirically observe that using a
zero-initialization for the projection matrix $\mathbf{W}$ gives the best
results. The $\lambda$ weight used with the cosine loss (_cf_. Eq. (5)) is set
to be $0.001$. We use a learning rate of $0.0016\times\text{scene\\_extent}$
for the TriPlane and $0.00125$ for the MLP.

###  0.C.3 Rendering details

As described in Sec. 3.1 of the paper, during rendering, we query the
TriPlane+MLP representation at the Gaussian centers to get the associated
feature for each Gaussian, and then use the Gaussian Splatting renderer to
obtain the rendered feature map. The default 3DGS implementation does not
support rendering arbitrary features, so we modify and use the NDRasterizer
provided by Nerfstudio [49] to render the features.

###  0.C.4 Efficient Scale-aware Feature Computation

When optimizing the Hierarchical Loss (_cf_. Eq. (5)), different pixels $u$ in
a batch will have different associated scales $s$ (and hence different mapped
dimensions $M(s)$). Thus, to efficiently compute the scale-aware features
$\mathbf{W}_{1:M(s)}\theta(u)_{1:M(s)}$, we implement a batchwise masking
mechanism. Specifically, for a given scale $s$, we employ a binary mask $B(s)$
where entries corresponding to the active dimensions $1:M(s)$ are set to $1$,
and all others are set to $0$. The scale-aware features for each pixel $u$ are
then computed as
$\mathbf{W}_{1:M(s)}\theta(u)_{1:M(s)}=\mathbf{W}\cdot\left(B(s)\odot\theta(u)\right)$,
where $\odot$ denotes element-wise multiplication. The mask $B$ is computed
batchwise with negligible overhead.

Table 9: Distinction between the simple and compound queries for each scene. Scene |  Simple queries |  compound queries
---|---|---
waldo_kitchen |  sink, refrigerator, cabinet, spatula, toaster, plate, ottolenghi, spoon, ketchup, pot, knife |  yellow desk, Stainless steel pots, frog cup, red cup, pour-over vessel, plastic ladle, dark cup
figurines |  jake, bag, spatula, porcelain hand, rubics cube, waldo, pumpkin, miffy, pirate hat, old camera, pikachu |  tesla door handle, rubber duck with hat, red toy chair, toy elephant, green toy chair, pink ice cream, green apple, red apple, rubber duck buoy, toy cat statue
ramen |  corn, plate, chopsticks, egg, bowl, kamaboko, onion segments, napkin, spoon, hand, nori |  sake cup, glass of water, wavy noodles
teatime |  three cookies, plate, hooves, apple, dall-e brand, coffee, sheep |  bear nose, bag of cookies, tea in a glass, stuffed bear, coffee mug, paper napkin, yellow pouf

##  Appendix 0.D Qualitative results

In Figs. 5, 6 and 7, we show qualitative comparisons between N2F2 and
LangSplat [43] on various challenging compounded queries.

Figure 5: Scene: teatime. Each row contains results for the text query shown
on the left. Columns: N2F2 (Relevancy and Segmentation maps), LangSplat [43]
(Relevancy and Segmentation maps). Figure 6: Scene: donuts. Each row contains
results for the text query shown on the left. Columns: N2F2 (Relevancy and
Segmentation maps), LangSplat [43] (Relevancy and Segmentation maps). Figure
7: Scene: figurines. Each row contains results for the text query shown on the
left. Columns: N2F2 (Relevancy and Segmentation maps), LangSplat [43]
(Relevancy and Segmentation maps).