# SANeRF-HQ: Segment Anything for NeRF in High Quality

Yichen Liu1 Benran Hu2 Chi-Keung Tang1 Yu-Wing Tai3
1The Hong Kong University of Science and Technology
2Carnegie Mellon University 3Dartmouth College

###### Abstract

Recently, the Segment Anything Model (SAM) has showcased remarkable
capabilities of zero-shot segmentation, while NeRF (Neural Radiance Fields)
has gained popularity as a method for various 3D problems beyond novel view
synthesis. Though there exist initial attempts to incorporate these two
methods into 3D segmentation, they face the challenge of accurately and
consistently segmenting objects in complex scenarios. In this paper, we
introduce the Segment Anything for NeRF in High Quality (SANeRF-HQ) to achieve
high quality 3D segmentation of any object in a given scene. SANeRF-HQ
utilizes SAM for open-world object segmentation guided by user-supplied
prompts, while leveraging NeRF to aggregate information from different
viewpoints. To overcome the aforementioned challenges, we employ density field
and RGB similarity to enhance the accuracy of segmentation boundary during the
aggregation. Emphasizing on segmentation accuracy, we evaluate our method
quantitatively on multiple NeRF datasets where high-quality ground-truths are
available or manually annotated. SANeRF-HQ shows a significant quality
improvement over previous state-of-the-art methods in NeRF object
segmentation, provides higher flexibility for object localization, and enables
more consistent object segmentation across multiple views. Additional
information can be found at https://lyclyc52.github.io/SANeRF-HQ/.

##  1 Introduction

Neural Radiance Field (NeRF) [41] has produced state-of-the-art results in
novel view synthesis for intricate real-world scenes. NeRF encodes a given
scene using Multi-Layer Perceptrons (MLP) and supports queries of density and
radiance given 3D coordinates and view directions, which are used to render
photo-realistic images from any view points. Moreover, during training, NeRF
only requires RGB images with camera poses, which directly links 3D to 2D. The
simple but ingenious architecture with its continuous representation quickly
starts challenging traditional representations using explicit discrete
structures, such as RGB-D images or point clouds. As a result, NeRF is poised
to tackle more challenging tasks in 3D vision.

One important downstream task that can benefit from NeRF representations is 3D
object segmentation, which is fundamental in 3D vision and widely used in many
applications. To address object segmentation in NeRF, researchers have
investigated various methods. Semantic-NeRF [65], targeting semantic
segmentation, is one of the first works in this direction. DFF [33] distills
the knowledge of pre-trained features such as DINO [9] into a 3D feature field
for unsupervised object decomposition. Supervised approaches, such as [47],
utilize Mask2Former [14] to obtain initial 2D masks and lifts them to 3D with
a panoptic radiance field. Although these methods demonstrate impressive
results, their performance is constrained by the pre-trained models used to
produce features.

Recently, large vision models, such as Segment Anything Model (SAM) [32], have
emerged with strong zero-shot generalization performance and can be adopted as
the backbone component for many downstream tasks. Specifically, SAM proposes a
new paradigm for segmentation tasks which can accept a wide variety of prompts
as input, and produce segmentation masks of different semantic levels as
output. The versatility and generalizability of SAM thus shed light on a new
way to perform promptable object segmentation in NeRF. While there exist some
investigations [10, 13, 21] into this area, the mask quality in novel views is
still unsatisfactory.

In view of this, we propose a new general framework to achieve prompt-based 3D
segmentation in NeRF. Our framework, termed Segment Anything for NeRF in High
Quality, or SANeRF-HQ, leverages existing 2D foundation models such as Segment
Anything to allow various prompts as input, and produces 3D segmentations with
high accuracy and multi-view consistency. The major contributions of our paper
are:

  * •

We propose SANeRF-HQ, which is one of the first attempts in producing high-
quality 3D object segmentation in NeRF in terms of more accurate segmentation
boundaries and better multi-view consistency.

  * •

We validate our method by assembling and evaluating quantitatively on a
challenging dataset with high-quality ground-truths.

  * •

We present a more general framework to embed foundation 2D image models into
Neural Radiance Field and extend it to different 3D segmentation tasks in
NeRFs.

Comparing with [21] and [10], SANeRF-HQ can produce more accurate segmentation
results and is more flexible to a variety of segmentation tasks. Similar to
SAM [32] in 2D segmentation, SANeRF-HQ can automatically segment a given
entire 3D scene in high quality without any user inputs. SANeRF-HQ inherits
the zero-shot performance from SAM, instead of being bounded by pre-trained
models with limited generalizability [38, 5, 47]. SANeRF-HQ is a general
framework which can be readily extended to 4D dynamic NeRFs, where temporal
consistency can be naturally handled in a similar way as our multi-view
consistency. Our preliminary results confirm the potential extension to
dynamic scenes.

##  2 Related Work

###  2.1 Object Segmentation

2D image segmentation is a thoroughly studied area where a lot of progress has
been made, including semantic segmentation [64, 22, 12, 63], instance
segmentation [51, 55, 23, 7, 6, 56], and panoptic segmentation [36, 28, 35,
31, 58, 18]. With the emergence of Transformer-based models [8], Vision
Transformer (ViT) [19, 39, 60, 61] has become increasingly popular as a
backbone structure, and pre-trained features from large models such as MAE
[24, 52, 57] have demonstrated great power in segmentation tasks. Moreover,
significant research effort [14, 62, 28] focuses on universal models to
accomplish multiple segmentation tasks under different training configuration.
However, the performance of these models on open-world images is not
satisfactory, as they cannot go beyond the limits prescribed by the underlying
training datasets.

Thus, latest research has been focusing on open-world segmentation, aiming to
generalize segmentation models to unseen data. The recent advancement in
visual foundation models have attracted great attention. DINOv2 [44] leverages
self-supervised distillation and produces visual features across domains
without any fine-tuning. Segment Anything Model (SAM) [32], as a more
significant breakthrough, shows promising results on promptable segmentation.
Given diverse and plentiful training data, the prompt-based architecture can
enable zero-shot generalization, which extends relevant tasks to wider data
categories. Recent works [45, 16, 66] have already adopted SAM in many
downstream tasks, enhancing the capability of different segmentation models.

In spite of the great success of 2D segmentation, 3D segmentation is
relatively underexplored. Traditional methods are usually based on RGB-D
images [25] or point clouds [54, 53]. However, they require explicit depth or
3D representations as input, and the generalizability is highly restricted by
the scarcity of the dataset and the expensive computational cost. Therefore,
exploiting the vast 2D image datasets and performing 3D segmentation directly
from 2D multi-view images warrants attention from the community.

###  2.2 Segmentation in Neural Radiance Field

The family of Neural Radiance Fields (NeRF) [41, 42, 11, 3, 2, 4] has become
the state-of-the-art for novel view synthesis. Beyond that, requiring only
multi-view images with camera parameters during training, NeRF can also be
regarded as an implicit 3D representation since it captures the 3D structural
details of the scenes. Due to the capability of linking 2D images to 3D
volumes, NeRF shows potential impact in various 3D visual tasks, and numerous
research works have been focusing on 3D object segmentation and scene
decomposition in NeRF.

Figure 1: SANeRF-HQ Pipeline. Our method is composed of three parts: a feature
container (feature cache or feature field), a mask decoder, and a mask
aggregator (object field). It first renders a set of images using a pre-
trained NeRF and encodes their SAM features, which are cached or used to
optimize a feature field. SAM decoder takes the feature maps from the cache or
the feature field, and generates 2D masks based on user prompts. The
aggregator fuses 2D masks from different views to produce an object field.

Semantic-NeRF [65] extends NeRF with an additional branch encoding 3D semantic
labels for semantic-level segmentation. Other research investigated 3D
instance segmentation and make great efforts in solving inconsistency across
views. For example, [38] adopts an 3D object detection method [26] to resolve
2D mask correspondences. [47] utilize linear assignment, while [5] employs a
contrastive loss to optimize 3D instance embeddings. Besides, there are
unsupervised methods [48, 59, 43], which can separate foreground objects from
background. However, these approaches are mostly limited to simple scenes and
often struggle to generalize to complex open-world problems. To fully utilize
2D features from pre-trained models, some research [30, 21, 33] introduces an
extra feature field to the vanilla NeRF model, which can fuse 2D features from
pre-trained models into 3D. For instance, LERF [30] splits images into patches
of different sizes to obtain multi-scale CLIP [27] feature maps that can
supervise the neural field training. ISRF [21] uses DINO [9] features and
K-Means clustering to separate user-selected regions from background. However,
lacking a powerful decoder, the segmentation based on these features is not
accurate and sharp along the boundaries.

With the proposal of SAM, SA3D [10] employs SAM on the NeRF-rendered images
and achieve 3D segmentation from a single-view 2D mask by self-prompting.
Nonetheless this pipeline relies on the first-view mask and is susceptible to
the ambiguity inherent in SAM in delineating intricate structures during its
self-prompting. SAN [13] chooses to distill the SAM encoder with a neural
field to render SAM feature maps from novel views, which are supplied to the
SAM decoder to produce the segmentation. However, it can produce inconsistent
masks in different views, and distilling low resolution SAM feature maps
results in aliasing in the output masks, in the form of jagged mask edges.
Contrary to these approaches, our proposed method directly aggregates masks
for neural field training, which naturally addresses the aliasing and
consistency issues.

##  3 Method

Given a pre-trained NeRF, our method aims to segment any kind of object in 3D,
conditioning on the manual prompts and/or other user supplied inputs. The
SANeRF-HQ pipeline in Figure 1 consists of three major components: a feature
container, a mask decoder, and a mask aggregator. The feature container
encodes the SAM features of images. The mask decoder propagates the user
supplied hints between different views and generates intermediate mask outputs
using the SAM features from the container. Finally, the mask aggregator
integrates the resulted 2D masks into 3D space and utilizes the color and
density fields from NeRF models to achieve high-quality 3D segmentation.

###  3.1 Feature Container

The first step of utilizing SAM is to encode the images into 2D features using
the SAM feature encoder. These features can be used repeatedly when predicting
and propagating masks, thus they can be precomputed or distilled for a scene
and reused for different input prompts.

We consider two different methods for the feature container. The first method
is to compute and cache the features of multiple views. This allows us to
reuse ground-truth SAM features for different user prompts and generate
accurate 2D mask when decoding. However, the cache size is constrained by the
memory available. It also requires extra time to run the encoder if users
choose to supply prompts on any of the uncached novel views.

Another method is to distill the SAM features using a neural field, which is
similarly done in SAN [13] and in [21, 30], where SAM, DINO, or CLIP features
are lifted into 3D. Instead of radiance or density, 3D SAM embeddings are
encoded in a neural field and the same volumetric rendering equation is
applied to render 2D feature maps. Specifically, vanilla NeRF [41] is
formulated as $f(\mathbf{x},\mathbf{d};\Theta_{N})=(\sigma,\mathbf{c})$, where
$\mathbf{x}=(x,y,z)$ is the position of the point, $\mathbf{d}=(\theta,\phi)$
is the view direction, and $\Theta_{N}$ is the set of parameters of the color
and density field. The RGB color at each pixel is estimated through a ray
casting process:

$\begin{array}[]{l}\hat{\mathbf{C}}(\mathbf{r})=\sum_{k=1}^{K}\hat{T}\left(t_{k}\right)\alpha\left(\sigma\left(t_{k}\right)\delta_{k}\right)\mathbf{c}\left(t_{k}\right),\\\
\hat{T}\left(t_{k}\right)=\exp\left(-\sum_{a=1}^{k-1}\sigma\left(t_{a}\right)\delta_{a}\right),\\\
\alpha(x)=1-\exp(-x),\\\ \delta_{k}=t_{k+1}-t_{k},\end{array}$ (1)

where $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$ is the ray emitted from the
camera center passing through that pixel, and $\sigma\left(t_{k}\right)$ and
$\mathbf{c}\left(t_{k}\right)$ are the volume density and color at the point
$\mathbf{o}+t_{k}\mathbf{d}$ along the ray.

To encode SAM features, the SAM embedding at $(\mathbf{x},\mathbf{d})$ is
defined as $\mathbf{f}=f(\mathbf{x},\mathbf{d};\Theta_{f})$, where
$\Theta_{f}$ is the set of parameters of the feature field. The feature
$\hat{\mathbf{F}}$ integrated over ray $\mathbf{r}$ is given as:

$\hat{\mathbf{F}}(\mathbf{r})=\sum_{k=1}^{K}\hat{T}\left(t_{k}\right)\alpha\left(\sigma\left(t_{k}\right)\delta_{k}\right)\mathbf{f}\left(t_{k}\right).\\\
$ (2)

And the feature field is optimized with the MSE loss $\mathcal{L}_{f}$:

$\mathcal{L}_{f}=\sum_{\mathbf{r}\in\mathcal{R}(\Phi)}\left\|\hat{\mathbf{F}}(\mathbf{r})-\mathbf{F}(\mathbf{r})\right\|_{2}^{2},$
(3)

where $\mathcal{R}(\Phi)$ is the set of rays from the feature map $\Phi$ and
$\mathbf{F}(\mathbf{r})$ is the ground-truth feature value of the ray
$\mathbf{r}$.

The feature field enables feature map rendering from any viewpoint
efficiently, as aggregating features in the neural field is typically faster
than running the original SAM encoder. However, the feature maps produced by
the SAN encoder has a relatively low resolution, which can cause severe
aliasing in the rendered feature maps. While this can be alleviated by
augmenting the input camera views and sample more rays during distillation,
the rendered features still deteriorate after distillation. The rendered SAN
features usually fail to retain accurate high frequency spatial information
along boundaries, which consequently leads to jagged mask boundaries after
decoding.

Noting the complementary advantages and disadvantages of the caching method
and the feature distillation method, we conducted experiments on both methods
in the ablation study.

###  3.2 Mask Decoder

The Mask Decoder $\mathbf{Dec}$ takes as input the feature map from the
feature container and generates 2D masks based on the input prompts (e.g., 2D
or 3D points, texts). Figure 2 illustrates the architecture of the decoder,
which is similar to the SAM decoder. The 2D mask decoding can be formulated as

$\mathbf{M}=\mathbf{Dec}(\hat{\Phi},\mathit{prompts}),$ (4)

where $\hat{\Phi}$ is the feature map. NeRF can estimate depth with Equation
5,

$\hat{\mathbf{D}}(\mathbf{r})=\sum_{k=1}^{K}\hat{T}\left(t_{k}\right)\alpha\left(\sigma\left(t_{k}\right)\delta_{k}\right)t_{k},$
(5)

so 3D points can be easily obtained by projecting 2D prompts from users back
to 3D with camera poses. Given a 2D point $(w,h)$, its depth $d(p)$, the
camera intrinsic matrix $\mathbf{K}$, and extrinsic matrix $\mathbf{P}$, the
corresponding 3D point $\mathbf{p}=(x,y,z)^{T}$ in the world space is

$\mathbf{p}=\mathbf{P}^{-1}\mathbf{K}^{-1}\begin{pmatrix}w\cdot d(p)\\\ h\cdot
d(p)\\\ d(p)\end{pmatrix}.$ (6)

The equation to project 3D points in world space to 2D pixel coordinates in
other camera views can be derived likewise.

Figure 2: Mask Decoder Architecture. The decoder consists of a prompt encoder
and an attention decoder. First, the prompts are fed into the prompt encoder.
The attention decoder takes the encoded prompts and the feature map from the
Feature Container, and uses attention to produce 2D masks for the given view.

###  3.3 Mask Aggregator

The decoder cannot produce correct 2D masks if the 3D points after projection
are not visible at certain viewpoints. Furthermore, despite good performance
in most cases, the predicted masks may include artifacts. The innate semantic
ambiguity of SAM predictions can also cause inconsistency across views. Hence,
we aggregate these imperfect 2D masks in the 3D space to generate high-quality
and consistent 3D masks.

####  3.3.1 Object Field

Given the decoder output from different views, the object field can fuse all
2D images and generate accurate 3D masks. The mask is represented by a
$L$-dimensional object identity vector $i$. To represent the identity value,
an additional branch is introduced into the pre-trained NeRF model,
parameterized as $\Theta_{o}$. Different from the function of RGB color, which
is view-dependent, object identity function is defined by
$\mathbf{i}=f(\mathbf{x};\Theta_{o})$, where the view direction vector is not
included in the inputs due to the view invariance of the object masks in 3D.
The equation for mask rendering is similar to that of image rendering:

$\hat{\mathbf{M}}(\mathbf{r})=\mathit{Softmax}\left(\sum_{k=1}^{K}\hat{T}\left(t_{k}\right)\alpha\left(\sigma\left(t_{k}\right)\delta_{k}\right)\mathbf{i}\left(t_{k}\right)\right),\\\
$ (7)

where $\sigma(t_{k})$ is inherited from the pre-trained NeRF model. Volume
density $\sigma$ which interprets the 3D geometry in NeRF makes the object
field aware of the structural information. The object field is trained with
the cross-entropy loss $\mathcal{L}_{o}$:

$\mathcal{L}_{o}(\mathcal{R})=-\frac{1}{|\mathcal{R}|L}\sum_{\mathbf{r}\in\mathcal{R}}\sum_{l=1}^{L}m^{l}(\mathbf{r})\log\hat{m}^{l}(\mathbf{r}),$
(8)

where $\mathcal{R}$ is a set of rays, and $m^{l}(\mathbf{r})$ and
$\hat{m}^{l}(\mathbf{r})$ are the $l$-th entry of the ground-truth mask
$\mathbf{M}(\mathbf{r})$ and the predicted mask
$\hat{\mathbf{M}}(\mathbf{r})$, respectively.

####  3.3.2 Ray-Pair RGB Loss

Segmentation errors in both 3D and 2D are more likely to occur at object
boundaries. One observation is that humans usually distinguish object
boundaries by the color and texture difference on the two sides. Here we
introduce the Ray-Pair RGB loss, aiming to incorporate color and spatial
information to improve the segmentation quality.

Given a batch of rays $\mathcal{R}$, we sample a subset of rays $\mathcal{K}$
from $\mathcal{R}$ as references. For each ray $\mathbf{r}_{k}\in\mathcal{K}$,
we calculate the RGB similarity between $\mathbf{r}_{k}$ and other rays
$\mathbf{r}\in\mathcal{R}$, denoted by
$g(\mathbf{c}(\mathbf{r}),\mathbf{c}(\mathbf{r}_{k}))$, where
$\mathbf{c}(\mathbf{r})$ is the rendered RGB color along $\mathbf{r}$. Next, a
subset $\mathcal{S}_{k}$ is selected from
$\mathcal{R}\setminus\mathbf{r}_{k}$, where for all
$\mathbf{r}_{s}\in\mathcal{S}_{k}$,
$g(\mathbf{c}(\mathbf{r}_{s}),\mathbf{c}(\mathbf{r}_{k}))\geq\tau$,
$\tau\in\mathbb{R}$ is a threshold. The RGB loss is defined as

$\mathcal{L}_{RGB}(\mathcal{R})=\frac{1}{|\mathcal{K}|}\sum_{\mathbf{r}_{k}\in\mathcal{K}}\frac{1}{|\mathcal{S}_{k}|}\sum_{\mathbf{r}_{s}\in\mathcal{S}_{k}}f(\hat{\mathbf{M}}(\mathbf{r}_{k}),\hat{\mathbf{M}}(\mathbf{r}_{s})),$
(9)

where $f$ is a distance function of two probability vectors. This loss
function encourages those rays with similar RGB colors to have similar object
identity predictions. $\hat{\mathbf{M}}(\mathbf{r}_{k})$ is detached from the
compute graph, so gradients from $\mathcal{L}_{RGB}$ only flow through
$\hat{\mathbf{M}}(\mathbf{r}_{s})$. In our implementation, $g,f$ are defined
as:

$g(\mathbf{c}_{0},\mathbf{c}_{1})=\|\mathbf{c}_{0}-\mathbf{c}_{1}\|_{2},$ (10)
$f(\mathbf{M}_{0},\mathbf{M}_{1})=\exp{\left(-w\frac{\mathbf{M}_{0}\cdot\mathbf{M}_{1}}{\max(\|\mathbf{M}_{0}\|_{2}^{2},\;\|\mathbf{M}_{1}\|_{2}^{2})}-\epsilon\right)},$
(11)

where $w$ and $\epsilon$ are hyperparameters.

Sampling Strategy. At the beginning, only $\mathcal{L}_{o}$ is used to
optimize the object field. Concurrently, an error map $\mathbf{E}_{t}$ is
updated to record the difference between the rendered mask and the ground-
truth mask for each training view:

$\mathbf{E}_{t}(\mathbf{r})=f(\mathbf{M}(\mathbf{r}),\hat{\mathbf{M}}(\mathbf{r})),$
(12)

where $f$ is the function in Eq. 11. In practice, the resolution of error maps
is smaller than that of training images to reduce memory usage and increase
update efficiency, so $\mathbf{E}_{t}(\mathbf{r})$ is approximated by
$f(\mathbf{M}(\mathbf{r}^{\prime}),\hat{\mathbf{M}}(\mathbf{r}^{\prime}))$,
where $\mathbf{r}^{\prime}$ is the sample nearest to $\mathbf{r}$ in the low-
resolution error map. After $k$ iterations of training, we include the Ray-
Pair RGB loss $\mathcal{L}_{RGB}$ in training, which is only applied on local
regions sampled according to the error maps. Specifically, a pixel $p$ from a
certain viewpoint with a large error is sampled and reprojected to different
viewpoints in a set $\mathcal{V}$, forming a set of pixels
$\\{p_{v}\,|\,v\in\mathcal{V}\\}$ in different views. From each $p_{v}$, we
cast a set of rays $R_{v,p}$ in the local $N\times N$ image patch around
$p_{v}$. The entire set of rays relevant to $p$, denoted by $\mathcal{R}_{p}$,
is defined as:

$\mathcal{R}_{p}=\bigcup_{v\in\mathcal{V}}R_{v,p},$ (13)

on which we compute the $\mathcal{L}_{RGB}$ for $p$. This allows us to enforce
the loss to rays in different views that are relevant to the same high-error
region.

To maintain the global segmentation results while refining local regions, we
combine Ray-Pair RGB loss with the loss function in Eq. 8 and adopt mixed
sampling: the cross entropy loss is applied to the rays sampled gloablly while
the RGB loss is only applied to certain local regions. The final loss function
$\mathcal{L}$ is

$\mathcal{L}=\mathcal{L}_{o}(\mathcal{R})+\frac{1}{|\mathcal{T}|}\sum_{p\in\mathcal{T}}\mathcal{L}_{RGB}(\mathcal{R}_{p}),$
(14)

where $\mathcal{R}$ is a set of rays sampled randomly from all training views,
and $\mathcal{T}$ is a set of points sampled based on error maps.

##  4 Experiments

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

Figure 3: Comparison with SA3D and ISRF on the Bonsai. SANeRF-HQ can produce
accurate segmentation around boundaries.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

Figure 4: Comparison with SA3D and ISRF on the Table. SANeRF-HQ can preserve
structure details of the table.

###  4.1 Datasets

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

Figure 5: Ablation Study on the Mask Aggregator. The red points in the RGB
images represent the prompts we use in the experiments. By leveraging the 3D
geometry, SANeRF-HQ can produce more accurate segmentation (the first two
rows). Moreover, our method can maintain the consistency since multi-view
information is fused in the object field (the last two rows).

We evaluate our performance on data from multiple datasets, mixed with
synthetic and real-world scenes. We compare the masks projected on 2D images
with ground-truth masks to evaluate our results quantitatively. For data
without ground-truth, we manually annotate object masks. We categorize the
dataset used into the following 5 groups:

  * •

Mip-NeRF 360: a dataset widely used in NeRF research including synthetic and
real-world examples. In our experiments, we use the data in [3].

  * •

LERF: a set of scenes captured by [30], which contains complex real-world
samples.

  * •

LLFF: first used in [40], the dataset contains scenes with only front views.
We use the masks released with [21].

  * •

3D-FRONT: a synthetic indoor scene dataset created in [20], further curated
for NeRF training and scene understanding in Instance-NeRF [38].

  * •

Others: the rest of our evaluation set is composed of the data used in
Panoptic Lifting [47] and Contrastive Lift [5]. The former uses scenes from
existing datasets like Hypersim [46], Replica [49] and ScanNet [17], while the
latter created a new dataset called Messy Rooms.

For each group above, we select scenes with representative objects and
segmentation outcomes that can be clearly identified by humans. In total, 24
scenes are selected, each containing 1 to 3 object segmentation. For those
without ground-truth masks, we use SAM and CascadePSP [15] with manual
annotation to create ground-truth.

Methods | Mip-NeRF 360 | LERF | LLFF | 3D-FRONT | Others
---|---|---|---|---|---
Acc.$\uparrow$ | mIoU.$\uparrow$ | Acc.$\uparrow$ | mIoU.$\uparrow$ | Acc.$\uparrow$ | mIoU.$\uparrow$ | Acc.$\uparrow$ | mIoU.$\uparrow$ | Acc.$\uparrow$ | mIoU.$\uparrow$
SA3D | 99.0 | 88.8 | 94.7 | 52.3 | 98.7 | 90.6 | 97.3 | 78.7 | 99.6 | 88.8
ISRF | 95.2 | 65.7 | 88.5 | 27.9 | 96.7 | 80.0 | 92.4 | 68.5 | 86.5 | 23.6
SAM | 97.9 | 80.4 | 98.0 | 82.9 | 99.1 | 93.7 | 97.4 | 77.7 | 99.2 | 83.6
SAN | 97.6 | 77.2 | 98.1 | 71.0 | 96.7 | 83.0 | 97.0 | 76.8 | 98.4 | 73.0
Ours | 99.2 | 91.0 | 99.0 | 90.7 | 99.3 | 95.2 | 98.6 | 89.9 | 99.6 | 91.1

Table 1: Quantitative Results on Different Datasets.

###  4.2 Metrics

We evaluate the segmentation performance of SANeRF-HQ using mean Intersection
over Union (mIoU) and Accuary (Acc). For each object, we render the object
masks in several novel test views, then combine the pixels in all test views
to calculate the per-object IoU and Acc. The overall mIoU and Acc are averaged
over all objects.

###  4.3 Comparison

We provide the comparison with some zero-shot segmentation methods mentioned
in Section 2.2. These methods leverages large vision models such as SAM [32]
or DINO [9] and can achieve zero-shot segmentation on general scenes when user
prompts are provided. Table 1 shows the quantitative comparison with 4 methods
on 5 different datasets mentioned in Section 4.1. We use point prompts as they
are more unequivocal.

To ensure fair comparison and eliminate bias in prompting, we use the same
point prompts to get the initial masks for SA3D [10]. In addition, since SA3D
requires a single-view mask as input, we manually select a view containing the
major component of the target object, and pick the mask that best matches the
ground-truth, instead of using the predicted scores from SAM to select the
initial mask. To our best extent, we make sure that SA3D is provided with good
initialization. For ISRF [32], strokes are used as prompts, so we manually
connect the point prompts to create strokes. Figure 3 and 4 demonstrate the
qualitative comparison. SA3D uses a self-prompting strategy and iteratively
inverse renders the 2D masks to a voxel grid, whereas our method uses a set of
global prompts and collectively optimize the object field. Despite the use of
IoU rejection, self-prompting may incorrectly include occluded regions in
novel views into prompts, which can accumulate errors, especially in initial
iterations. ISRF lifts DINO features [9] into a neural field, but its
clustering and searching process produces less accurate mask boundaries
compared to SAM.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 6:  Qualitative Results of text prompts and auto segmentation.

###  4.4 More Qualitative Results

Figure 6 illustrates the qualitative results of our methods on other
segmentation tasks. Utilizing Grounding-DINO [37], our method is also capable
of segmenting objects based on text prompts. Additionally, our method can
perform automatic 3D segmentation, utilizing the video rendered by NeRF, the
auto-segmentation function of SAM, and incorporating [16] into the mask
decoder. These results show the potential of our method in readily extending
to dynamic NeRF.

###  4.5 Ablation Study

Mask Aggregator. Based on point prompts, we perform ablation study on the mask
aggregator, comparing the intermediate results from the feature container,
i.e., directly decoding the SAM features from the encoder or the feature
field, to the final outputs of the mask aggregator. The quantitative results
are in Table 1.

Directly propagating point prompts between different views and applying SAM
cannot guarantee cross-view consistency. When prompts are sparse, some 3D
point prompts may be occluded at certain viewpoints, and no mask can be
produced. Even when a large number of prompts are provided, SAM masks may
still cover different objects across views, and this naive approach fail to
utilize masks from other views to collectively refine the results. The same
issue also exists in SAN [10], which distills SAM features with another neural
field and later decodes the rendered features using the decoder. In addition
to the consistency issue, the rendered feature maps from SAN further suffer
from aliasing, and fine spatial semantics in the SAM features along object
boundaries can be lost during interpolation. This can lead to less accurate
segmentation results on the object boundaries, like jagged mask edges. To
ensure fairness in comparison, we introduce enough point prompts to guarantee
masks can be generated from every viewpoint where the target object is
visible.

Figure 5 illustrates the qualitative comparison results. Although SANeRF-HQ
uses the potentially inconsistent segmentation from SAM as input, by
aggregating multi-view information and integrating 3D geometry captured by
NeRF, it can still produce a underlying 3D mask close to the ground-truth
geometry, which guarantees consistent multi-view masks and usually comes with
higher quality.

Ray-Pair RGB Loss. We perform ablations on the Ray-Pair RGB loss in Section
3.3.2. Table 2 shows the quantitative results, where the Ray-Pair RGB loss
slightly enhances the mask quality. The visual improvement of the masks
illustrated in Figure 7 is more significant. The loss helps fill the missing
interior and boundaries of the masks by enforcing a local match between the
similarity in labels, and the similarity in appearance.

For more ablation study, please refer to our supplementary materials.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 7:  Qualitative Results of Ray-Pair RGB Loss. The Ray-Pair RGB loss can
help to recover local regions and make the results more solid.

Metrics | $\mathcal{L}_{RGB}$ | Mip-NeRF | LERF | LLFF | 3D-FRONT | Others
---|---|---|---|---|---|---
mIoU.$\uparrow$ | w/o | 91.0 | 88.3 | 95.2 | 89.3 | 90.9
w/ | 91.3 | 90.7 | 95.8 | 89.9 | 91.1
Acc.$\uparrow$ | w/o | 99.2 | 98.9 | 99.4 | 98.6 | 99.6
w/ | 99.2 | 99.0 | 99.5 | 98.7 | 99.6

Table 2: Ablation Results of the Ray-Pair RGB Loss.

##  5 Conclusion

In this paper, we introduced the Segment Anything for NeRF in High Quality
(SANeRF-HQ) framework. By combining the strengths of the Segment Anything
Model (SAM) for open-world object segmentation and NeRF for aggregating
information from multiple viewpoints, SANeRF-HQ represents a significant
advancement in high-quality 3D segmentation. Our method was quantitatively and
qualitatively evaluated on various NeRF datasets, which demonstrates SANeRF-
HQ’s advantages over previous state-of-the-art methods. Furthermore, we
demonstrate the potential of extending our work to 4D dynamic NeRF object
segmentation (see supplementary materials). SANeRF-HQ holds promise for
contributing significantly to the evolving landscape of 3D computer vision and
segmentation techniques.

## References

  * Attal et al. [2023] Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O’Toole, and Changil Kim.  HyperReel: High-fidelity 6-DoF video with ray-conditioned sampling.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Barron et al. [2021] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Barron et al. [2022] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Barron et al. [2023] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman.  Zip-nerf: Anti-aliased grid-based neural radiance fields.  _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023.
  * Bhalgat et al. [2023] Yash Bhalgat, Iro Laina, João F Henriques, Andrew Zisserman, and Andrea Vedaldi.  Contrastive lift: 3d object instance segmentation by slow-fast contrastive fusion.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2023.
  * Bolya et al. [2019] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.  Yolact: Real-time instance segmentation.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * Bolya et al. [2020] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.  Yolact++: Better real-time instance segmentation.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 2020.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European Conference on Computer Vision (ECCV)_ , pages 213–229. Springer, 2020.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Cen et al. [2023] Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Chen Yang, Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, and Qi Tian.  Segment anything in 3d with nerfs.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2023.
  * Chen et al. [2022] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In _European Conference on Computer Vision (ECCV)_ , 2022.
  * Chen et al. [2018] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In _European Conference on Computer Vision (ECCV)_ , pages 801–818, 2018.
  * Chen et al. [2023] Xiaokang Chen, jiaxiang Tang, Diwen Wan, Jingbo Wang, and Gang Zeng.  Interactive segment anything nerf with feature imitation.  _arXiv preprint arXiv:2211.12368_ , 2023.
  * Cheng et al. [2021] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  _arXiv_ , 2021.
  * Cheng et al. [2020] Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, and Chi-Keung Tang.  CascadePSP: Toward class-agnostic and very high-resolution segmentation via global and local refinement.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Cheng et al. [2023] Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee.  Tracking anything with decoupled video segmentation.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 1316–1326, 2023.
  * Dai et al. [2017] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * De Geus et al. [2018] Daan De Geus, Panagiotis Meletis, and Gijs Dubbelman.  Panoptic segmentation with a joint semantic and instance segmentation network.  _arXiv preprint arXiv:1809.02110_ , 2018.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Fu et al. [2021] Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al.  3d-front: 3d furnished rooms with layouts and semantics.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 10933–10942, 2021.
  * Goel et al. [2023] Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and P.J. Narayanan.  Interactive Segmentation of Radiance Fields.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Guo et al. [2022] Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu.  Segnext: Rethinking convolutional attention design for semantic segmentation.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 35:1140–1156, 2022.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2961–2969, 2017.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16000–16009, 2022.
  * Hou et al. [2019] Ji Hou, Angela Dai, and Matthias Nießner.  3d-sis: 3d semantic instance segmentation of rgb-d scans.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Hu et al. [2023] Benran Hu, Junkai Huang, Yichen Liu, Yu-Wing Tai, and Chi-Keung Tang.  Nerf-rpn: A general framework for object detection in nerfs.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Ilharco et al. [2021] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip, 2021.
  * Jain et al. [2023] Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi.  OneFormer: One Transformer to Rule Universal Image Segmentation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Ke et al. [2023] Lei Ke, Mingqiao Ye, Martin Danelljan, Yifan Liu, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu.  Segment anything in high quality.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2023.
  * Kerr et al. [2023] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik.  Lerf: Language embedded radiance fields.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023.
  * Kirillov et al. [2019] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár.  Panoptic feature pyramid networks.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6399–6408, 2019.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Kobayashi et al. [2022] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann.  Decomposing nerf for editing via feature field distillation.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Li et al. [2022] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al.  Neural 3d video synthesis from multi-view video.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5521–5531, 2022.
  * Li et al. [2019] Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, and Xingang Wang.  Attention-guided unified network for panoptic segmentation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7026–7035, 2019.
  * Li et al. [2021] Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Tong Lu, and Ping Luo.  Panoptic segformer: Delving deeper into panoptic segmentation with transformers, 2021.
  * Liu et al. [2023a] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023a.
  * Liu et al. [2023b] Yichen Liu, Benran Hu, Junkai Huang, Yu-Wing Tai, and Chi-Keung Tang.  Instance neural radiacne field.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023b.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 10012–10022, 2021.
  * Mildenhall et al. [2019] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  _ACM TOG_ , 2019.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _European Conference on Computer Vision (ECCV)_ , 2020.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Trans. Graph._ , 41(4):102:1–102:15, 2022.
  * Niemeyer and Geiger [2021] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Rajič et al. [2023] Frano Rajič, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, and Fisher Yu.  Segment anything meets point tracking.  _arXiv preprint arXiv:2307.01197_ , 2023.
  * Roberts et al. [2021] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind.  Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Siddiqui et al. [2023] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder.  Panoptic lifting for 3d scene understanding with neural fields.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9043–9052, 2023.
  * Stelzner et al. [2021] Karl Stelzner, Kristian Kersting, and Adam R Kosiorek.  Decomposing 3d scenes into objects via unsupervised volume segmentation.  _arXiv:2104.01148_ , 2021.
  * Straub et al. [2019] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe.  The Replica dataset: A digital replica of indoor spaces.  _arXiv preprint arXiv:1906.05797_ , 2019.
  * Tang [2022] Jiaxiang Tang.  Torch-ngp: a pytorch implementation of instant-ngp, 2022.  https://github.com/ashawkey/torch-ngp.
  * Tian et al. [2020] Zhi Tian, Chunhua Shen, and Hao Chen.  Conditional convolutions for instance segmentation.  In _European Conference on Computer Vision (ECCV)_ , pages 282–298. Springer, 2020.
  * Tong et al. [2022] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 35:10078–10093, 2022.
  * Tuan Duc Ngo [2023] Khoi Nguyen Tuan Duc Ngo, Binh-Son Hua.  Isbnet: a 3d point cloud instance segmentation network with instance-aware sampling and box-aware dynamic convolution.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Vu et al. [2022] Thang Vu, Kookhoi Kim, Tung M. Luu, Xuan Thanh Nguyen, and Chang D. Yoo.  Softgroup for 3d instance segmentation on 3d point clouds.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Wang et al. [2020a] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li.  SOLO: Segmenting objects by locations.  In _European Conference on Computer Vision (ECCV)_ , 2020a.
  * Wang et al. [2020b] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen.  Solov2: Dynamic and fast instance segmentation.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020b.
  * Wei et al. [2022] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer.  Masked feature prediction for self-supervised visual pre-training.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 14668–14678, 2022.
  * Xiong et al. [2019] Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun.  Upsnet: A unified panoptic segmentation network.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8818–8826, 2019.
  * Yu et al. [2022] Hong-Xing Yu, Leonidas J. Guibas, and Jiajun Wu.  Unsupervised discovery of object radiance fields.  In _ICLR_ , 2022.
  * Yuan et al. [2021] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan.  Tokens-to-token vit: Training vision transformers from scratch on imagenet.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 558–567, 2021.
  * Yue et al. [2021] Xiaoyu Yue, Shuyang Sun, Zhanghui Kuang, Meng Wei, Philip HS Torr, Wayne Zhang, and Dahua Lin.  Vision transformer with progressive sampling.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 387–396, 2021.
  * Zhang et al. [2021] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy.  K-Net: Towards unified image segmentation.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Zhao et al. [2017] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2881–2890, 2017.
  * Zheng et al. [2021] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6881–6890, 2021.
  * Zhi et al. [2021] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison.  In-place scene labelling and understanding with implicit scene representation.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Zou et al. [2023] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee.  Segment everything everywhere all at once.  _arXiv preprint arXiv:2304.06718_ , 2023.

##  Appendix A Implementation Details

We use torch-ngp [50] as our initial NeRF implementation. When we use 3D
points as prompts in evaluation, the views containing less than $k$ visible
points get filtered out automatically and will not be used to train the object
field, where $k$ is a hyperparameter, depending on the total number of input
points.

For both the SAM feature field and the object field, we use a hash grid as in
[42] with 16 levels and feature dimension of 8 per level. The lowest and
highest level are of resolution 16 and $2^{19}$, respectively. We use a
5-layer 256-hidden dimensional MLP with skip connections and Layer
Normalization after the feature field hash grid, and a 3-layer 256 hidden
dimensional MLP with skip connections after the object field hash grid. In
addition to the features from their respective hash grid, both MLPs also take
the features from the density field as input, where feature MLP also takes the
viewing directions as input. The initial radiance and density field, the SAM
feature field, and the object field are trained for 15,000, 5,000, and 600
iterations, respectively. All models are trained on an NVIDIA RTX 4090 GPU.

Ray-Pair RGB loss is included after 300 iterations of warm-up. We use error
maps downsampled by 4 times compared to original training images for Ray-Pair
RGB loss sampling. In each iteration, we update the error maps using the
training ray batch, and for every 200 iterations, we perform a full update for
all error map pixels. During sampling, we independently sample initial rays on
each error map weighted by their errors, reproject them onto each view, and
subsequently sample 32 additional rays in each $N\times N$ patch centered at
the reprojected pixels randomly. Here we choose $N=8$ or 16. A subset of 20
rays are then sampled from each set as references in the Ray-Pair RGB loss.

##  Appendix B More Ablation Study

SAM Backbones. We perform ablation study on different types of backbones for
feature extracting and decoding, which are SAM and SAM-HQ [29].

Feature Container Method. We propose two methods to store the pre-computed
features in feature container Section 3.1. The first one is to store the
feature maps in a cache. The second one is to distill the features into a
feature field and render the feature map during inference. We investigate the
performance of these two methods on our evaluation set.

Table C.3 demonstrates our the quantitative results on the ablation study.
SAM-HQ with a cache container performs the best in most cases, while SAM-HQ
with a distillation container does not produce similar results as others on
our evaluation set. Nevertheless, our method demonstrates advantages over
other 3D segmentation methods in general. Notice that we do not choose the
best results from all of them in our comparison to ensure the fairness.
Considering that all the previous methods leverage SAM, we use SAM with
feature distillation as our feature container in comparison.

##  Appendix C Comparison with Instance Segmentation Methods

We also compare our method with some instance segmentation methods. The
instance segmentation methods in NeRF mentioned in our related works do not
require user prompts and can automatically generate segmentation of salient
objects in NeRF. These methods also leverage 2D segmentation methods for NeRF
training but they mainly focus on the challenge of 3D consistency. Despite
their different configurations and issues of concern, we still provide the
comparison with these automatic end-to-end pipelines, showing that our prompt-
based method can produce comparable results to these state-of-the-art auto-
segmentation methods. Instance-NeRF [38] is a training-based methods so we
only compare with it on 3D-FRONT dataset. Figure C.1 and Table C.1 illustrates
the visual results and quantitative comparison respectively. For Panoptic
Lifting [47] and Contrastive Lift [5], we also compare on the scenes they
mentioned in the papers to ensure the fairness. Results are shown in Figure
C.2 and Table C.2.

We use the objects in our evaluation sets as targets and choose the object
that has the largest IoU with the target object as the predicted results of
the instance segmentation methods. Notice that we only compare with those
methods on the datasets mentioned in their papers, since they do not leverage
SAM to achieve zero-shot generalization.

Metrics | Ours | Instance-NeRF
---|---|---
Acc.$\uparrow$ | 98.7 | 99.2
mIoU.$\uparrow$ | 89.9 | 92.8

Table C.1: Comparison with Instance-NeRF on 3D-FRONT.

Metrics | Ours | Panoptic | Contrastive
---|---|---|---
Lifting | Lift
Acc.$\uparrow$ | 99.6 | 94.3 | 94.1
mIoU.$\uparrow$ | 91.1 | 84.5 | 81.5

Table C.2: Comparison with Panoptic Lifting and Contrastive Lift on the data
mentioned in their papers.

(a)

(b)

(c)

(d)

(e)

(f)

Figure C.1: Qualitative Comparison with Instance-NeRF. Zoom in for details
especially along the segmentation boundaries.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure C.2: Qualitative Comparison with Panoptic Lifting and Contrastive Lift.
Zoom in for details.

Backbone | Container | Mip-NeRF 360 | LERF | LLFF | 3D-FRONT | Others
---|---|---|---|---|---|---
Acc.$\uparrow$ | mIoU.$\uparrow$ | Acc.$\uparrow$ | mIoU.$\uparrow$ | Acc.$\uparrow$ | mIoU.$\uparrow$ | Acc.$\uparrow$ | mIoU.$\uparrow$ | Acc.$\uparrow$ | mIoU.$\uparrow$
SAM | Cache | 99.3 | 93.6 | 98.9 | 90.3 | 99.5 | 95.8 | 97.6 | 84.8 | 99.6 | 91.1
Distillation | 99.2 | 91.0 | 99.0 | 90.7 | 99.3 | 95.2 | 98.6 | 89.9 | 99.6 | 91.1
SAM-HQ | Cache | 99.4 | 94.4 | 99.2 | 93.1 | 99.6 | 96.8 | 98.9 | 91.8 | 99.5 | 90.1
Distillation | 98.9 | 88.1 | 98.0 | 86.3 | 98.8 | 89.8 | 96.3 | 79.4 | 99.4 | 89.4

Table C.3: Quantitative Results on different backbones and feature containers.

##  Appendix D Extending to Dynamic NeRFs

We present a preliminary demonstration in Figure D.1 on the easy extension of
our method to 4D dynamic NeRF representations. We use HyperReel [1] as our
reference NeRF representation and only supply user prompts for the first frame
of each camera. The prompts are fed into SAM to retrieve initial masks, whose
bounding boxes are used as the prompts for the next frame. This process
repeats until masks are acquired from all video frames, after which we proceed
to object field training as in previous static scene cases. The scene is from
the Neural 3D Video dataset [34].

(a)

(b)

(c)

(d)

(e)

(f)

(g)

Figure D.1: Demonstration of Applying SANeRF-HQ to Dynamic NeRFs. The first
row are the NeRF RGB images over time, and the second row are the masks from
SANeRF-HQ, which is also dynamic. Our method can be easily adapted to dynamic
NeRFs and still retains reasonable performance. The implementation is based on
HyperReel, and the cook spinach scene shown is from the Neural 3D Video
dataset.

##  Appendix E More Qualitative Results

We demonstrate extra qualitative comparisons between our method and other
zero-shot 3D segmentation methods as mentioned in the main paper. The results
are given in Figures E.1, E.2, E.3, E.4. Please watch the video for more
qualitative results.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure E.1: Comparison with SA3D and ISRF on the Replica Room. Data is from
the Others subset. SANeRF-HQ can maintain the object structure while excludes
the background.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure E.2: Comparison with SA3D and ISRF on the Shoe Rack. Data is from the
LERF subset. Our method can reproduce the segmentation details even with some
occlusion.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure E.3: Comparison with SA3D and ISRF on the Hypersim. Data is from the
Others subset. ISRF contains too many false positives, while SA3D cannot cover
the whole object.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure E.4: Comparison with SA3D and ISRF on the Espresso. Data is from the
LERF subset. Our method produces the most reasonable segmentation in the
distant, complex setting.


# A real-time, hardware agnostic framework for close-up branch reconstruction
using RGB data
††thanks: This work has been submitted to the IEEE for possible publication.
Copyright may be transferred without notice, after which this version may no
longer be accessible. ††thanks: This research is supported in part by USDA-
NIFA through the Agriculture and Food Research Initiative, Agricultural
Engineering Program (award No. 2020-67021-31958) and the AI Research
Institutes program supported by NSF and USDA-NIFA under the AI Institute:
Agricultural AI for Transforming Workforce and Decision Support (AgAID) (award
No. 2021-67021-35344). ††thanks: 1Collaborative Robotics and Intelligent
Systems (CoRIS) Institute, Oregon State University, Corvallis OR 97331, USA
{youa, cindy.grimm<EMAIL_ADDRESS>2Wageningen
University & Research, 6708 PB Wageningen, The Netherlands
<EMAIL_ADDRESS>
Alexander You1, Jochen Hemming2, Cindy Grimm1, Joseph R. Davidson1

###### Abstract

Creating accurate 3D models of tree topology is an important task for tree
pruning. The 3D model is used to decide which branches to prune and then to
execute the pruning cuts. Previous methods for creating 3D tree models have
typically relied on point clouds, which are often computationally expensive to
process and can suffer from data defects, especially with thin branches. In
this paper, we propose a method for actively scanning along a primary tree
branch, detecting secondary branches to be pruned, and reconstructing their 3D
geometry using just an RGB camera mounted on a robot arm. We experimentally
validate that our setup is able to produce primary branch models with 4-5 mm
accuracy and secondary branch models with 15$\degree$ orientation accuracy
with respect to the ground truth model. Our framework is real-time and can run
up to 10 cm/s with no loss in model accuracy or ability to detect secondary
branches.

##  I Introduction

Robotic fruit tree pruning is an active area of research [1] motivated by
rising production costs and increasing worker shortages. Over the past several
years, a team at the AI Institute for Transforming Workforce and Decision
Support (https://agaid.org/) has been developing methods for robotic pruning.
In our most recent field trial [2], we demonstrated an integrated system
capable of scanning a tree, searching for branches to prune, and then
executing precision cuts using a combination of visual servoing and admittance
control. In contrast with other existing pruning systems that use stereo
vision for perception, our system relied solely on an RGB camera. However, one
notable deficiency of the system was that because it did not use depth data,
we assumed that each pruning point was a fixed distance away from the camera
and then let the visual servoing controller correct for this, resulting in
wasted execution time. A more efficient approach would be to obtain a 3D
estimate of each pruning point using a 3D model of the tree (still using just
an RGB camera).

Historically, the predominant camera-based method for creating 3D models of
trees has been to use a stereo vision system to obtain a 3D point cloud of the
entire tree, which is then post-processed into skeleton or mesh form to
describe the topology of the tree. While this approach may be necessary for
trees with more complex pruning rules, it is largely unnecessary for simpler
pruning tasks in which pruning decisions can be made “on the fly”.
Furthermore, a full 3D reconstruction requires an imaging system that is
either separate from the pruning robot itself to be able to view the whole
tree (leading to calibration issues) or a lengthy scanning procedure that
moves the imaging system on the end of a robot, processes the data, and then
revisits the pruning points.

Figure 1: Our framework uses 2D RGB data and knowledge of the robot’s
kinematics, camera intrinsics, and hand-eye calibration to follow along a
primary branch and create highly accurate 3D branch reconstructions of the
primary and secondary branches that can be used for making pruning decisions
and executing cuts. Figure 2: A system overview of our framework for branch
scanning and reconstruction. It consists of 4 primary system nodes: A
segmentation node producing binary masks; a point tracking node and
triangulation node that reads in RGB images and returns 3D estimates for
queried 2D pixels in an image; the core node that performs the branch
reconstruction; and the controller node that moves the robot along the 3D
primary branch. Dotted lines represent asynchronous system inputs.

In this paper, we introduce our framework (Fig. 1) designed to address the
aforementioned limitations. Our system is capable of scanning up a primary
branch, detecting secondary branches coming off of the primary branch, and
creating 3D branch models that are accurate enough to be used for on the fly
pruning decisions. The core components of the system are the controller that
moves the camera on the robot’s end effector to follow a primary branch and
the logic and active camera control for extracting the 3D geometry of the
primary branch and its secondary branches. Our work is the first such work to
focus on real-time modeling for close-up tree views where only a portion of
the tree is visible. One benefit of this approach is that it uses an arbitrary
off-the-shelf RGB camera mounted in an eye-in-hand configuration — no
specialized stereo or RGB-D camera.

We evaluate our framework in both simulated and real environments and show
that the 3D reconstruction is remarkably accurate, with an average primary
branch position accuracy of 4mm, an average secondary branch orientation error
of 15$\degree$, and a branch detection rate of 76% on a real robot.
Furthermore, we can run at speeds of at least 10 cm/s without any noticeable
loss of accuracy.

##  II Related Work

The problem of creating a topological model of a tree (or any 3D object) is
known as skeletonization. We previously developed our own skeletonization
algorithm [3] that uses semantic knowledge about a specific tree architecture,
plus a convolutional neural network measuring connectedness between two
points, to model the quality of the reconstruction. This objective function
was used to search for an optimal reconstruction. Other methods also exist for
apple trees [4, 5, 6], grapevines [7], jujube trees [8], and rose bushes [9].
All of these methods convert point clouds directly to skeletons, with the
exception of [7] that instead builds a 3D model by matching canes across 2D
images (similar to our own work). Also of note is the method of [9] that
adopts two key principles we also explore: segmentation masks to rectify 3D
estimates and real-time algorithms. However, all of these methods focus on
skeletonization of a full global scan of the corresponding plant, whereas our
goal here is to focus on local branch reconstruction.

The approach we use is also inspired by structure from motion (SfM)
techniques. These methods use a stream of 2D camera images with (potentially
unknown) camera poses to create a 3D reconstruction of a scene by matching
features between neighboring images and running an optimization method (bundle
adjustment) to estimate the camera poses and construct the 3D scene. Such
methods have been used for orchard-related applications such as fruit
detection [10, 11], though none seem to exist for modeling tree branches.
Although full SfM algorithms are often computationally expensive and output
high density point clouds, they can be scaled down to track just a few points,
as we do here.

Though not the main focus of the work, we also touch on active vision, i.e.
using visual feedback to plan robot movement with the goal of maximizing
information gain (detecting prunable branches). Our framework extends upon our
previous Follow the Leader (FTL) controller [12] tasked with tracking a
primary branch, though it did not perform 3D modeling of the primary and
secondary branches. In the field of agri-food robotics, examples of similar
work that locate objects of interest in the presence of occlusions using an
eye-in-hand configuration can be found in [13] (greenhouse tomato
localization) and [14, 15] (sweet pepper detection).

##  III System Design

Our work focuses on pruning in modern, high-density orchards. We assume that
each tree architecture consists of primary branches (vertical or horizontal)
and secondary side branches that grow perpendicularly off the primary
branches. These side branches are candidates for pruning. Regardless of the
specific fruit cultivar (e.g. apple, sweet cherry, currant, etc.), training
system (V-trellis, spindle, UFO), or specific pruning rules, all share a need
to search for secondary branches, model their locations, and execute cuts. The
goals of our framework are to move the camera along the primary branch at a
set distance of $z_{target}$, keep the branch centered in the camera’s view,
and identify secondary branches coming off the primary branch. The output is a
3D branch reconstruction that captures the 3D geometry of the primary branch
(skeleton and radius) along with the angle and position of the secondary
branches.

A flow chart of our system is shown in Figure 2. The core component of our
framework is the system that constructs the 3D skeleton of the tree using a
stream of binary masks as input, which are produced by our segmentation
framework. The 3D modeling node combines 2D information from the mask with 3D
information from the point tracking and triangulation node to iteratively
build a model of the scanned tree. The movement of the robot is handled by the
primary branch following controller node that uses the existing 3D model to
follow along the branch at a set distance.

Our segmentation system is described in detail in [16]. It produces masks by
augmenting RGB images with optical flow and processing them through a
generative adversarial network. For this paper, we focus on the other three
novel components that allow for 3D perception of the scene as well as
discovery of occluded branches.

##  IV 3D Branch Reconstruction

In this section, we discuss the subsystem that iteratively constructs the 3D
model of the primary and secondary branches. Each branch model consists of a
linear sequence of 3D points, as well as radius estimates at each point. The
main steps are shown in the 3D Modeling Node box in Figure 2. At each
iteration (which occurs after the robot has moved 10 mm from the last
successful iteration), the modeling node takes in a segmentation mask and
attempts to fit 2D Bézier curves for the primary and secondary branches and
estimate their 2D radii (Figure 2, (1)); this operation is discussed below in
Section IV-A. We then attempt to correct the existing 3D primary branch model
by projecting it into the current image frame and checking if the projected
points are consistent with the detected primary branch model (Figure 2, (2)),
i.e. if at least 60% of the projected branch points are in the foreground mask
and within 4 px of the 2D curve. There are several outcomes of the
consistency-checking operation:

  * •

(Consistent) All inconsistent 3D branch points (i.e. ones that are over 4 px
away from the curve) are deleted from the model and reinitialized.

  * •

(Inconsistent) We assume the mask data is bad and terminate processing of the
existing mask.

  * •

If we receive 3 inconsistent masks in a row, we assume the 3D model itself is
wrong and delete all of the in-frame points from the 3D model. We then
continue processing the existing mask.

Once we filter out inconsistent primary branch points, we then take the fit 2D
primary Bézier curve and subsample it to a representative set of 2D pixels
(Figure 2, (3)) by matching projected primary branch points to the closest
pixel on the curve and adding new points to the model as they become visible,
spacing out new points every 30 px. We then pass the 2D points through the
point triangulation node (Section IV-B) to obtain 3D estimates of the points.
We filter out bad 3D estimates by discarding those with a depth value above
1.0 m or a maximum reprojection error above 4 px. From the remaining points,
we attempt to fit a 3D cubic Bézier curve using RANSAC with an inlier
threshold of 3 cm. If the fit 3D curve has 60% inliers, we update the 3D
primary branch model by averaging in inlier 3D points from the fit curve into
their corresponding point in the 3D branch model (Figure 2, (4)). Furthermore,
we estimate the 3D radius of each point as $zr_{px}/f$, where $z$ is the
camera frame $z$-coordinate of the point, $r_{px}$ is the point’s
corresponding 2D pixel radius estimate, and $f$ is the camera focal length.

We follow the same procedure with identified 2D secondary branch Bézier curves
to yield 3D secondary branch models by matching each projected 3D secondary
branch to a 2D Bézier curve, checking their consistency, feeding the
representative 2D points through the point triangulation node, and fitting a
3D Bézier curve. Furthermore, to prevent adding background branches, we
perform a check to verify that the 3D secondary branch actually intersects
with the 3D primary branch by taking the ray at start of the secondary Bézier
curve and checking if it comes within 3 cm of any edge in the primary model.
If so, the branch is considered a match and is added to the list of secondary
branches.

###  IV-A 2D curve fitting

Figure 3: The process for extracting 2D primary and secondary branches from a
skeleton. We start with the medial axis skeleton and find the subpath with the
best fit to be the main branch. We then check skeletal paths extending off the
primary branch to see if they are sufficiently long and a Bézier curve can be
fit to them. We subsample the curves to identify pixels for which we will
estimate 3D positions.

Figure 3 shows the process of locating the primary and secondary branch curves
in a 2D mask. We start by using medial axis skeletonization [17] to obtain a
pixel skeleton and radius estimates for each skeletal pixel. The pixel
skeleton is then turned into a graph in which the nodes are all pixels at an
endpoint (degree 1) or at a junction (degree 3 or above), with an edge between
two nodes if they are connected by a linear segment of the skeleton (Step 2).
These edges are oriented to point in the up ($-Y$) direction of the image,
resulting in a multi-tree directed graph with no cycles.

To fit the primary Bézier curve, we run an exhaustive search of all possible
pixel paths in the multi-tree (Step 3). For each pixel path, we perform a
least squares regression to fit a cubic Bézier curve to the set of pixel
points and select all inlier skeleton pixels that are within a given threshold
(4 px) of the Bézier curve. The score of the pixel path is the size of the set
of unique $y$-values of all inlier pixels. Once we have identified the highest
scoring pixel path and associated Bézier curve $B(t)$, we subsample the curve
to obtain representative 2D pixels (Step 4, blue curve) and match each
subsampled pixel to the closest skeletal point to obtain its radius.

Once we identify the primary branch curve, we then identify all secondary
branch curves. First, from the original (undirected) pixel skeleton graph, we
identify all edges that are connected to the inside of the identified primary
branch; these edges represent the starts of potential subpaths for secondary
branches. For each edge, we run an exhaustive search like before to find the
subpath with the best fitting Bézier curve. If the curve pixel length and the
inlier rate is high enough, we add the detection to the set of “potential”
secondary branches.

###  IV-B 3D triangulation via pixel tracking

Figure 4: Using the Persistent Independent Particles point tracker, along with
knowledge of the camera’s poses and intrinsics, we can obtain 3D estimates of
an arbitrary subset of pixels in an image.

To obtain 3D point estimates from the identified 2D pixels, we use the
Persistent Independent Particles (PIPs) method [18] to track a set of queried
pixels over $M=8$ video frames, which are accumulated as the robot moves every
1.0 mm (Figure 4). We then combine each pixel’s trajectory with knowledge of
the camera’s world frame pose and intrinsics at each frame to triangulate each
pixel’s 3D coordinate.

Given a fixed camera calibration matrix $\mathbf{K}\in\mathbb{R}^{3\times 3}$
and a series of $M=8$ camera poses defined by rotation matrices
$\mathbf{R}_{i}\in SO(3)$ and translations $T_{i}\in\mathbb{R}^{3}$, we obtain
a series of projection matrices
$\mathbf{P}_{i}=\mathbf{K}\begin{bmatrix}\mathbf{R}_{i}&T_{i}\end{bmatrix}\in\mathbb{R}^{3\times
4}$. Each projection matrix transforms homogeneous 3D points into 2D plus
depth image coordinates. Denoting the rows of $\mathbf{P}_{i}$ as
$\mathbf{p}_{i}^{1}$, $\mathbf{p}_{i}^{2}$, and $\mathbf{p}_{i}^{3}$, and
given the list of $M$ pixel correspondences $(u,v)_{j,i}$ ($j$ keypoints for
each of $i=M$ camera frames), we can define the matrix
$\mathbf{D}_{j}\in\mathbb{R}^{2M\times 4}$ as follows:

$\mathbf{D}_{j}=\begin{bmatrix}\mathbf{D}_{j,1}\\\ \vdots\\\
\mathbf{D}_{j,M}\end{bmatrix},\mbox{ where
}\mathbf{D}_{j,i}=\begin{bmatrix}\left(u_{j,i}\mathbf{p}_{i}^{3}\right)-\mathbf{p}_{i}^{1}\\\
\left(v_{j,i}\mathbf{p}_{i}^{3}\right)-\mathbf{p}_{i}^{2}\end{bmatrix}$ (1)

The problem of finding a point $\mathbf{X}_{j}\in\mathbb{R}^{3}$ that
minimizes the reconstruction error across all views is the same as finding a
point
$\mathbf{\underline{X}}_{j}\in\mathbb{R}^{4},\|\mathbf{\underline{X}}_{j}\|=1$
that minimizes the algebraic least-squares error
$\|\mathbf{D}_{j}\mathbf{\underline{X}}_{j}\|^{2}$, which can be solved with
singular value decomposition (SVD).

##  V 3D branch-following controller

In this section, we discuss the controller used to follow along the primary
branch (Figure 5). This controller also performs the function of periodically
rotating the camera around the primary branch, with the goal of gaining a view
of branches that may be occluded by the original view.

Figure 5: Components of the primary branch following controller. The
controller follows the gradient of the model in front of the camera, with
velocity offsets to keep a fixed distance from the branch and to keep the
branch centered in the image (top-right). To gain a view of potentially
occluded branches, it also periodically cycles through 3 camera orientations
at an angle $\theta$ (bottom-right, shown from a top view).

To output the camera-frame velocity that tracks the primary branch, we use the
3D primary branch model described in the previous section. Supposing that the
camera has an image width of $w$ and height $h$, we project the 3D primary
branch model into the camera image frame and identify the pixel that falls
closest to the vertical center at $\frac{h}{2}$. Let the identified pixel be
$(u,v)$ with the corresponding camera frame 3D point $p=(x,y,z)$ having a
gradient of $\nabla$. The formula for the velocity is a simple proportional
controller with scaling factors $k_{u}$ for the horizontal pixel difference
and $k_{z}$ for the target distance error (this velocity is subsequently
scaled to a predetermined speed):

$v_{tool}=\nabla+k_{u}\left(u-\frac{w}{2}\right)+k_{z}\left(z-z_{target}\right)$
(2)

In addition, to gain a view of branches that are either directly in front of
or behind the primary branch, the controller can also be configured to
periodically rotate the camera by a given angle $\theta$ in the world
$Z$-plane about $p$ after moving a set distance up in the world (Figure 5,
bottom-right). It is useful to characterize this distance as a frequency with
respect to the length of the camera’s vertical field of view at a distance of
$z_{target}$; ideally the frequency should be greater than 1 to make sure the
camera will not accidentally skip potentially occluded branches. Note that we
disable mask generation while the camera is rotating since the optical flow-
based segmentation scheme relies on translational camera movement for clean
optical flow estimates.

##  VI Experiments

To validate the quality of the models produced by our framework, we performed
experiments both in simulation and in a lab environment on real tree branches
(Figure 6).

###  VI-A Simulated Experiments

The simulated experiment uses the Blender environment (Figure 6 left) to
render a mock primary branch with secondary side branches. The primary branch
is modeled as a randomized cubic Bézier curve extending from $z=0$ to $1$
meters in the world frame. Within the range of $z\in[0.325,0.75]$, we add side
branches by uniformly sampling $z$-values and placing branches with random
curvature at random orientations. The branch elevations are limited to
$[-15\degree,45\degree]$, while the yaw can be any value.

Figure 6: Setups used for evaluating our framework. Left: Our simulated
Blender environment showing a mock spindle tree and a rendered camera image.
Right: Our physical setup for the experiments (top), the detachable probe used
to measure the ground truth (bottom-left), and the four branches we attempted
to reconstruct (bottom-right).

The simulated robot is a UR5e manipulator (Universal Robots, Odense, Denmark)
with a 320x240 camera coincident with the end effector frame. Each experiment
started with sending the simulated robot to a home position at $z=0.275$ and
running the framework until completion (the robot reaches $z=0.75)$ or failure
(typically due to hitting a kinematic singularity).

To evaluate the ability of the controller to accurately detect branches and
reconstruct their 3D geometry, we ran experiments with 5 different parameter
sets for the branch-following controller. The baseline parameter set is no
rotation in which the robot simply scans straight up the tree without changing
its viewpoint. The other parameter sets add angular rotations of
$\\{22.5\degree,45\degree\\}$ with rotation frequencies of $\\{1.5,2.5\\}$ per
vertical FoV, which for our setup with a distance $z_{target}=0.20$ was 0.218
m.

###  VI-B Lab Experiments

The setup for the laboratory experiments is shown in Figure 6 (right). We
utilize a UR5e with an Intel RealSense D405 camera mounted to the wrist
flange. The experimental setup is designed to mimic a trellis-based
environment similar to a real orchard. For each experiment, we secured a
branch to the trellis and ran the system to extract a 3D model. To obtain
“ground truth” data for the branch, we attached a probe to the robot and
manually sampled the location of the surface of the primary and secondary
branches. We also used digital calipers to record the radius at each sampled
point.

The lab experiments follow the same procedure as the simulated ones. We ran
our experiments for 4 different branches with 4 parameter sets: the baseline
controller with no rotation at $\\{0.02,0.05,0.10\\}$ m/s, as well as the
rotating controller with a lookat angle of $22.5\degree$ and 1.5 rotations per
vertical FoV at $0.05$ m/s.

##  VII Results and Discussion

###  VII-A Simulated Experiments

| Simulated Experiments | Real Experiments
---|---|---
Lookat Angle ° | 0 | 22.5 | 45 | 22.5 | 45 | 22.5 | 0 | 0 | 0
Rotation Freq | 0 | 1.5 | 1.5 | 2.5 | 2.5 | 1.5 | 0 | 0 | 0
Speed | N/A | N/A | N/A | N/A | N/A | 0.05 | 0.02 | 0.05 | 0.10
PB Resid. (mm) | 3.5 (4.0) | 4.2 (4.9) | 4.5 (8.9) | 3.8 (3.7) | 4.8 (7.5) | 4.3 (4.6) | 11.9 (15.9) | 6.5 (13.1) | 4.7 (7.5)
PB Rad. Error % | 6 (10) | 5 (9) | 6 (9) | 5 (7) | 6 (9) | 18 (15) | 16 (31) | 16 (18) | 19 (18)
SB $\angle$ Error ° | 13 (8) | 16 (11) | 15 (9) | 15 (10) | 17 (10) | 13 (8) | 17 (13) | 11 (5) | 15 (6)
SB Detection % | 91 | 95 | 91 | 95 | 91 | 76 | 76 | 73 | 76
SB Len Error % | 5 (91) | 11 (84) | -4 (32) | 4 (75) | 4 (51) | N/A | N/A | N/A | N/A
SB Rad. Error % | -13 (16) | -13 (15) | -12 (16) | -12 (15) | -11 (13) | N/A | N/A | N/A | N/A
TABLE I: Results of the simulated and real experiments for each parameter set
(PB = primary branch, SB = secondary branch, Resid = residual, Rad. = radius).
Standard deviation values are given in parentheses.

Table I shows a summary of the experiments (simulated and real), and Figure 7
shows some sample reconstructions. First, we focus on the results of the
baseline controller with no rotation before analyzing the results of adding
rotation. As a whole, the reconstruction accuracy of the framework was
remarkably accurate, with the average primary branch residual (i.e. the
distance from the constructed model at a given $z$-value to the corresponding
ground truth model) of just 3.5mm and an average radius estimate error of
6.4%. These values were generally positively skewed, characterized by
infrequent but large outliers.

Meanwhile, considering the secondary branches, we were able to positively
detect almost 91% of all secondary branches, even without rotating the camera
to get new views. Typically, the branches that were missed were either
pointing straight towards or away from the camera. For the branches that were
detected, the average orientation error from the ground truth was 13.1
degrees, a value that is more than sufficient for pruning. Radius estimation
was also reasonably accurate with an average of -13%. One area that requires
significant improvement is length estimation; though the average error was
5.2%, the standard deviation was 90.7%. Large errors tended to have two
causes: Either a background point was selected for 3D estimation and not
filtered out, or poor 3D estimates in the secondary branch caused the cubic
Bézier curve to have an unusual shape with sharp bends. Though the length
estimation was by far the weakest part of the framework, it is also not as
essential for the pruning task as detection and estimation of orientation.

Figure 7: Examples of 3D reconstructions of the branches from the framework
with the corresponding primary branch residual values. The ground truth is
faded while the reconstructed model is shown in red.

The effects of adding rotation to the controller were mixed. First, there were
not any notable differences between the two rotation frequencies (1.5 and
2.5), indicating that 1.5 is preferred since the robot will spend less time
rotating the camera. Regarding the different rotation angles, there was a
slight increase in branch detection rate using the 22.5 degree value, which is
expected since the robot should be able to spot branches that initially point
directly toward or away from the camera. Surprisingly, this advantage
disappeared when increasing the rotation to 45 degrees. Indeed, doubling the
rotation generally reduced the accuracy metrics and also led to a few
instances where the manipulator became stuck in a singularity. The one
advantage the 45 degree controller had was in reducing the magnitude of the
length errors of the secondary branches, since the robot was able to gain a
more clear view of poorly detected secondary branches and remove sections of
the model that were not in the mask. As a whole, the metrics for the
controllers with rotation were not notably statistically different than those
of the baseline. Despite this, the results do support our initial motivation
of incorporating multiple viewpoints to discover hidden branches.

One other notable weakness in secondary branch detection was redetecting the
same branch multiple times. We did incorporate some heuristics such as branch
origin location and orientation to detect if a 3D secondary curve was the same
as a previous one, but there was still an “overdetection” rate of 32%. Despite
this, it was quite rare for the system to have completely spurious secondary
branch detections, with a rate of just 4%.

###  VII-B Real Experiments

Due to the difficulty of manually probing the entire tree, we focus on a
subset of metrics for the real experiments: The primary branch residual and
radius errors as well as the secondary branch detection and orientation
estimates. In comparison to the simulated data, there were higher primary
residuals and radius errors on real branches. The main source of the higher
residuals is from instances in which a model point was incorrectly projected
into the background and was never corrected (e.g. Figure 7). Unlike with the
simulated experiments, including rotation in the controller significantly
reduced the presence of large residuals caused by poor 3D estimates since the
system could detect such errors as inconsistent with the mask and correct
them.

For the secondary branches, compared to the simulated experiments, the
detection rate was lower at about 76% and the orientation errors had roughly
the same magnitude. The primary cause of the lower success rate was the real
branches being thinner and irregularly shaped. Another major contributor was
that the background wall was closer to the robot than the background in the
simulated environment, which caused the masks to be noisier. This led to
issues with spurious branch detections and situations where secondary branches
were poorly modeled due to failing to properly filter out points projected
into the background.

There was no significant difference between the performance of the controllers
based on speed. Somewhat surprisingly, the primary branch residuals decreased
as speed increased. This likely has to do with the point triangulation node,
which is set to acquire 8 images over a minimum distance of 7.5 mm. However,
at a speed of 10 cm/s with a frame rate of 30 images per second, the distance
covered would be 26.7 mm. It is likely that the images being further apart
contributes to more reliable 3D point estimates (or makes bad 3D point
estimates easier to detect).

###  VII-C Runtime

We used a computer with an i7-11800H 8-core processor and an NVIDIA GeForce
RTX 3060 Laptop GPU, running ROS2 Humble on Ubuntu 22.04. With this setup and
320x240 image inputs, the average runtimes to process one input for each node
were:

  * •

Segmentation node: 0.037 s (26.9 Hz)

  * •

3D curve modeling node: 0.146 s (6.8 Hz). This runtime includes the
synchronous call to the point tracking and triangulation node, which took
0.068 s (14.6 Hz)

  * •

Primary branch following node: 0.00011 s (9300 Hz)

Even with unoptimized Python code, it is clear that the pipeline is suitable
for real-time use, with the main bottleneck (the 3D curve modeling node) still
running nearly seven times per second. The limiting speed of the system (not
accounting for practical effects such as motion blur or vibration) would
correspond to a speed such that the end effector would move over 1 vertical
FoV during the curve processing time, which for our setup would be about 1.5
m/s (much faster than we would desire to run the system).

##  VIII Conclusion

In this paper, we introduced a framework for scanning along a primary branch
while constructing a model of the branch, motivated by the need to quickly and
efficiently locate prunable secondary branches and the corresponding 3D
estimates of their cut points. By combining traditional 2D skeletonization
methods with deep neural networks for segmentation and 3D point estimation, we
developed a system that can create accurate branch models while running at
practical speeds. This work represents a useful framework for pruning locally,
on-the-fly without the need for a global reconstuction of the plant.

There are many notable improvements that could be made to the system. The
first is the branch modeling, since the current method of projecting the
previous tree model into the current mask and deleting parts of the model that
repeatedly fail to project inside the mask is overly simplistic. A more
rigorous approach would be to use a probabilistic particle model, such as
those used in 3D SLAM algorithms, to keep track of each branch point estimate.
Another aspect that could be improved is the rotation to discover new
branches, which currently occurs at a fixed frequency. To reduce the amount of
time spent rotating the camera, it would be useful for the system to perform
rotation only when it identifies the possible presence of a new branch but
needs a better view of it. Finally, there is currently no consideration for
avoiding detected branches or kinematic singularities. For a practical
implementation, it will be essential for the system to be able to plan
kinematically feasible paths to avoid hitting front-facing branches.

## References

  * [1] A. Zahid, M. S. Mahmud, L. He, P. Heinemann, D. Choi, and J. Schupp, “Technological advancements towards developing a robotic pruner for apple trees: A review,” _Computers and Electronics in Agriculture_ , vol. 189, p. 106383, Oct. 2021.
  * [2] A. You, N. Parayil, J. G. Krishna, U. Bhattarai, R. Sapkota, D. Ahmed, M. Whiting, M. Karkee, C. M. Grimm, and J. R. Davidson, “Semiautonomous precision pruning of upright fruiting offshoot orchard systems: An integrated approach,” _IEEE Robotics & Automation Magazine_, pp. 2–11, 2023.
  * [3] A. You, C. Grimm, A. Silwal, and J. R. Davidson, “Semantics-guided skeletonization of upright fruiting offshoot trees for robotic pruning,” _Computers and Electronics in Agriculture_ , vol. 192, p. 106622, 2022.
  * [4] M. Karkee and B. Adhikari, “A method for three-dimensional reconstruction of apple trees for automated pruning,” _Transactions of the ASABE_ , vol. 58, no. 3, pp. 565–574, 2015.
  * [5] S. A. Akbar, N. M. Elfiky, and A. Kak, “A novel framework for modeling dormant apple trees using single depth image for robotic pruning application,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 5136–5142.
  * [6] A. Bucksch and R. Lindenbergh, “Campino—a skeletonization method for point cloud processing,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 63, no. 1, pp. 115–127, 2008.
  * [7] T. Botterill, S. Paulin, R. Green, S. Williams, J. Lin, V. Saxton, S. Mills, X. Chen, and S. Corbett-Davies, “A robot system for pruning grape vines,” _Journal of Field Robotics_ , vol. 34, no. 6, pp. 1100–1122, 2017.
  * [8] Y. Fu, Y. Xia, H. Zhang, M. Fu, Y. Wang, W. Fu, and C. Shen, “Skeleton extraction and pruning point identification of jujube tree for dormant pruning using space colonization algorithm,” _Frontiers in Plant Science_ , vol. 13, p. 1103794, 2023.
  * [9] H. Cuevas-Velasquez, A.-J. Gallego, and R. B. Fisher, “Segmentation and 3d reconstruction of rose plants from stereoscopic images,” _Computers and electronics in agriculture_ , vol. 171, p. 105296, 2020.
  * [10] J. Gené-Mola, R. Sanz-Cortiella, J. R. Rosell-Polo, J.-R. Morros, J. Ruiz-Hidalgo, V. Vilaplana, and E. Gregorio, “Fruit detection and 3d location using instance segmentation neural networks and structure-from-motion photogrammetry,” _Computers and Electronics in Agriculture_ , vol. 169, p. 105165, 2020.
  * [11] X. Liu, S. W. Chen, S. Aditya, N. Sivakumar, S. Dcunha, C. Qu, C. J. Taylor, J. Das, and V. Kumar, “Robust fruit counting: Combining deep learning, tracking, and structure from motion,” in _2018 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2018, pp. 1045–1052.
  * [12] N. Parayil, A. You, C. Grimm, and J. Davidson, “Follow the leader: a path generator and controller for precision tree scanning with a robotic manipulator,” in _Precision agriculture’23_. Wageningen Academic, 2023, pp. 167–174.
  * [13] D. Rapado-Rincón, E. J. van Henten, and G. Kootstra, “Development and evaluation of automated localisation and reconstruction of all fruits on tomato plants in a greenhouse based on multi-view perception and 3d multi-object tracking,” _Biosystems Engineering_ , vol. 231, pp. 78–91, 2023\.
  * [14] R. Barth, J. Hemming, and E. J. van Henten, “Design of an eye-in-hand sensing and servo control framework for harvesting robotics in dense vegetation,” _Biosystems Engineering_ , vol. 146, pp. 71–84, 2016.
  * [15] C. Lehnert, D. Tsai, A. Eriksson, and C. McCool, “3d move to see: Multi-perspective visual servoing towards the next best view within unstructured and occluded environments,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 3890–3897.
  * [16] A. You, C. Grimm, and J. R. Davidson, “Optical flow-based branch segmentation for complex orchard environments,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 9180–9186.
  * [17] H. Blum, “A transformation for extracting new descriptions of shape,” _Models for the perception of speech and visual form_ , pp. 362–380, 1967\.
  * [18] A. W. Harley, Z. Fang, and K. Fragkiadaki, “Particle video revisited: Tracking through occlusions using point trajectories,” in _ECCV_ , 2022.