# HashPoint: Accelerated Point Searching and Sampling for Neural Rendering

Jiahao Ma1,2, Miaomiao Liu1, David Ahmedt-Aristizabal2, Chuong Nguyen2
Australian National University1, CSIRO Data612
{jiahao.ma<EMAIL_ADDRESS>
{jiahao.ma, david.ahmedtaristizabal<EMAIL_ADDRESS>
###### Abstract

In this paper, we address the problem of efficient point searching and
sampling for volume neural rendering. Within this realm, two typical
approaches are employed: rasterization and ray tracing. The rasterization-
based methods enable real-time rendering at the cost of increased memory and
lower fidelity. In contrast, the ray-tracing-based methods yield superior
quality but demand longer rendering time. We solve this problem by our
HashPoint method combining these two strategies, leveraging rasterization for
efficient point searching and sampling, and ray marching for rendering. Our
method optimizes point searching by rasterizing points within the camera’s
view, organizing them in a hash table, and facilitating rapid searches.
Notably, we accelerate the rendering process by adaptive sampling on the
primary surface encountered by the ray. Our approach yields substantial speed-
up for a range of state-of-the-art ray-tracing-based methods, maintaining
equivalent or superior accuracy across synthetic and real test datasets. The
code will be available at https://jiahao-ma.github.io/hashpoint/.

(a) HashPoint combines rasterization and ray-tracking approaches to optimize
point searching and adaptive sampling on primary surfaces. (A) A ray
intersects at six surfaces a$\sim$f. (B) Zoomed-in surfaces to show sample
points in green and red, with the nearby retrieved point cloud in blue. A
shift from green to red represents increasing blend weight of the sample
points. (C) The graph illustrates opacity and rendering weight changes along a
camera ray. The first peak, labeled ‘a’, indicates the primary surface where
the ray first hit. Higher rendering weights mean more significant sample
points. Notably, the primary surface often dominates the rendering process.
(D) Sampling across multiple surfaces versus our HashPoint which accelerates
the process via sampling solely on the primary surface.

##  1 Introduction

Photo-realistic rendering, a significant challenge in computer vision, has
seen notable advancements with NeRF [36] and its extensions [4, 5, 33, 65].
Approaches in [36, 65, 61, 36, 14] rely on global MLPs to reconstruct radiance
fields across the entire space through ray marching. However, this method
results in slow per-scene neural network fitting and extensive, often
unnecessary, sampling of vast empty space, leading to prohibitive
reconstruction and rendering times. To overcome this challenge, point clouds
are introduced as a straightforward representation of surfaces in space,
forming point-based neural radiance fields [10, 62, 48, 25, 2, 46, 41, 23, 66,
11]. Point clouds approximate scene geometry and encode appearance features,
expediting the rendering process by sampling and aggregating features near
multiple surfaces represented by the point cloud. Existing methods in this
field typically adopt two rendering strategies: rasterization and ray tracing.

In the rasterization framework [2, 46, 24, 8, 23, 66, 25, 32, 64, 17], points
are initially projected onto the image plane, where pixel values are
determined by the color of the nearest point using z-buffer mechanism. While
this approach enables real-time rendering, it often exhibits visible holes
stemming from the density of the point cloud. To address this limitation, some
methods [2, 46, 23, 66, 13] leverage networks such as U-Net [47] for hole
filling by employing feature downsampling and upsampling. However, this
approach struggles to generate high-fidelity details. Alternative strategies
are to assign each point as a 3D shape such as an oriented disk [24, 32, 42,
64, 17] or a sphere [27]. Nevertheless, determining the optimal size for these
shapes is a challenging task. Small shapes can lead to gaps in the rendering,
while larger ones may introduce artifacts. Additionally, significant memory is
often required for storage.

Ray-tracing-based point cloud rendering [10, 62, 48, 20, 70, 21, 41], directly
casts rays onto a point cloud and interpolates nearby points’ features along
each camera ray, addressing issues like holes found in rasterization-based
methods and enabling high-fidelity novel view synthesis. However, real-time
rendering poses challenges due to the complex point cloud search process and
excessive sampling of multiple surfaces. To enhance point cloud searching
efficiency, methods in [10, 62, 48, 41] employ accelerated data structure like
Uniform grid [18], K-d tree [16, 19, 68], Octree [34, 49] or bounding volume
hierarchies (BVH) [56]. However, not all surface features are equally
important for rendering, as illustrated in Figure 1(a). Typically, only
surfaces closest to the camera significantly contribute to rendering, making
subsequent sampling redundant. Other methods attempt to collect the $K$
nearest points for each ray to predict colors [41, 10], struggling with sparse
point density for extracting primary surface features.

In this work, we introduce HashPoint, an optimized point cloud searching
approach designed to address the hole issue and accelerate the search process
by adaptive sampling on primary surfaces. The core of our method involves
transforming the 3D search space into a 2D image plane for hash table lookup.
Unlike traditional rasterization methods using the z-buffer to retain only the
nearest points, we project all points onto the image plane and preserve all
points within each pixel. These points are then stored in a hash table. This
accelerated structure enables a swift point location near the camera ray
through the hash table lookup. What distinguishes our approach is the adaptive
searching range, determined by the distance between points and the viewpoint,
rather than relying on a fixed radius or $K$ nearest points. After identifying
the point clouds near the camera ray, these points are projected onto the ray,
with each projected point serving as a potential sample point candidate. The
selection of sample points is crucial, as it directly influences the feature
aggregation quality on the primary surface. We calculate the importance of
each sample point candidate based on the distance between the point cloud and
the candidate itself. Drawing inspiration from volume rendering techniques, we
retain high-importance sample point candidates, those closer to the viewpoint,
becoming the definitive sample points. Consequently, the number of sample
points for each ray varies adaptively, ranging from 0 to $n$, ensuring a
dynamic sampling process. In summary, our main contributions consist of two
key techniques:

  * •

We introduce _Hashed-Point Searching_ as a novel technique that accelerates
the ray-tracing approach by optimizing point searching for improved
efficiency.

  * •

We also propose a novel technique called _Adaptive Primary Surface Sampling_
to adaptively sample on the first encountered surface by the ray determined by
the distance between points and the viewpoint.

  * •

We validate our approach on various benchmark datasets (synthetic, real,
indoor, and outdoor), demonstrating its significant potential to accelerate
the rendering process by a large margin with similar or better accuracy.

##  2 Related work

Point cloud rasterization. Rasterization is a widely used technique for
rendering point clouds. The basic concept involves projecting each point onto
the image plane while ensuring that closer points occlude those that are
farther away via the z-buffer mechanism. A significant challenge in point-
cloud rasterization is the unwanted occurrence of gaps or holes in the
rendered output. Classical methods such as visibility splatting [42, 71]
address these gaps by substituting points with oriented disks. However,
determining the optimal size and shape of these disks is complex, and they may
not always completely cover the visible gaps in the image. In contrast, recent
methods such as differentiable splatting, employed by [24, 64, 17, 35], have
made a notable improvement in rendering quality by fine-tuning the shape of
these disks through optimization. Recent advancements in the field involve the
integration of rasterization with neural networks. NPBG [2] and NPBG++ [46]
employ U-Net [47] refinement to learn the rasterizing features, minimizing the
disparity between the rendered images and ground truth. These approaches
address the issue of holes by leveraging both feature downsampling and
upsampling. Huang _et al._ [23] introduce radiance mapping as a means to
combat spatial frequency collapse. However, this technique still encounters
difficulties in generating high-fidelity details in coarse regions. As a
solution, FreqPCR [66] introduces an adaptive frequency modulation module
designed to capture the local texture frequency information.

Ray tracing for point cloud. This presents an alternative approach for
rendering point clouds. Early works [1, 30, 55] developed iterative strategies
to determine ray intersections with surfaces approximated from point clouds.
Recently, methods such as NPLF [41] focus on embedding features at each point
and aggregating them during a query, while Pointersect [10] directly
determines ray intersections with the inherent surface. PAPR [67] employs
point cloud positions to capture scene geometry, even when the initial
geometry substantially differs from the target geometry. However, this method
mainly focuses on the nearest points around the ray, potentially missing the
features of the primary surface, particularly when dealing with sparse point
clouds. On the other hand, Point-SLAM [48] and Point-NeRF [62] uniformly
sample across multiple surfaces to produce high-quality renderings. Despite
avoiding unnecessary sampling of vast empty space, such processes of extensive
feature aggregation and MLP prediction on subsequent surfaces remain time-
consuming. Thus, there is a need for selective sampling on correct surfaces to
improve both accuracy and efficiency in point cloud rendering.

Efficiency for neural radiance field. Recent works [28, 59, 38, 60] improve
the memory and optimization efficiency via hash encoding. Some other works
[57, 39, 43, 52, 26] are designed to improve the sampling efficiency of NeRF
while maintaining a compact memory footprint. DONeRF [39] improves sampling
efficiency with a depth-trained oracle network. TermiNeRF [43] uses a density-
based sampling network from a pre-trained NeRF. AutoInt [52] predicts segment
lengths along rays for sampling. AdaNeRF [26] employs a dual-network
architecture with incremental sparsity for fewer samples of higher quality. In
general, these methods utilize pre-trained modules and complex optimization
for sample point distribution along rays. In addition to neural network-
assisted sampling, some other approaches [44, 12, 15, 29] adopt multi-view
stereo (MVS) technique [51, 63] or depth sensor to obtain depth
information[48], subsequently sampling only on the surfaces of scenes, closely
resembling ray-tracing based point cloud rendering. Yet, they rely on dense
depth map input and tend to encounter inefficiencies from oversampling on
multiple surfaces during rendering. This paper presents our pioneer concept of
sampling on the primary surface, further optimizing the use of the explicit
geometric representation.

##  3 Method

Figure 1: Efficient point clouds searching for ray tracing. (A) Top row shows
traditional point cloud search strategies: Uniform Grid, K-D tree, and Octree
visualized in 2D for clarity. (A) Bottom row shows _Hashed-Point Searching_ ,
our method of rasterizing the point cloud onto an image plane and then
reorganizing the point cloud list to optimize queries, resulting in the
construction of a hash table. Each key (as pixel index) in the table maps to
the start index in the point list and the count of points within that pixel
with $\mathcal{O}(1)$ complexity. (B) The final selection depicts the search
mechanism: using a magnified search kernel, the neighbor points of a target
ray (black arrow) are swiftly identified through the hash table and assessed
based on their distance to the ray.

Following preliminaries in Section 3.1, we detail our HashPoint method with
our point searching technique in Section 3.2, and our surface sampling
approach in Section 3.3, and then integrate these with existing methods in
Section 3.4. Comparative analysis is explained in Section 3.5.

###  3.1 Preliminaries of neural rendering

Point cloud rendering based on ray tracing typically employs two strategies to
ascertain the color of each ray:

Image-based rendering. Given a point cloud denoted as $\mathcal{P}=$
$\left\\{\left(x^{pc}_{i},c^{pc}_{i},f_{i}\right)\right\\}_{i=1\ldots n}$,
where $x^{pc}_{i}\in\mathbb{R}^{3}$ represents position,
$c^{pc}_{i}\in\mathbb{R}^{3}$ is RGB color and $f_{i}$ depicts high-
dimensional appearance feature. Several methods [10, 41] predict the blending
weight $w_{i}(\mathbf{r},\mathcal{P})$ for camera ray $\mathbf{r}$ utilizing
the appearance feature, while others, like [41, 67], use the geometric
relationship with the ray. The final color of a camera ray $c(\mathbf{r})$ can
be computed by K nearest points, represented as K-NP in Figure 3A:

$c(\mathbf{r})=\sum_{i=1}^{K}w_{i}(\mathbf{r},\mathcal{P})c^{pc}_{i}.\vspace{-4pt}$
(1)

Although methods [10, 41, 67] exhibit some variations, fundamentally, they all
harness the relationship between rays and proximate points to determine color.

Volume rendering. Unlike predicting colors from neighboring points of a ray,
some methods [62, 48] uniformly sample on surfaces, and aggregate point
features $f_{i}$ to sampled points, as shown in Figure 3B. These methods
subsequently employ physically-based volume rendering to compute the color of
individual pixels. Specifically, the radiance of a pixel is derived by
traversing a ray through it, sampling N sample points at
$\left\\{x^{sp}_{j}\mid j=1,\ldots,N\right\\}$ along the ray, and accumulating
radiance using volume density, as:

$\displaystyle c$
$\displaystyle=\sum_{j=1}^{N}\tau_{j}\left(1-\exp\left(-\sigma_{j}\Delta_{t}\right)\right)c^{sp}_{j},$
(2) $\displaystyle\tau_{j}$
$\displaystyle=\exp(-\sum_{k=1}^{j-1}\sigma_{k}\Delta_{k}).$

Here, $\tau$ represents volume transmittance; $\sigma_{j}$ and $c^{sp}_{j}$
correspond to the density and color for each sample point $j$ at $x^{sp}_{j}$.
$\Delta_{t}$ is the distance between adjacent sample points.

###  3.2 Hashed-point searching

Accelerated structure construction. In contrast to traditional approaches that
perform point cloud searching in 3D space, we innovate by shifting the search
from 3D space to a 2D image plane for hash table lookup. As illustrated in
Figure 1, our approach diverges from classic rasterization methods that employ
the z-buffer to retain the nearest points for each pixel. Instead, we project
all points onto the image plane and preserve all points within each pixel,
storing them in a hash table for efficient retrieval. Our HashPoint approach
to reorganized point cloud is inspired by the Morton code [37] (often referred
to as Z-order), which provides a linear ordering of multi-dimensional data for
efficient retrieval and storage. Specifically, we arrange the points falling
on the same pixel in a Z-order manner to ensure they are stored as closely as
possible in sequence. Therefore, we adjust the order of points in the point
cloud list. The positional adjustment of each point is achieved through atomic
operations in CUDA [40], with a time complexity of only $\mathcal{O}(1)$ for
high efficiency. Following these adjustments, we use a hash table to store the
position of each point in the point list. In the hash table, the key is the
index of the pixel, and the value includes the position of the first point
stored for the current pixel in the point list, along with the number of
points that fall on that pixel.

Searching. Inspired by [4, 22, 6] each pixel emits a cone for point cloud
searching and feature aggregation, which is more in line with the principles
of imaging. The use of a fixed searching radius may include features that do
not belong to the pixel, introducing noise. For a detailed comparison, please
refer to our Supplementary. In this work, we propose an adaptive searching
radius that is proportional to the distance of the sampling point relative to
the ray origin.

Following [22], we represent the pixel as a circle on the image plane, as an
approximation to the area of the pixel. As shown in Figure 2A, the radius of
the disc can be calculated by $\dot{r}=\sqrt{\Delta x\cdot\Delta y/\pi}$,
where $\Delta x$ and $\Delta_{y}$ are the width and height of the pixel in
world coordinates, derived from the calibrated camera parameters. For each
pixel, a cone emits from the ray origin $\mathbf{o}$ (optical center of the
camera) along the ray direction $\mathbf{d}$, passing through the pixel center
$\mathbf{p_{o}}$. Due to the sparsity of the point cloud, there might be no
points falling within the ray cone, causing holes in the rendering. To
mitigate this, we modify the ray cone for broader coverage, shifting from a
disc with radius $\dot{r}$ for one pixel to a larger disc of radius $\ddot{r}$
for the searching kernel. We deduce the magnitude of the searching kernel $s$
by $s=2\cdot\left\lceil\frac{\ddot{r}}{\dot{r}}\right\rceil+1$. Any sample
point that lies on the ray can be derived by
$\mathbf{x^{sp}_{j}}=\mathbf{o}+t\mathbf{d}$, and the adaptive radius $r$ of
sample point can be computed by,

$\displaystyle r=\frac{\|\mathbf{x^{sp}_{j}}-\mathbf{o}\|_{2}\cdot
f\ddot{r}}{\|\mathbf{p_{o}}-\mathbf{o}\|_{2}\cdot\sqrt{\left(\sqrt{\|\mathbf{p_{o}}-\mathbf{o}\|_{2}^{2}-f^{2}}-\ddot{r}\right)^{2}+f^{2}}.}$
(3)

Here, $f$ is the focal length. For derivation, please refer to the
Supplementary. During the rendering process, we sample between $t_{near}$ and
$t_{far}$. As $t$ increases, the radius of the sample point also increases. As
the minimum radius, $r_{min}$, is at the $t_{near}$ position and matches with
our input hyperparameter radius, we modify $s$ accordingly. In the target
ray’s search kernel, we traverse each pixel and assess the number of
associated points per pixel to determine if the search is needed. If required,
we look up the point list for further judgment. The query complexity remains
at $\mathcal{O}(1)$.

Figure 2: Illustration of _Adaptive Primary Surface Sampling_. (A) The diagram
depicts the generation of the searching kernel on the image plane. (B) We
project adjacent points to the ray as sample point candidates (red crosses).
Each candidate’s importance, determined by its distance distribution to points
within its radius, influences its preservation for final feature aggregation.

###  3.3 Adaptive primary surface sampling

Another core of our proposed method lies in the adaptive aggregation of
features from the primary surface, guided by the distribution of the nearby
point cloud. As shown in Figure 2B, neighboring points are projected onto the
camera ray, forming a set of sample point candidates denoted as $x^{sp}_{j}$
where $j$ represents the index of the sample point. The geometric distribution
of these sample points is determined by calculating the average distance
between each candidate and its K neighboring points. The average distance
$d_{j}$ can be computed by

$\displaystyle
d_{j}=\frac{1}{K}\sum_{i=1}^{K}\left\|x_{j}^{sp}-x_{i}^{pc}\right\|_{2}.$ (4)

While this distance between the sample point and the surface presented by the
point cloud adheres to the principles of the Unsigned Distance Function (UDF)
[31], it deviates from a strict UDF definition. Therefore, we term it as
“pseudo-UDF”. Drawing inspiration from volume rendering [36] and its extension
[58, 31], we adhere to two rules:

  1. 1.

Unbiased intersection. A reduced pseudo-UDF value indicates that the candidate
is proximal to the surface, thus deserving a higher weighting.

  2. 2.

Occlusion-aware. When candidates exhibit identical pseudo-UDF values, the one
positioned closer to the viewport is assigned a greater weight to the final
color output.

To follow the first principle, the distance $d_{j}$ is transformed into a
confidence $a_{j}$ by
$\alpha_{j}=\gamma\exp\left(-\frac{d_{j}^{2}}{\beta^{2}}\right),$ where
$\beta$ is the hyperparameter which depends on the density of the point cloud
and $\gamma$ is to control the range of sampling. The proximity of the sample
point to the point cloud, lesser $d_{j}$, necessitates a reduced $\beta$ to
ensure distinct differentiation in the confidence $a_{j}$ cross sample points.
To make the distribution of sample point occlusion aware, based on volume
rendering, we define the weight function by

$w_{j}=\alpha_{j}\prod_{k=1}^{j-1}\left(1-\alpha_{k}\right).\vspace{-4pt}$ (5)

We retain the sample point candidates with $w_{j}$ larger than zero, which are
located near the primary surface.

###  3.4 Integration with existing methods

The proposed method is adaptable to most ray-tracing point cloud rendering
techniques. After sampling on the primary surface and identifying surface
point cloud, we can use MLP, as in [48, 62], to predict density and color for
sample points. The color of the pixel is then rendered by volume rendering.
Additionally, we can integrate approaches from [10, 41, 67] to estimate the
ray’s color based on point features. Please refer to Section 4 for more
comparison.

###  3.5 Comparative analysis

Our hashed-point searching vs. traditional point cloud searching. Efficient
retrieving neighboring points for a camera ray is non-trivial. A
straightforward method is to compare all rays with all points, brute force
searching, leading to high computation costs. Common approaches employ space
partitioning to speed up the search. Below, taking an example with $n$ points
and $m$ rays with an average of $q$ points falling in the radius $\delta$ of
each ray, we introduce the search processes and the corresponding complexity
of these algorithms.

  * •

K-d tree [7]: a space-partitioning data structure for organizing points in a
k-dimensional space. To locate points near a ray using this structure, two
common strategies exist: (1) Sample uniformly along the ray followed by radius
searches at each sample point, and (2) leverage sub-tree bounding boxes in the
K-d tree and apply axis-aligned bounding box (AABB [50]) for intersections,
then compare points in intersecting sub-trees. Building the K-d tree needs to
iterate $n$ points with the complexity of $\mathcal{O}(n)$, and in the case of
searching points for $m$ rays, the complexity of finding all intersecting sub-
trees and retrieving $q$ points within the cylinder of rays is
$\mathcal{O}(mlog(n)+mq)$.

  * •

Octree [34]: a tree data structure used to partition a three-dimensional space
by recursively subdividing it into eight octants. This strategy for finding
points near a ray is similar to the K-d tree mentioned above, and the
complexity is the same - constructing an Octree requires $\mathcal{O}(n)$,
while searching requires $\mathcal{O}(mlog(n)+mq)$.

  * •

Uniform grid [18]: a data structure that divides space into regular grids and
allocates data to each grid cell. Building this data structure requires
traversing all points with complexity $\mathcal{O}(n)$. For searches, using
algorithms like 3DDA [3] or AABB to find intersecting voxel with rays, the
complexity is $\mathcal{O}(mg+mq)$, where $g$ is the number of grid cells per
dimension.

  * •

Hashed-point searching: In terms of design, prior methods involve the ray
actively seeking point cloud in 3D space. In contrast, our method allows the
point cloud to seek rays. To accelerate queries, we construct an acceleration
structure in the form of a hash table by iterating through all points,
followed by searching via hash table lookup. The complexity of this process is
$\mathcal{O}(n+mq)$.

Figure 3: Comparison of different point cloud selection strategies. (A) K-NP
Extract: extracting K nearest point features per ray. (B) Depth Map Sampling:
sampling based on dense depth maps. (C) Multi-Surface Sampling: uniform
sampling over multiple surfaces. (D) Our method: adaptive sampling on the
primary surface.

Our adaptive primary surface sampling vs. existing point cloud selection
strategies. The selection of the number and location of points significantly
determines the quality and efficiency of rendering. As shown in Figure 3, we
compare various methods of point cloud selection as follows:

  * •

“K-NP Extract” [67, 10, 41] refers to extracting the nearest K point features
around a ray to predict its color. As shown in Figure 5, due to the density of
point cloud, the K nearest points often don’t necessarily fall on the primary
surface, leading to feature noise.

  * •

“Depth Map Sampling” [44, 48] refers to using sensor depth or predicted depth
via MVS [51, 63] to assist sampling. Although this method is highly efficient,
it relies on dense depth maps and also faces the issue of redundant sampling
on multiple surfaces (especially in the case of relying on nearby views with
depth input).

  * •

“Multi-Surface Sampling” [62] involves finding all surfaces intersecting with
the ray and then uniformly sampling on them. The method provides the best
result at the high cost of computation.

  * •

”Adaptive Primary Surface Sampling” gives a good balance between precision and
efficiency by sampling on the primary surface, as explained in Sections 3.3,
4.2 and 4.3.

Ours vs. 3D Gaussian splatting [24]. While our method shares similarities with
3D Gaussian splatting in utilizing rasterization for point searching, a key
distinction lies in our rendering approach. While Gaussians have shape, they
are often small, requiring more points for ray rendering. In contrast, our
method interpolates ray color from nearby points, requiring fewer points. For
instance, in experiments with the same Lego scene (NeRF-Synthesis Dataset),
our method uses only 35Mb of storage, compared to Gaussian splatting’s 200Mb.
The proposed method takes advantage of both rasterization and ray tracing,
complementing each other. The combination of the two methods further enhances
the approach to promising neural rendering.

##  4 Experiments

###  4.1 Baselines and integration

This section outlines four key baselines in ray-tracing point cloud rendering,
focusing on searching and sampling, as detailed in Table 1. We compare the
corresponding components with the proposed enhancement as follows:

  * •

Point-NeRF [62]: Searches via uniform grid (UG) and then samples across
multiple surface (MS). We replace the UG and MS with our HashPoint (HS) and
primary surface sampling (PS) and benchmark as shown in Table 2. PS alone
impedes the point optimization due to limited gradient propagation. Thus, MS
is initially used for 10$K$ iterations for geometry optimization, then
switched to PS for boosting efficiency. The transition is controlled by
adjusting the parameter $\gamma$. For more details, please refer to the
supplementary material.

  * •

Point-SLAM [48]: Employs single-surface (SS) sampling, different from Point-
NeRF’s MS approach. While Point-SLAM’s depth-guided sampling is efficient, it
relies on dense depth input, not always available during rendering. Our
method’s performance as shown in Table 3 is compared with both of Point-SLAM’s
sampling strategies: depth-guided and uniform multi-surface.

  * •

NPLF [41]: Starts with downsampling points, searches nearby points using brute
force (BF), and renders on K nearest points (K-NP). Our method maintains high
fidelity by selecting K (K = 8) nearest points from the primary surface
without downsampling as shown in Table 4.

  * •

Pointersect [10]: Searches using a UG, retaining K (K = 40) nearest points for
rendering. We improve this by selecting K (K = 6) nearest points from primary
surfaces, gaining higher speed and quality as shown in Table 5.

We evaluate our search and selection modules, along with overall performance,
in Sections 4.2 and 4.3. The evaluation metrics include Peak Signal-to-Noise
Ratio (PSNR), Structural Similarity Index Measure (SSIM), Learned Perceptual
Image Patch Similarity (LPIPS), and frames per second (FPS). Our method,
integrated with these baselines, is tested on Synthetic-NeRF [36], Waymo [54],
Replica [53], and ShapeNet [9] datasets. We adopt their training approach,
with a key difference: in Point-NeRF, we refine the geometry, unlike other
experiment baselines. This refinement impacts accuracy; for more details, see
Section 7.

Method | Searching | Selection
---|---|---
Point-NeRF [62] | Uniform grid | Multiple surfaces
NPLF [41] | Brute force |  K nearest points
Pointersect [10] | Uniform grid |  K nearest points
Point-SLAM [48] | Depth map sampling | Single surface

Table 1: Comparison of key components in various baselines.

| Radiance-based | Point-based (Rasterization) | Point-based (Ray tracing)
---|---|---|---
| NeRF [36] | NPBG [2] | NPBG++ [46] | Huang et al. [23] | FreqPCR [66] | Point-NeRF [62] | Point-NeRF + Ours
PSNR $\uparrow$ | 31.01 | 24.56 | 28.12 | 28.96 | 31.24 | 33.31 | 33.22
SSIM $\uparrow$ | 0.947 | 0.923 | 0.928 | 0.932 | 0.950 | 0.978 | 0.961
LPIPS$\downarrow$ | 0.081 | 0.109 | 0.076 | 0.061 | 0.049 | 0.049 | 0.055
FPS $\uparrow$ | 0.05 | 33.64 | 35.21 | 39.67 | 39.56 | 0.12 | 9.60($\times$80 speed up)

Table 2: Comparison of our method integrated with Point-NERF with a radiance-
based model [36], rasterization-based models [2, 46, 23, 66] and a ray-
tracing-based model [62] on the Synthetic-NeRF dataset.

###  4.2 Evaluation

Evaluation on Synthetic-NeRF [36]. We incorporate the proposed method into
Point-NeRF [62] and compare with both NeRF [36] and various point-based
approaches [2, 46, 23, 66, 62]. As shown in Table 2, this integration achieves
an 80-fold speedup while ranking as the second-highest performance among
point-based methods. The results of Figure 4 qualitatively show that solely
sampling on the primary (single) surface can produce multi-surface effects,
presenting the right balance between performance and speed for ray-tracing-
based methods.

Evaluation on Replica [53]. We also compare the integration of Point-SLAM [48]
with voxel-based method [69] and the original [48] on the Replica [53] indoor
dataset, solely focusing on rendering rather than tracking and mapping. As
outlined in Table 3, our approach outperforms Point-SLAM in both PSNR and SSIM
with parity in LPIPS. In terms of speed, we analyzed Point-SLAM’s depth-guided
and uniform multi-surface sampling (US), particularly for depth-unknown
scenarios. In contrast to Point-SLAM’s fixed $\hat{n}$ point collection
($\hat{n}=5$ ), our method dynamically gathers 1 to n points based on nearby
point distributions ($\textit{n}=4$). The efficiency of our approach is
notably superior - 1.8 times faster than depth-guided sampling and 11.5 times
faster than US sampling.

Evaluation on Waymo [54]. In the evaluation of the Waymo dataset, we
substitute the BF and K-NP with our methods while comparing radiance-based
methods [36, 14] and the original [41]. Figure 5 illustrates that our method
accurately samples object surfaces, outperforming K-NP. Table 4 shows our
method’s superior accuracy over NPLF and its enhanced efficiency compared to
BF.

Evaluation on ShapeNet [9]. To evaluate on the ShapeNet dataset without per-
scene optimization, we adopt the training setting of Pointersect - training on
48 training meshes from the sketchfab [45] dataset and testing on 30 meshes in
ShapeNet. We streamline the architecture of Pointersect by reducing the input
of neighbor points (K: 40 to 6), significantly enhancing the efficiency of
inference. The results presented in Table 5 indicate that using only the K
nearest points of the primary surface produces robust outcomes and accelerates
the process.

Figure 4:  Results on the NeRF-Synthesis [36] dataset shows that our primary
surface sampling (blue points) is more efficient than Point-NeRF’s sampling
(red points) while preserving accuracy.  Figure 5:  Comparison on Waymo [54]
dataset shows how our primary surface sampling (blue points) more accurately
samples the car body than the K nearest point selection of NPLF (red points).

| Voxel-based | Point-based (Ray tracing)
---|---|---
| NICE-SLAM [69] | Point-SLAM [48] | Point-SLAM + Ours
PSNR $\uparrow$ | 24.42 | 35.17 | 35.43
SSIM $\uparrow$ | 0.809 | 0.975 | 0.983
LPIPS$\downarrow$ | 0.233 | 0.124 | 0.126
FPS $\uparrow$ | 0.43 | 0.95($Depth$) $|$ 0.15($US$) |  1.72 ($\times$1.8 $|$ $\times$11.5 speed up)

Table 3: Comparison of our method integrated with Point-SLAM [48] with NICE-
SLAM [69] and Point-SLAM on Replica dataset [53]. $Depth$ and $US$ denote
depth-guided sampling on a single surface and uniformly sampling across
multiple surfaces separately. The speed evaluates the performance of rendering
instead of mapping and tracking.

| Radiance-based | Point-based (Ray tracing)
---|---|---
| NeRF [36] | DS-NeRF[14] | NPLF [41] | NPLF + Ours
PSNR $\uparrow$ | 22.47 | 26.15 | 29.96 | 30.57
SSIM $\uparrow$ | 0.700 | 0.772 | 0.868 | 0.912
LPIPS$\downarrow$ | 0.389 | 0.310 | 0.119 | 0.105
FPS $\uparrow$ | 0.11 | 0.11 | 0.33 |  1.98 ($\times$ 6 speed up)

Table 4: Comparison of our method integrated with NPLF [41] with two radiance-
based model [36, 14], and a ray-tracing-based model NPLF on the Waymo Open
dataset.

Comparison with traditional point cloud search for ray casting. In Figure 6,
our method outperforms traditional methods (Uniform grid, K-d tree, and
Octree) in point cloud search efficiency for ray casting. All comparisons are
conducted on a CPU (Intel(R) Core(TM) i9-12900K) without GPU acceleration. The
left graph shows our method’s superior performance with increasing point
numbers, while the right graph demonstrates consistent efficiency with rising
ray counts. Notably, all methods retrieve the same number of points,
underscoring that differences in performance result from search efficiency.
Our proposed method running on a NVIDIA RTX 4090 GPU for 1 million points only
takes 4 ms to complete searching (0.5 ms) and sampling (3.5 ms) proving its
high efficiency of processing large-scale point cloud data for ray casting.

| Point-based
---|---
| Rasterization | Ray tracing
| NPBG++ [46] | Pointersect [10] | Pointersect + Ours
PSNR $\uparrow$ | 19.3 $\pm$ 4.0 | 28.0 $\pm$ 6.4 | 29.5 $\pm$ 5.6
SSIM $\uparrow$ | 0.8 $\pm$ 0.1 | 1.0 $\pm$ 0.0 | 1.0 $\pm$ 0.0
LPIPS$\downarrow$ | 0.18 $\pm$ 0.08 | 0.04 $\pm$ 0.04 | 0.02 $\pm$ 0.01
FPS $\uparrow$ | 33.40 | 1.25 | 10.12 ($\times$8 speed up)

Table 5:  Comparison on the ShapeNet [9] dataset shows that the integration of
our approach with Pointersect[10] yields improved performance over NPBG++ [46]
and Pointersect.  Figure 6:  Comparative search performance for neighbor
points searching for ray casting: ours vs. uniform grid [18], K-d tree [7] and
Octree [34]. Left: search times for various numbers of points with a fixed
number of rays. Right: search time for a fixed number of points with
increasing numbers of rays.

###  4.3 Ablation Study

Figure 7:  Ablation study NeRF-Synthesis dataset.HS and PS represent our
_Hashed-Point Searching_ and _Adaptive Primary Surface Sampling_. (A)
Combination of UG (Uniform Grid) searching with MS (Multi-Surfaces) sampling,
alongside our techniques. (B) Combination of SS (single surface sampling) with
Our PS using Depth guided sampling. (C) Combination with UG and K-NP (K
Nearest Points) selection for sampling. (D) Combination with K-NP and BF
(Brute Force) searching.

Comparison with Point-NeRF [62]. Figure 7A illustrates the comparison between
Point-NeRF’s (UG) and (MS), and our (HS) and (PS). HS notably outperforms UG
by 5 times in speed with MS, while PS leads to a dramatic speed increase of 60
to 80 times. HS and PS both increase efficiency without sacrificing accuracy,
with PS dominating due to reduced point cloud input for feature extraction and
MLP prediction.

Comparison with Point-SLAM [48]. Figure 7B presents the advantage of primary
surface sampling (PS), which adaptively collects 1 to n points on the surface
guided by point cloud distribution, over single surface sampling (SS), which
consistently gather $\hat{n}$ points. The experiment demonstrates that our
primary surface sampling not only speeds up the process but also enhances
precision.

Comparison with Pointersect [10]. Figure 7C, illustrates that selecting the K
nearest points (K-NP, K=40) from noise-initialized point clouds near a camera
ray results in higher accuracy compared to using only PS. However, our HS
method proves faster than the UG, contributing significantly to the overall
process acceleration.

Comparison with NPLF [41]. Figure 7D shows that in noisy point cloud, the K
nearest points method (K-NP) outperforms primary surface sampling (PS).
Conversely, Figure 5 illustrates that PS is superior to K-NP in correct point
clouds. Additionally, Hashpoint searching proves to be five times faster than
the brute force (BF) approach.

Geometry refinement. In the ablation study, we initialized point clouds with
Mvsnet[63] and followed Point-NeRF’s strategy for optimizing noisy points
through point pruning and growing (P&G). Geometric optimization was not
applied in Point-SLAM, Pointersect, and NPLF, resulting in lower performance.
To isolate the algorithm design factors, we test our method with and without
refinement on the base of [62]. Figure 8 shows the optimization improves
performance, as it mitigates the impact of geometric noise on feature
extraction from primary surfaces. Despite relying on geometric structures, it
is easily obtained with a brief 10-minute point optimization during training,
enabling significant speed gains without losing accuracy. We also explored the
combination of $\gamma$ and $\beta$ parameters to manage noisy input. Further
details are available in our Supplementary.

Figure 8:  Illustration of geometry refinement necessity. Left: Comparison of
methods with and without point pruning and growing (P&G) for primary surface
sampling. Without P&G, points (in red) are sampled from a noisy surface (in
purple), deviating from the true surface (in blue). Right: Performance
comparison.

##  5 Conclusion

Our method enhances point cloud rendering speed by combining ray tracing with
rasterization. Using our HashPoint technique, point clouds are efficiently
organized in a hash table through rasterization, accelerating searches. This
approach, coupled with primary surface sampling, reduces input points and
leverages geometric distribution, significantly speeding up rendering. For
instance, searching through a million points takes only 0.5 ms on a standard
GPU. In selection, our approach outperforms K nearest point methods in
accuracy and is faster than multi-surface sampling. Easily integrated with
existing methods, HashPoint advances accuracy and rendering speed, pushing the
boundaries of point cloud rendering.

## References

  * Adams et al. [2005] Bart Adams, Richard Keiser, Mark Pauly, Leonidas J Guibas, Markus Gross, and Philip Dutré.  Efficient raytracing of deforming point-sampled surfaces.  In _Computer Graphics Forum_ , pages 677–684, 2005.
  * Aliev et al. [2020] Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky.  Neural point-based graphics.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16_ , pages 696–712. Springer, 2020.
  * Amanatides et al. [1987] John Amanatides, Andrew Woo, et al.  A fast voxel traversal algorithm for ray tracing.  In _Eurographics_ , pages 3–10. Citeseer, 1987.
  * Barron et al. [2021] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5855–5864, 2021.
  * Barron et al. [2022] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5470–5479, 2022.
  * Barron et al. [2023] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Zip-nerf: Anti-aliased grid-based neural radiance fields.  _arXiv preprint arXiv:2304.06706_ , 2023.
  * Bentley [1975] Jon Louis Bentley.  Multidimensional binary search trees used for associative searching.  _Communications of the ACM_ , 18(9):509–517, 1975.
  * Cao et al. [2022] Ang Cao, Chris Rockwell, and Justin Johnson.  Fwd: Real-time novel view synthesis with forward warping and depth.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15713–15724, 2022.
  * Chang et al. [2015] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.
  * Chang et al. [2023] Jen-Hao Rick Chang, Wei-Yu Chen, Anurag Ranjan, Kwang Moo Yi, and Oncel Tuzel.  Pointersect: Neural rendering with cloud-ray intersection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8359–8369, 2023.
  * Chen et al. [2023] Zhang Chen, Zhong Li, Liangchen Song, Lele Chen, Jingyi Yu, Junsong Yuan, and Yi Xu.  Neurbf: A neural fields representation with adaptive radial basis functions.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4182–4194, 2023.
  * Dadon et al. [2023] David Dadon, Ohad Fried, and Yacov Hel-Or.  Ddnerf: Depth distribution neural radiance fields.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 755–763, 2023.
  * Dai et al. [2020] Peng Dai, Yinda Zhang, Zhuwen Li, Shuaicheng Liu, and Bing Zeng.  Neural point cloud rendering via multi-plane projection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7830–7839, 2020.
  * Deng et al. [2022] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised nerf: Fewer views and faster training for free.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12882–12891, 2022.
  * Dey et al. [2022] Arnab Dey, Yassine Ahmine, and Andrew I Comport.  Mip-nerf rgb-d: Depth assisted fast neural radiance fields.  _arXiv preprint arXiv:2205.09351_ , 2022.
  * Foley and Sugerman [2005] Tim Foley and Jeremy Sugerman.  Kd-tree acceleration structures for a gpu raytracer.  In _Proceedings of the ACM SIGGRAPH/EUROGRAPHICS conference on Graphics hardware_ , pages 15–22, 2005.
  * Gao et al. [2023] Yiming Gao, Yan-Pei Cao, and Ying Shan.  Surfelnerf: Neural surfel radiance fields for online photorealistic reconstruction of indoor scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 108–118, 2023.
  * Glassner [1989] Andrew S Glassner.  _An introduction to ray tracing_.  Morgan Kaufmann, 1989.
  * Horn et al. [2007] Daniel Reiter Horn, Jeremy Sugerman, Mike Houston, and Pat Hanrahan.  Interactive kd tree gpu raytracing.  In _Proceedings of the 2007 symposium on Interactive 3D graphics and games_ , pages 167–174, 2007.
  * Hu et al. [2023a] Tao Hu, Xiaogang Xu, Ruihang Chu, and Jiaya Jia.  Trivol: Point cloud rendering via triple volumes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20732–20741, 2023a.
  * Hu et al. [2023b] Tao Hu, Xiaogang Xu, Shu Liu, and Jiaya Jia.  Point2pix: Photo-realistic point cloud rendering via neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8349–8358, 2023b.
  * Hu et al. [2023c] Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma.  Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 19774–19783, 2023c.
  * Huang et al. [2023] Xiaoyang Huang, Yi Zhang, Bingbing Ni, Teng Li, Kai Chen, and Wenjun Zhang.  Boosting point clouds rendering via radiance mapping.  In _Proceedings of the AAAI conference on artificial intelligence_ , pages 953–961, 2023.
  * Kerbl et al. [2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  _ACM Transactions on Graphics (ToG)_ , 42(4):1–14, 2023.
  * Kopanas et al. [2021] Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis.  Point-based neural rendering with per-view optimization.  In _Computer Graphics Forum_ , pages 29–43. Wiley Online Library, 2021.
  * Kurz et al. [2022] Andreas Kurz, Thomas Neff, Zhaoyang Lv, Michael Zollhöfer, and Markus Steinberger.  Adanerf: Adaptive sampling for real-time rendering of neural radiance fields.  In _European Conference on Computer Vision_ , pages 254–270. Springer, 2022.
  * Lassner and Zollhofer [2021] Christoph Lassner and Michael Zollhofer.  Pulsar: Efficient sphere-based neural rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1440–1449, 2021.
  * Lee et al. [2023] Yongjae Lee, Li Yang, and Deliang Fan.  Mixnerf: Memory efficient nerf with feature mixed-up hash table.  _arXiv preprint arXiv:2304.12587_ , 2023.
  * Lin et al. [2022] Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou.  Efficient neural radiance fields for interactive free-viewpoint video.  In _SIGGRAPH Asia 2022 Conference Papers_ , pages 1–9, 2022.
  * Linsen et al. [2007] Lars Linsen, Karsten Müller, and Paul Rosenthal.  Splat-based ray tracing of point clouds.  2007\.
  * Long et al. [2023] Xiaoxiao Long, Cheng Lin, Lingjie Liu, Yuan Liu, Peng Wang, Christian Theobalt, Taku Komura, and Wenping Wang.  Neuraludf: Learning unsigned distance fields for multi-view reconstruction of surfaces with arbitrary topologies.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20834–20843, 2023.
  * Luiten et al. [2023] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan.  Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis.  _arXiv preprint arXiv:2308.09713_ , 2023.
  * Martin-Brualla et al. [2021] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7210–7219, 2021.
  * Meagher [1982] Donald Meagher.  Geometric modeling using octree encoding.  _Computer graphics and image processing_ , 19(2):129–147, 1982.
  * Mihajlovic et al. [2021] Marko Mihajlovic, Silvan Weder, Marc Pollefeys, and Martin R Oswald.  Deepsurfels: Learning online appearance fusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14524–14535, 2021.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021.
  * Morton [1966] Guy M Morton.  A computer oriented geodetic data base and a new technique in file sequencing.  1966\.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ , 41(4):1–15, 2022.
  * Neff et al. [2021] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H Mueller, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger.  Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks.  In _Computer Graphics Forum_ , pages 45–59. Wiley Online Library, 2021.
  * NVIDIA et al. [2020] NVIDIA, Péter Vingelmann, and Frank H.P. Fitzek.  Cuda, release: 10.2.89, 2020.
  * Ost et al. [2022] Julian Ost, Issam Laradji, Alejandro Newell, Yuval Bahat, and Felix Heide.  Neural point light fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18419–18429, 2022.
  * Pfister et al. [2000] Hanspeter Pfister, Matthias Zwicker, Jeroen Van Baar, and Markus Gross.  Surfels: Surface elements as rendering primitives.  In _Proceedings of the 27th annual conference on Computer graphics and interactive techniques_ , pages 335–342, 2000.
  * Piala and Clark [2021] Martin Piala and Ronald Clark.  Terminerf: Ray termination prediction for efficient neural rendering.  In _2021 International Conference on 3D Vision (3DV)_ , pages 1106–1114. IEEE, 2021.
  * Prinzler et al. [2023] Malte Prinzler, Otmar Hilliges, and Justus Thies.  Diner: Depth-aware image-based neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12449–12459, 2023.
  * Qian et al. [2020] Yue Qian, Junhui Hou, Sam Kwong, and Ying He.  Pugeo-net: A geometry-centric network for 3d point cloud upsampling.  In _European conference on computer vision_ , pages 752–769. Springer, 2020.
  * Rakhimov et al. [2022] Ruslan Rakhimov, Andrei-Timotei Ardelean, Victor Lempitsky, and Evgeny Burnaev.  Npbg++: Accelerating neural point-based graphics.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15969–15979, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Sandström et al. [2023] Erik Sandström, Yue Li, Luc Van Gool, and Martin R Oswald.  Point-slam: Dense neural point cloud-based slam.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 18433–18444, 2023.
  * Schnabel and Klein [2006] Ruwen Schnabel and Reinhard Klein.  Octree-based point-cloud compression.  _PBG@ SIGGRAPH_ , 3, 2006.
  * Schneider and Eberly [2002] Philip Schneider and David H Eberly.  _Geometric tools for computer graphics_.  Elsevier, 2002.
  * Seitz et al. [2006] Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski.  A comparison and evaluation of multi-view stereo reconstruction algorithms.  In _2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06)_ , pages 519–528. IEEE, 2006.
  * Song et al. [2019] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang.  Autoint: Automatic feature interaction learning via self-attentive neural networks.  In _Proceedings of the 28th ACM international conference on information and knowledge management_ , pages 1161–1170, 2019.
  * Straub et al. [2019] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al.  The replica dataset: A digital replica of indoor spaces.  _arXiv preprint arXiv:1906.05797_ , 2019.
  * Sun et al. [2020] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2446–2454, 2020.
  * Wald and Seidel [2005] Ingo Wald and Hans-Peter Seidel.  Interactive ray tracing of point-based models.  In _ACM SIGGRAPH 2005 Sketches_ , pages 54–es. 2005.
  * Wald et al. [2007] Ingo Wald, Solomon Boulos, and Peter Shirley.  Ray tracing deformable scenes using dynamic bounding volume hierarchies.  _ACM Transactions on Graphics (TOG)_ , 26(1):6–es, 2007.
  * Wang et al. [2022] Huan Wang, Jian Ren, Zeng Huang, Kyle Olszewski, Menglei Chai, Yun Fu, and Sergey Tulyakov.  R2l: Distilling neural radiance field to neural light field for efficient novel view synthesis.  In _European Conference on Computer Vision_ , pages 612–629. Springer, 2022.
  * Wang et al. [2021a] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  _arXiv preprint arXiv:2106.10689_ , 2021a.
  * Wang et al. [2023a] Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, and Wenping Wang.  F2-nerf: Fast neural radiance field training with free camera trajectories.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4150–4159, 2023a.
  * Wang et al. [2023b] Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu.  Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3295–3306, 2023b.
  * Wang et al. [2021b] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu.  Nerf–: Neural radiance fields without known camera parameters.  _arXiv preprint arXiv:2102.07064_ , 2021b.
  * Xu et al. [2022] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann.  Point-nerf: Point-based neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5438–5448, 2022.
  * Yao et al. [2018] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 767–783, 2018.
  * Yifan et al. [2019] Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung.  Differentiable surface splatting for point-based geometry processing.  _ACM Transactions on Graphics (TOG)_ , 38(6):1–14, 2019.
  * Zhang et al. [2020] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  _arXiv preprint arXiv:2010.07492_ , 2020.
  * Zhang et al. [2023a] Yi Zhang, Xiaoyang Huang, Bingbing Ni, Teng Li, and Wenjun Zhang.  Frequency-modulated point cloud rendering with easy editing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 119–129, 2023a.
  * Zhang et al. [2023b] Yanshu Zhang, Shichong Peng, Alireza Moazeni, and Ke Li.  Papr: Proximity attention point rendering.  _arXiv preprint arXiv:2307.11086_ , 2023b.
  * Zhou et al. [2008] Kun Zhou, Qiming Hou, Rui Wang, and Baining Guo.  Real-time kd-tree construction on graphics hardware.  _ACM Transactions on Graphics (TOG)_ , 27(5):1–11, 2008.
  * Zhu et al. [2022] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys.  niceslam: Neural implicit scalable encoding for slam.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12786–12796, 2022.
  * Zuo and Deng [2022] Yiming Zuo and Jia Deng.  View synthesis with sculpted neural points.  _arXiv preprint arXiv:2205.05869_ , 2022.
  * Zwicker et al. [2001] Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross.  Surface splatting.  In _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_ , pages 371–378, 2001.

Supplementary Material

In this supplementary material, we provide details about the following topics:

  * •

Derivation of searching radius in Appendix 1;

  * •

Additional experiments in Appendix 2;

  * •

Breakdown results in Appendix 3;

  * •

Visualization in Appendix 4;

  * •

Limitation in Appendix 5.

Figure 9:  (A) Adaptive searching kernel applied on an image plane. The square
represents a pixel to be rendered on the image. The circle represents a cross-
section of the searching kernel of a cone to select points to render. (B)
Detailed derivation of the searching kernel.

##  1 Adaptive searching radius derivations

In this section, we elaborate on the derivation of adaptive searching kernel
$r$ as mentioned in Section 3.2. We initiate the derivation with a disc
representation on the image plane, approximating the area of the pixel. The
radius of the disc can be calculated by $\dot{r}=\sqrt{\Delta x\cdot\Delta
y/\pi}$, where $\Delta x$ and $\Delta_{y}$ are the width and height of the
pixel in world coordinates. For a broader coverage area, we shift from a disc
with radius $\dot{r}$ for one pixel to a larger disc with radius $\ddot{r}$
for the searching kernel. The searching kernel size $s$ can be expressed as
$s=2\cdot\left\lceil\frac{\ddot{r}}{\dot{r}}\right\rceil+1$. Specifically,
given the searching kernel $s$ and radius $\dot{r}$, the suitable range for
$\ddot{r}$ is,

$\displaystyle\left(\frac{s-1}{2}-1\right)\cdot\dot{r}<\ddot{r}\leq\left(\frac{s-1}{2}\right)\cdot\dot{r}.$
(6)

As shown in Figure 9B, to determine the radius $r$ for sample points, we
employ the concept of similar triangles, specifically $\triangle
ABF\sim\triangle ADC$. This set of similar triangles establishes a
proportionality that allows us to solve for $r$ as,

$\displaystyle r=CD=\frac{AD\times BF}{AB}.$ (7)

Figure 10:  Radius Strategies and Performance: (A) depicts an adaptive radius
correlated with viewport distance, (B) shows a constant fixed radius, and (C)
compares their PSNR performance. Additional comparison of point cloud
selection strategies in Figure 3 in the main text.

Here, $AD$ represents the Euclidean distance between the camera ray
$\mathbf{o}$ and the sample point position $\mathbf{x_{j}^{sp}}$,given by
$AD=\left\|\mathbf{x_{j}^{sp}}-\mathbf{o}\right\|_{2}$. By invoking another
set of similar triangles, $\triangle BEF\sim\triangle AEG$, we can calculate
$BF$ as,

$\displaystyle BF=\ddot{r}\cdot cos{\theta}=\frac{\ddot{r}\cdot
AG}{AE}=\frac{\ddot{r}f}{\left\|\mathbf{p_{o}}-\mathbf{o}\right\|_{2}},$ (8)

where $f$ is the local length and $\mathbf{p_{o}}$ is the pixel center
intersected by the camera ray. The length $AB$ is the hypotenuse of the right
triangle $\triangle AGB$ and is computed using the Pythagorean theorem as,

$\displaystyle AB$ $\displaystyle=\sqrt{GB^{2}+AG^{2}}$ (9)
$\displaystyle=\sqrt{\left(GE-BE\right)^{2}+AG^{2}}$
$\displaystyle=\sqrt{\left(\sqrt{AE^{2}-AG^{2}}-BE\right)^{2}+AG^{2}}$
$\displaystyle=\sqrt{\left(\sqrt{\left\|\mathbf{p_{o}}-\mathbf{o}\right\|_{2}^{2}-f^{2}}-\ddot{r}\right)^{2}+f^{2}}$

Figure 11: Adaptive sampling for noisy input, showing the transition from
primary-surface to multiple-surface sampling by adjusting $\gamma$ values,
with visual results for different surfaces. It features a visual comparison
across different surfaces. The first column displays the ground truth, and the
last column showcases the region where multi-surface sampling is applied in
Point-NeRF. The middle part illustrates the gradual transition. In the main
text, the default value of $\beta^{2}$ is set to 0.02 when dealing with
refined geometry input.

Combining these elements, the final expression for the radius $r$ of the
searching kernel is,

$\displaystyle r=\frac{\|\mathbf{x^{sp}_{j}}-\mathbf{o}\|_{2}\cdot
f\ddot{r}}{\|\mathbf{p_{o}}-\mathbf{o}\|_{2}\cdot\sqrt{\left(\sqrt{\|\mathbf{p_{o}}-\mathbf{o}\|_{2}^{2}-f^{2}}-\ddot{r}\right)^{2}+f^{2}}.}$
(10)

This integrated expression encapsulates the geometric relationship between the
image plane, the searching kernel, and the camera parameters, completing the
derivation of the adaptive searching kernel’s radius.

Room for improvement. As the angle $\widehat{BOE}$ is minuscule (less than 1
degree), we can make the below approximation and then apply a similar triangle
relationship:

$\displaystyle AB$ $\displaystyle\approx AF=AE-EF=AE-\frac{BF\times EG}{AG}$
(11)

In fact, we can further simplify eq. 11 by applying small angle assumption of
$\widehat{BOE}$ twice: $AB\approx AF\approx AE$. Eq. 7 becomes:

$\displaystyle r$ $\displaystyle\approx\frac{AD\times
BF}{AE}=\frac{\|\mathbf{x^{sp}_{j}}-\mathbf{o}\|_{2}\cdot\frac{\ddot{r}f}{\|\mathbf{p_{o}}-\mathbf{o}\|_{2}}}{\|\mathbf{p_{o}}-\mathbf{o}\|_{2}}$
(12)
$\displaystyle\approx\frac{\|\mathbf{x^{sp}_{j}}-\mathbf{o}\|_{2}\cdot\ddot{r}f}{\|\mathbf{p_{o}}-\mathbf{o}\|_{2}^{2}}$

Eq. 12 is simpler and faster to computer than eq. 10.

##  2 Additional experiments

Comparison between the fixed radius and our adaptive radius sampling.
Regarding the discussion in Section 3, we compared the performance differences
between adaptive radius and fixed radius on the NeRF-Synthesis dataset,
building upon the Point-NeRF framework. Figure 10 illustrates the differences
between adaptive and fixed radius strategies for point cloud sampling in
neural radiance fields. As explained in Section 3.5, the adaptive radius
expands with increased distance from the ray origin, resembling a camera’s
view frustum, while the fixed radius used by Point-NeRF remains constant. The
performance in Figure 10C demonstrates that the adaptive radius yields higher
PSNR values, indicating a reduction in noise by selecting only points relevant
to the rendered pixel.

Method | Rm 0 | Rm 1 | Rm 2 | Off 0 | Off 1 | Off 2 | Off 3 | Off 4 | Avg.
---|---|---|---|---|---|---|---|---|---
NICE-SLAM [69] | 0.97 | 1.31 | 1.07 | 0.88 | 1.00 | 1.06 | 1.10 | 1.13 | 1.06
Point-SLAM [48] | 0.61 | 0.41 | 0.37 | 0.38 | 0.48 | 0.54 | 0.69 | 0.72 | 0.52
Point-SLAM + Ours | 0.69 | 0.53 | 0.37 | 0.47 | 0.45 | 0.65 | 0.64 | 0.54 | 0.54

Table 6: Tracking performance on Replica [53] (ATE RMSE $\downarrow$ [cm]).

Method | Metric | Rm 0 | Rm 1 | Rm 2 | Off 0 | Off 1 | Off 2 | Off 3 | Off 4 | Avg.
---|---|---|---|---|---|---|---|---|---|---
NICE-SLAM[69] |  Depth L1 [cm] $\downarrow$ | 1.81 | 1.44 | 2.04 | 1.39 | 1.76 | 8.33 | 2.01 | 2.97 | 2.97
Precision [%] $\uparrow$ | 45.86 | 43.76 | 44.38 | 51.40 | 50.80 | 38.37 | 40.85 | 37.35 | 44.10
Recall [%] $\uparrow$ | 44.10 | 46.12 | 42.78 | 48.66 | 53.08 | 39.98 | 39.24 | 35.77 | 43.69
F1 [%] $\uparrow$ | 44.96 | 44.84 | 43.56 | 49.99 | 51.91 | 39.16 | 39.92 | 36.54 | 43.86
Point-SLAM[48] |  Depth L1 [cm] $\downarrow$ | 0.53 | 0.22 | 0.46 | 0.30 | 0.57 | 0.49 | 0.51 | 0.46 | 0.44
Precision [%] $\uparrow$ | 91.95 | 99.04 | 97.89 | 99.00 | 99.37 | 98.05 | 96.61 | 93.98 | 96.99
Recall [%] $\uparrow$ | 82.48 | 86.43 | 84.64 | 89.06 | 84.99 | 81.44 | 81.17 | 78.51 | 83.59
F1 [%] $\uparrow$ | 86.90 | 92.31 | 90.78 | 93.77 | 91.62 | 88.98 | 88.22 | 85.55 | 89.77
Point-SLAM + Ours |  Depth L1 [cm] $\downarrow$ | 0.51 | 0.23 | 0.55 | 0.37 | 0.46 | 0.54 | 0.52 | 0.42 | 0.45
Precision [%] $\uparrow$ | 92.31 | 98.96 | 97.56 | 98.85 | 99.31 | 99.33 | 97.45 | 93.95 | 97.22
Recall [%] $\uparrow$ | 83.32 | 85.98 | 84.65 | 88.86 | 85.23 | 81.74 | 82.71 | 79.38 | 83.98
F1 [%] $\uparrow$ | 87.58 | 92.01 | 90.64 | 93.58 | 91.73 | 89.68 | 89.38 | 86.05 | 90.11

Table 7: Reconstruction performance on Replica [53].

Our adaptive sampling for noisy input. This section extends the discussion on
the solution to noisy input from Section 7. Primary surface sampling
encounters challenges in accurately extracting surface features from noisy
point cloud input. As outlined in the ablation study, we proposed two methods
to address issues with noisy inputs. The first involves a brief 10-minute
geometry optimization that significantly enhances the benefit of our methods
by filling gaps and refining noisy point clouds from noisy surfaces. The
second method adaptively adjusts the sampling scope, shifting from primary
surface to multi-surface sampling. Notably, unlike full multi-surface
sampling, which gathers features from all intersected surfaces, our approach,
as illustrated in Figure 11, samples just slightly beyond the primary surface,
avoiding extensive multi-surface feature collection. Figure 12 demonstrates
that reducing $\gamma$ increases precision and reduces the speed of sampling.
A larger $\gamma$ (e.g., 0.9) focuses on primary surface sampling, maximizing
speed for interactive frame rate while maintaining competitive precision.
Conversely, a smaller $\gamma$ captures more point cloud features below the
primary surface, exceeding the precision of multi-surface sampling and still
achieving speeds over 20 times faster. In summary, the $\gamma$ adjustment
effectively balances sampling speed and precision, with each setting offering
unique benefits in efficiency and accuracy.

Figure 12:  The illustration of the trade-off between efficiency and accuracy
in the transition from primary surface to multiple surface sampling under the
noisy input by adjusting the $\gamma$ parameter. Point-NeRF with our HashPoint
and $\gamma=0.01$ outperforms original Point-NeRF in both _PSNR_ and _FPS_.

Method | Performance
---|---
Searching | Selection |  PSNR $\uparrow$ |  FPS $\uparrow$
Point-NeRF
Uniform grid | Multiple surface | 33.31 | 0.12
HashPoint | Multiple surface | 33.34 | 0.78
Uniform grid | Primary surface | 33.20 | 6.92
HashPoint | Primary surface | 33.22 | 9.60
Point-SLAM
Depth | Single surface | 30.01 | 4.64
Depth | Primary surface | 30.32 | 6.72
Pointersect
Uniform grid |  K nearest points | 31.23 | 1.30
HashPoint |  K nearest points | 31.23 | 1.75
Uniform grid | Primary surface | 30.01 | 8.89
HashPoint | Primary surface | 30.03 | 9.13
NPLF
Brute force |  K nearest points | 30.44 | 0.48
HashPoint |  K nearest points | 30.42 | 3.12
Brute force | Primary surface | 29.88 | 0.50
HashPoint | Primary surface | 29.90 | 3.40

Table 8: Ablation study evaluates on NeRF-Synthesis dataset.

| Chair | Drums | Lego | Mic | Materials | Ship | Hotdog | Ficus
---|---|---|---|---|---|---|---|---
PSNR$\uparrow$
NPBG | 26.47 | 21.53 | 24.84 | 26.62 | 21.58 | 21.83 | 29.01 | 24.60
NeRF | 33.00 | 25.01 | 32.54 | 32.91 | 29.62 | 28.65 | 36.18 | 30.13
NSVF | 33.19 | 25.18 | 32.54 | 34.21 | 29.62 | 27.93 | 37.14 | 31.23
Point-NeRF | 35.40 | 26.06 | 35.04 | 35.95 | 29.61 | 30.97 | 37.30 | 36.13
Point-NeRF + Ours | 35.54 | 26.12 | 34.68 | 36.34 | 30.64 | 31.08 | 37.02 | 34.30
SSIM$\uparrow$
NPBG | 0.939 | 0.904 | 0.923 | 0.959 | 0.887 | 0.866 | 0.964 | 0.940
NeRF | 0.967 | 0.925 | 0.961 | 0.980 | 0.949 | 0.856 | 0.974 | 0.964
NSVF | 0.968 | 0.931 | 0.960 | 0.987 | 0.973 | 0.854 | 0.980 | 0.973
Point-NeRF | 0.984 | 0.935 | 0.978 | 0.990 | 0.948 | 0.892 | 0.982 | 0.987
Point-NeRF + Ours | 0.977 | 0.931 | 0.967 | 0.984 | 0.949 | 0.920 | 0.978 | 0.978
LPIPS${}_{vgg}\downarrow$
NPBG | 0.085 | 0.112 | 0.119 | 0.060 | 0.134 | 0.210 | 0.075 | 0.078
NeRF | 0.046 | 0.091 | 0.050 | 0.028 | 0.063 | 0.206 | 0.121 | 0.044
Point-NeRF | 0.023 | 0.078 | 0.024 | 0.014 | 0.072 | 0.124 | 0.037 | 0.022
Point-NeRF + Ours | 0.028 | 0.101 | 0.047 | 0.018 | 0.075 | 0.097 | 0.036 | 0.041
LPIPS${}_{alex}\downarrow$
NSVF | 0.043 | 0.069 | 0.029 | 0.010 | 0.021 | 0.162 | 0.025 | 0.017
Point-NeRF | 0.010 | 0.055 | 0.011 | 0.007 | 0.041 | 0.070 | 0.016 | 0.009
Point-NeRF + Ours | 0.012 | 0.067 | 0.014 | 0.010 | 0.049 | 0.054 | 0.018 | 0.017

Table 9: Quantitative results in the NeRF Synthetic dataset. Figure 13:  The
qualitative results of Point-SLAM [62] with ours on the Replica [53] dataset.

Tracking and mapping. We supplement the integration of our method with Point-
SLAM [48] for tracking and reconstruction on the Replica [53] dataset. As
shown in Tables 6 and 7, our approach outperforms the original depth-guided
sampling in terms of precision and recall, while maintaining competitive
performance in other metrics.

##  3 Breakdown results

Table 9 provides detailed per-scene quantitative results comparing integration
with Point-NeRF on the NeRF-Synthesis dataset. Our integration significantly
accelerates the process while preserving competitive performance. The
comparative data for our ablation study is outlined in Table 8.

##  4 Visualization

We also experiment with the integration with Point-SLAM and NPLF on Replica
and Waymo datasets respectively. The qualitative results are shown in Figure
13 and 14. Please find more visual results for sampling comparison in our
video.

Figure 14:  The qualitative results of NPLF [41] with ours on the Waymo [54]
dataset.

##  5 Limitation

During optimization, due to gradient propagation issues, multi-surface
sampling is still necessary to sample and optimize all points as much as
possible. Currently, our choice of $\beta$ is fixed and does not dynamically
adjust based on the geometry’s distribution and noise level. Future work could
explore dynamic adjustment of the sampling process. For instance, as the
geometry is progressively optimized, $\beta$ could increase gradually,
transitioning from multi-surface to primary surface sampling.