# Real-Time Rendering of Arbitrary Surface Geometries using Learnt Transfer

Sirikonda Dhawal<EMAIL_ADDRESS>CVIT, KCIS, IIIT
HyderabadIndia500032 ,  Aakash KT<EMAIL_ADDRESS>CVIT, KCIS,
IIIT HyderabadIndia500032 and  P.J. Narayanan<EMAIL_ADDRESS>CVIT, KCIS, IIIT
HyderabadIndia500032

(2022; 2022)

###### Abstract.

Precomputed Radiance Transfer (PRT) is widely used for real-time
photorealistic effects. PRT disentangles the rendering equation into transfer
and lighting, enabling their precomputation. Transfer accounts for the cosine-
weighted visibility of points in the scene while lighting for emitted radiance
from the environment. Prior art stored precomputed transfer in a tabulated
manner, either in vertex or texture space. These values are fetched with
interpolation at each point for shading. Vertex space methods require densely
tessellated mesh vertices for high quality images. Texture space methods
require non-overlapping and area-preserving UV mapping to be available. They
also require a high-resolution texture to avoid rendering artifacts. In this
paper, we propose a compact transfer representation that is learnt directly on
scene geometry points. Specifically, we train a small multi-layer perceptron
(MLP) to predict the transfer at sampled surface points. Our approach is most
beneficial where inherent mesh storage structure and natural UV mapping are
not available, such as Implicit Surfaces as it learns the transfer values
directly on the surface. We demonstrate real-time, photorealistic renderings
of diffuse and glossy materials on SDF geometries with PRT using our approach.

Rendering, PRT, Neural Networks, Sign-distance functions, Real-time Rendering

††copyright: rightsretained††isbn: 978-1-4503-9823-7††conference: 13th Indian Conference on Computer Vision, Graphics and Image Processing; December 2022; Gandhinagar, India††journalyear: 2022††doi: 10.1145/3571600.3571640††article: 40††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the Thirteenth Indian Conference on Computer Vision, Graphics and Image Processing; December 8–10, 2022; Gandhinagar, India††booktitle: Proceedings of the Thirteenth Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP’22), December 8–10, 2022, Gandhinagar, India††price: 15.00††doi: 10.1145/3571600.3571640††isbn: 978-1-4503-9822-0/22/12††ccs: Computing methodologies Ray tracing††ccs: Computing methodologies Rasterization††ccs: Computing methodologies Visibility††ccs: Computing methodologies Neural networks |
---|---
(a) | (b)
|
(c) | (d)
Figure 1. We propose a learnt representation of transfer that can be used with
surface representations other than meshes. In this figure, we show the Old-Car
scene defined as a Signed Distance Function (SDF) rendered with our approach
for two different materials types: (a) Diffuse, (b)-(d) Glossy with increasing
Phong exponent. SDF rendering is not easily possible with traditional PRT
methods since they rely on vertices or textures sampled with UV co-ordinates
to fetch transfer, both of which are not available in SDFs. Our method
plausibly renders soft shadows and glossy highlights and achieves real-time
framerates.

##  1\. Introduction

Photo-realistic image generation is one of the most explored problems in the
field of computer graphics. Image generation with path tracing uses Monte
Carlo (MC) methods to estimate the rendering equation (Kajiya, 1986). The
stochastic nature of MC introduces noise in the images which progressively
reduces with additional MC samples. This makes path tracing computationally
expensive for real-time applications. In Precomputed Radiance Transfer (PRT)
the transfer and lighting of the rendering equation are calculated ahead of
time and stored. The stored values are then utilized for real-time
photorealistic rendering. PRT uses the Spherical Harmonic (SH) representation
to efficiently store both transfer and lighting to render complex effects in
video games and offline rendering for movie production (Pantaleoni et al.,
2010).

The transfer values are spatially-varying, scene-dependent functions defined
over the surface points of the scene. Conceptually, transfer values should be
precomputed and stored for every surface point. At rendering time, the
transfer values for each visible point should be combined with the lighting
and albedo to produce the surface radiance values. Storage at all surface
points is impossible in practice due to difficulty in computation, storage,
and parametrization. Can we directly fit a mapping function $\sigma$ from
surface parameters $(p,\hat{n})$ to transfer values $\mathcal{T}^{p}_{i}$?
That is the question we address in this work. Neural approximation for complex
functions is being used in many fields today. Recently, NeRF (Mildenhall et
al., 2020) introduced radiance functions to be learned over coordinates of
volumetric space. We extend this idea to fit a function $\sigma(p)$ from
surface points to corresponding transfer values using a Multi-Layer Perceptron
(MLP) network. The network learns a continuous representation of transfer over
the surface points rather than store discrete points. In the past, Sloan et
al. (Sloan et al., 2002) proposed to store pre-computed transfer values at
each vertex of the scene’s mesh representation. The radiance computed on each
vertex point was interpolated to interior points. Storing the transfer values
in a texture to be interpolated during rendering (McKenzie Chapter, 2010;
Iwanicki and Sloan, 2009; Dhawal et al., 2022) has also been tried out. Such
methods need dense tessellation and/or high-resolution textures on complex
scenes. These methods are limited to surface representations with good UV
parametrization. Meshless hierarchical light transport (Lehtinen et al., 2008)
removes the dependence on triangle meshes using sampled scene points and KNN
search. This requires iterative deferred samplings to get good enough shading.

In this paper, we present a method that directly learns a function to generate
transfer values to surface points. A small MLP holds the continuous
representation of transfer values. Since the function is over surface points,
our method can work on mesh models, implicit models, parametric models, etc.
The MLP can directly be used for real-time rendering by encoding the network
in a GLSL shader. We also present a CUDA-based system to handle larger and
variable MLPs. We show high-quality rendering using PRT on surfaces
represented using Signed Distance Fields (SDFs). These representations are
gaining popularity (Davies et al., 2021; Takikawa et al., 2021) and ours is
the first method of photorealistic rendering on them. We validate our
renderings with traditional PRT renderings and demonstrate equivalent quality
using mesh-based geometric representations Fig 6, 7. Our method represents
large scene by dividing them into smaller segments (Sec 4.4), each with its
own MLP function.

##  2\. Related Work

Spherical Harmonic (SH) lighting was first proposed by (Ramamoorthi and
Hanrahan, 2001). This led to the development of PRT, proposed by (Sloan et
al., 2002; Sloan, 2008) to disentangle rendering equations into the transfer,
lighting, and material while individually projecting them to the SH domain.
For diffuse materials, the transfer is stored as a vector while it is stored
as a matrix for glossy materials. The work by (Ng et al., 2004) allows storing
a vector for transfer even in-case of glossy materials by using the tripling
coefficient matrix. These traditional approaches as well as the newer ones for
polygonal lighting shading in PRT (Wang and Ramamoorthi, 2018; Wu et al.,
2020) store transfer at vertices. These approaches necessitate dense
tessellation to account for transfer changing at high frequency in the scene.
Adaptive re-meshing techniques proposed by (Křivánek et al., 2004) store
transfer at vertices while re-meshing the region where the shadows are
missing. Works by (Iwanicki and Sloan, 2009; McKenzie Chapter, 2010) store
low-order SH coefficients at UV-mapped textures to account for diffuse
results. The limitations of these methods are, they are constrained to mesh
representation either by using a vertex storage approach or by using a texture
storage approach. Both of these storage schemes are not present in Implicit
surface representations like SDFs limiting such geometric representations to
be used in PRT frameworks.

Discrete Storage of transfer: To remove the dependency on the mesh
representation, Meshless hierarchical transport proposed by (Lehtinen et al.,
2008) samples and stores selective points by accounting for high-fidelity
changes in the transport function. But this approach requires multiple re-
sampling and weighted k-nearest neighbor searches to obtain a transfer to a
given query point. Another approach called Irradiance Volumes (Greger et al.,
1998) can be used to store the SH-transfer vectors in volumetric grids.
Usually, this approach is prone to bleeding and leakage, causing artifacts.
All of these methods rely on a discrete representation of transfer.

Function Representation of Transfer: We propose to learn a functional
representation of transfer which regresses transfer given a query point in the
scene geometry. While the works like (Ren et al., 2013) try to regress
radiance, we aim to regress the transfer. In our case, the transfer is learnt
on the scene geometry, which enables us to use other surface representations
like implicit surface representations and parametric surface representations
e.g. SDF which lack storage schemes. Recent works (Li et al., 2019) uses
harmonic mapped mesh geometry to learn a transfer similar to ours to reduce
the memory limitation on dynamic scene. But the work depends on the spatial
information preserved between animated harmonic maps leveraging the spatial
prior of convolution operations, while also still being limited to a mesh
representation. NeuralPRT by Rainer et al. (2022) and Neural Radiance Transfer
Fields by Lyu et al. (2022) take inspiration from PRT, learning latent
representation for transfer with a lighting, diffuse and glossy descriptor.
Furthermore, they operate in the image space and perform loss calculations in
the image domain, whereas we operate in the SH space. For disentangling
components from the final rendered images they employ large neural MLPs for
their constituent rendering components, limiting their run times. We on the
other hand stick to traditional formulations of PRT allowing our method to be
easily integrated into existing frameworks like (Wu et al., 2020; Wang and
Ramamoorthi, 2018)

##  3\. Background

In the Triple Product formulation (Ng et al., 2004) of PRT that we use, the
rendering equation for direct lighting at point $p$ is given by:

(1) $\useshortskip
B^{p}(\omega_{o})=\int_{\Omega}L(\omega_{i})\rho^{p}(\omega_{o},\omega_{i})V^{p}(\omega_{i})(\omega_{i}\odot
n)d\omega_{i},$

where $\omega_{o}$ is the direction towards the viewer from $p$, $\omega_{i}$
is the incoming direction on the unit hemisphere $\Omega$ and $n$ is the
surface normal at $p$. $B^{p}$ is the reflected radiance in direction
$\omega_{o}$, $L$ is the incoming environment light from $\omega_{i}$, $V^{p}$
is the binary visibility function and $\rho^{p}$ is the Phong Bi-directional
Reflectance Distribution Function (BRDF) (Phong, 1975). Eq. 1 is decomposed
into the lighting $L$ and transfer
$T^{p}(\omega_{i})=V^{p}(\omega_{i})(\omega_{i}\odot n)$ which are then
projected to the SH basis with coefficients $\mathcal{L}_{i}$ and
$\mathcal{T}^{p}_{i}$ respectively. For a glossy BRDF, the SH coefficients
$\mathcal{H}^{p}_{k}$ of transferred radiance at point $p$ are given by:

(2)
$\useshortskip\mathcal{H}^{p}_{k}=\int_{S^{2}}y_{k}(\omega)\left(\sum_{i=1}^{n^{2}}\mathcal{T}^{p}_{i}y_{i}(\omega)\right)\left(\sum_{j=1}^{n^{2}}\mathcal{L}_{j}y_{j}(\omega)\right)d\omega=\sum_{ij}\tau_{ijk}\mathcal{T}^{p}_{i}\mathcal{L}_{j},$

where $\tau_{ijk}$ is the triple product tensor (tripling coefficient matrix)
and $y$ are SH basis functions. The reflected radiance $B^{p}$ is calculated
by convolution of $H_{k}^{p}$ with SH coefficients of the BRDF and evaluation
at reflection direction. For a diffuse BRDF, reflected radiance $B^{p}$ can be
directly calculated as:

(3) $\useshortskip B^{p}=\sum_{i}\mathcal{T}^{p}_{i}\mathcal{L}_{i}.$

In both cases, this formulation results in a $k$ dimensional vector for
transfer at point $p$. Note that computing the reflected radiance is
computationally more expensive for glossy BRDFs since it involves a matrix-
vector product, followed by convolution and dot product rather than just a dot
product in the case of diffuse BRDFs.

Figure 2. CUDA Implementation: We extract G-buffers that store the position
$p$ and normal $\hat{n}$ in texture space. Along with G-buffers we also
project per fragment view directions to SH and store them in textures. All
OpenGL Textures ( ), are shared with CUDA buffers( ). The CUDA buffers are
mapped to the same memory location as OpenGL texture to avoid expensive GPU -
Host transfers. The CUDA buffers are copied to preloaded torch-sensors( )
residing on GPU. The tensors are then fed to the network residing on GPU and
the output is obtained. Since the operations are only between GPU-GPU without
involving the host we avoid host latency. The network outputs a transfer
vector which is clubbed with lighting and SH representation of view direction
extracted as torch tensor to obtain color at each pixel using the Triple
Product Formulation (Ng et al., 2004). Note: Before passing the G-buffers to
MLP, fragments that intersect the geometry are separated from the ones that do
not and packed to avoid unnecessary network computations.

##  4\. Method

In this section, we first motivate and describe our learnt representation of
transfer. Next, we discuss training details along with training data
generation which ensures that our learnt representation is continuous.
Finally, we describe the GLSL and CUDA implementations of our learnt
representation. We evaluate and show the results of both these implementations
in Sect. 5.

Figure 3. Our network is a Multi-Layer Perceptron (MLP) with $k$ neurons per
layer and $l$ layers. A scene point $p$ and its corresponding normal
($\hat{n}$) are fed to the network as input with positional encoding. The
network outputs a vector, which are SH coefficients of transfer. The black-
arrow lines represent a leaky-relu activation while the dotted arrow
represents a tanh activation. Scenes rendered in this paper use $k=64,l=4$ or
$k=128,l=4$ unless otherwise specified.

###  4.1. A learnt representation of transfer

The PRT formulation in Equation 3 describes the evaluation of direct lighting
at every point $p$. Ideally, it needs to be pre-computed at every visible
point in the rendered view requiring transfer-vectors
($\overrightarrow{\mathcal{T}^{p}}$) at all surface points of geometry. This
is practically impossible. Instead, we learn a continuous function on the
surface of the scene geometry as a function $\sigma$

(4) $\sigma:(p,\hat{n})\to\overrightarrow{\mathcal{T}^{p}}$

that maps surface position($p$) and normal($\hat{n}$) to a transfer value. We
use a Multi-Layered Perceptron (MLP) network as $\sigma$. Neural networks are
known to be universal function approximators (Hornik et al., 1989). Recently,
neural representations using coordinate-base MLPs have found a great utility
to learn the distribution of radiance in a scene (Mildenhall et al., 2020;
Gafni et al., 2021; Reiser et al., 2021; Zhang et al., 2021). We adapt this
idea and learn $\sigma$ as an MLP to represent a continuous function over the
given scene surface. We train the MLP by precomputing the transfer values at
densely sampled surface points. This method works on all types of surface
representations: meshes, implicit surfaces, SDFs, parametric surfaces, etc.

Formally, given a scene point $p$ and its normal $\hat{n}$, we train a MLP to
learn a mapping function $\sigma$ presented in Eq 4 to learn the SH-vector of
transfer $\overrightarrow{\mathcal{T}^{p}}$ with components
$\mathcal{T}^{p}_{i}$ (Eq. 2, 3). In our experiments, we use $16$ SH
coefficients similar to Wu et al. (2020), making
$\overrightarrow{\mathcal{T}^{p}}$ $16$ dimensional. The network architecture
used is shown in Fig 3. It contains $l$ layers with $k$ neurons per layer.
Each layer is followed by the leaky-relu (Maas, 2013) activation function,
except the last which is followed by tanh activation. In practice, the network
inputs are projected to a high dimensional representation using positional
encoding (Rahaman et al., 2019; Mildenhall et al., 2020; Tancik et al., 2020).
We set $k=64$ and $l=4$ or $k=128$ and $l=4$ for scenes in this paper unless
otherwise specified. Previously, Sloan et al. (2002) and Křivánek et al.
(2004) stored transfer values on mesh vertices. Textures in a UV-space were
also used to store transfer values (McKenzie Chapter, 2010; Dhawal et al.,
2022). These strategies strongly rely on dictionary storage structures
inherently present in mesh-based geometric representations and do not extend
to implicitly represented scene objects. They also incur high memory costs to
store transfers for complex scenes.

###  4.2. Training details

Our training dataset consists of transfer values for densely sampled points in
the scene. We ensure an equal distribution of these points by an area-based
sampling of the mesh surface (Turk, 1990). The ground truth transfer is
calculated using ray-tracing for each sampled point, as is done traditionally
(Ramamoorthi, 2009). Note that training data can be generated for any other
geometric representation by first converting it to a mesh and then applying
the above routine. For example, in the SDF case, we use marching cubes to
first extract a mesh, densely sample the mesh using the above method and
project these points to the SDF surface. The Transfer is then calculated for
these points by sphere tracing the SDF. We project the resulting transfer to
SH basis with $16$ coefficients which are used as ground truth. By training
our network with dense and equally distributed samples, we ensure that it
learns a continuous representation of transfer. For scenes used in this paper,
we generate $800k-1M$ points on the geometric surface and calculate the
associated transfer vectors to train our network. The data generation takes
around 2 hours per scene. We use a batch size of 8192 and train our network
with this data for around 200 epochs with the $\ell_{1}$ loss. The time taken
for training is about 10 minutes on NVIDIA RTX 3090 GPU. We use sampled points
as a training set and try to regress on the vertex locations which are not
included in the training sample to ensure the fitted function is consistent
with the surface geometry. Our sampling scheme (Turk, 1990) ensures that the
sampled points do not co-inside with the vertex locations whereby keeping the
test set different from the train set ensuring uniform fitting rather than an
overfit of the region. Refer to Suppl. for more details about training and
sampling.

###  4.3. Real-time rendering with GLSL & CUDA

Once trained on a given scene, our network can output transfer vectors for any
point in that scene. Furthermore, since our network is small, it allows for
efficient per-pixel evaluation on the GPU. Below we discuss two different
implementations, one with GLSL and the other with CUDA.

####  4.3.1. GLSL

We implement the GLSL version within the ModernGL framework (Dombi, 2020). The
network weights and biases are hardcoded in $4\times 4$ matrices (mat4 type)
in the fragment shader. Specifically, the weights of each layer are divided to
fit into multiple mat4’s. This allows for an efficient forward pass using
matrix-vector products on the GPU. We have automated this using a script-based
extraction of weights into the shader. With the network weights available as
matrices within the shader, the rendering proceeds as follows. We first obtain
the surface position $p$, its normal $\hat{n}$, and the view vector
$\omega_{o}$ in the fragment shader using the standard OpenGL pipeline. An
early depth pass is used to avoid the processing of unnecessary fragments.
Next, we apply positional encoding (Rahaman et al., 2019) to $p$ and $\hat{n}$
and evaluate the forward pass of our network with the hardcoded weights. This
results in the SH vector of transfer $\overrightarrow{\mathcal{T}^{p}}$ at
$p$. The transfer is used with the triple product formulation (Ng et al.,
2004) with a global light matrix to obtain the shading at $p$ according to Eq.
2, 3. Our GLSL implementation works for both AMD and NVIDIA GPUs since OpenGL
itself is supported on both hardware architectures. A downside of this
implementation is that the network size is constrained by the amount of local
memory available for each shader.

####  4.3.2. CUDA

To alleviate the dependence of network size on local shader memory, we
implement the second version in CUDA. Before rendering starts, we pre-load the
trained weights of our network on the GPU. Rendering is then performed in two
separate passes: (1) An OpenGL pass to extract G-buffers, (2) A dedicated CUDA
pass for network forward evaluation using the pre-loaded weights. The OpenGL
pass works similarly to the GLSL implementation, except it only extracts
G-buffers and does not perform shading. We create CUDA buffers that point to
the resulting G-buffers from the previous step, which implies that these share
the same GPU memory. These CUDA buffers are then converted to PyTorch (Paszke
et al., 2019) tensors using the PyCuda (Klöckner et al., 2012) interface. This
approach avoids expensive transfers between GPU and the host and is crucial to
achieving real-time framerates. The PyTorch tensors are then used for the
network forward pass with positional encoding. The network evaluation only
processes parts of the G-buffers that intersect geometry by packing them
before the network forward evaluation. The transfer vectors obtained from the
network are then unpacked to their original locations in the G-buffer. This
avoids unnecessary evaluations of the network further improving the FPS. The
final image is obtained by combining this transfer with global light matrices
with the triple product formulation (Ng et al., 2004), similar to the GLSL
pass. This implementation allows the network to be as large as the entire GPU
memory since it has a dedicated forward pass for it. Refer to Fig.2 for a
visual explanation of this implementation and a caption for details.

###  4.4. Handling Large Scene without loss of performance

In certain scenarios, while handling large scenes, a small neural network
might not be sufficient to regress the transfer-accounting for the cosine
weighted visibility of the rendering equation. In such a scenario, we can
either engage a large neural network or subdivide the scene into smaller
segments and assign a small network to regress the respective transfer values.
The latter is a more performance-friendly approach as the number of operations
required to calculate transfer will be substantially smaller than the former.
This is due to the fact that per fragment only a small MLP needs to be
evaluated as opposed to a large network, resulting in real-time performance
obtaining render-framerates of over 200FPS. To achieve this we subdivided the
scene into sub-scenes and sample points in each sub-scene using the strategy
explained in Sec.4.2. To maintain continuous seamless regression of transfer
at boundaries we sample points for extra $\delta$ area over the adjacent sub-
scene refer to Fig.4 for details. For each sample point, we trace rays into
the un-subdivided(whole scene) scene. This ensures the visibility accounting
for the full scene rather than the sub-scene. Once transfer vectors are
calculated, to ensure a minimal number of MLPs being used to regress the
transfer of the whole scene we cluster neighboring sub-scenes using Variances
analysis of the transfer vectors similar to the techniques of CPCA(Sloan et
al., 2003). Contrary to their method, we do not use Principle components
rather just rely on the small MLPs to regress transfer. At run-time, every
sub-scene is assigned respective MLP either in GLSL or CUDA based
implementation. Thus, enabling parallel execution of the fragments achieving a
real-time framerates of 200FPS. A visual representation of the above strategy
is given in Fig. 5.

Figure 4. Sampling at boundaries: While handling the boundary regions of
neighboring sub-scenes which are caused due to the sub-division. We sample
$\delta$ area more into the adjacent sub-scene and include the sample points
falling into that $\delta$ region into the training set of the sub-scenes MLP.
This ensures smooth learning of transfer vectors and avoids seam artifacts.
Figure 5. Large Scene: The Figure on the top-left shows the top-view of Sponza
Cathedral divided into 12 even parts. We use the heuristic presented in
Sec.4.4 to join the few sub-scene to reduce the number of networks required to
regress the Transfer. The clubbed geometries are visualized in the bottom-left
image where the components of the middle section are joined with their
respective neighbors. Please observe that both top-left and bottom-left images
are color-coded, each unique colors represent a sub-scene. We utilize the
bottom-left configuration of the sub-scene which only requires 9 small MLPs
rather than 12 as in the case of top-left. We visualized the resulting render
in the right. It is to be noted that we have used White light to show the
transfer regressed from the network is independent of the Lighting and
material of the object and can be swapped with ease as depicted in Fig.(8,1).
FPS of respective materials are mentioned in Figure.

##  5\. Validation & Results

In this section, we validate and show the results of our learnt representation
(Sect. 4.1) implemented in GLSL and CUDA (Sect. 4.3). Our network is trained
for each scene and we generate training data as outlined in Sect. 4.2. We
compare our renderings with baseline PRT, which uses texture-based storage of
transfer (Dhawal et al., 2022; McKenzie Chapter, 2010) with the triple product
formulation (Ng et al., 2004). We use the texture version as the baseline
since it produces artifact-free renderings in most cases and avoids the memory
overhead of vertex-based approaches for dense meshes. The baseline
implementation stores transfer in a $1024\times 1024$ texture with optional
texture-sets for large meshes. In Sect. 5.1, we validate our renderings of
triangle meshes by qualitative and quantitative comparison with baseline PRT.
In Sect. 5.2, we show results on scenes with SDF as the geometric
representation. This is possible since our network learns a continuous
representation of transfer and can predict it for any point in the scene. We
show results on both analytical and neural SDFs. Lastly in Sect. 5.3, we
compare and contrast our GLSL and CUDA implementations and discuss their
framerates for different network sizes. All scenes are rendered using our GLSL
implementation (unless otherwise specified) on an NVIDIA RTX 3090 GPU and at a
resolution of $1024\times 1024$.

Figure 6. Diffuse Mesh Results: Since our learned function is only accounting
for transfer we can dynamically change the light on the fly without retraining
the network. We show results of Roza with two different lightings while using
the same set of learned weights. Our approach plausibly renders soft shadows,
especially in high-frequency visibility changes in the hair. Figure 7. We show
the results of our approach and compare them with the baseline (left), which
is a texture-based transfer storage implementation of PRT. Our results are
rendered with $k=64,l=4$ network (middle) and $k=128,l=4$ network (right).
Insets are provided at the right-extreme, where we can observe that $k=64,l=4$
network produces plausible results albeit with a few jaded lines in the case
of Plants and Dining-Table , while $k=128,l=4$ network gets rid of those
artifacts and matches the baseline. We note that $k=64,l=4$ network
comfortably achieves real-time FPS while $k=128,l=4$ has a lower FPS (Tbl. 1).
Please inspect insets closely

###  5.1. Validation on triangle meshes

We validate our results on four scenes with triangle meshes: Teaset , Plants ,
Roza , & Dining-Table . Additionally, we also show results on a large scene
leveraging the strategy presented in Sec. 4.4 and Fig. 5 shows rendered
results. These scenes contain high-frequency changes in visibility, for
example in the hair of Roza and shadows due to leaves in Plants . And our
network is able to regress the SH vectors of transfer with ease. We render our
results with diffuse and glossy materials using two different sizes of the
network: $k=64,l=4$ and $k=128,l=4$ and compare them with baseline PRT.
Rendering results are shown in Fig. 7 and quantitative metrics (MAE, PSNR,
SSIM) along with frame times (FPS) are shown in Table 1. The quantitative
metrics are calculated by comparing our rendering with baseline PRT as a
reference. Our rendering results closely match the baseline and achieve high
PSNR and SSIM values, which validates our method. We further compare our
rendering of the diffuse Roza scene with baseline PRT in Fig. 6. Our method is
able to reproduce fine soft shadows which further strengthens our validation.
As shown in Table 1, the $k=64,l=4$ network comfortably achieves real-time FPS
for both diffuse and glossy materials. A larger network ($k=128,l=4$) achieves
better rendering quality and metrics albeit at a drop in FPS.

Diffuse
---
Scene | k=64, l=4 | k=128, l=4
MAE | PSNR | SSIM | FPS | MAE | PSNR | SSIM | FPS
Teaset | 0.00074 | 49.227 | 0.99708 | 350.5 | 0.00055 | 51.050 | 0.99757 | 35.1
Plants | 0.00109 | 44.450 | 0.99399 | 201.2 | 0.00076 | 46.290 | 0.99573 | 20.5
Roza | 0.00123 | 44.068 | 0.99462 | 330.2 | 0.00080 | 48.261 | 0.99658 | 38.1
Dining-Table | 0.00168 | 41.917 | 0.99066 | 300.0 | 0.00106 | 43.494 | 0.99280 | 40.1
Glossy
Scene | k=64, l=4 | k=128, l=4
MAE | PSNR | SSIM | FPS | MAE | PSNR | SSIM | FPS
Teaset | 0.00133 | 45.752 | 0.99356 | 105.5 | 0.00102 | 48.129 | 0.99504 | 25.1
Plants | 0.00319 | 35.701 | 0.98237 | 64.5 | 0.00250 | 36.614 | 0.98661 | 12.2
Roza | 0.00122 | 42.449 | 0.99701 | 120.5 | 0.00083 | 46.768 | 0.99817 | 31.1
Dining-Table | 0.00411 | 32.638 | 0.97393 | 130.1 | 0.00312 | 33.414 | 0.97886 | 32.1
Table 1. This table shows comparison among two different network sizes used to
learn the transfer function. The quantitative metrics (MAE, PSNR & SSIM) are
calculated by comparing our rendering with baseline PRT as reference, which is
a texture storage implementation of PRT. These metrics averaged amongst 30
uniformly sampled views in a trajectory for both Diffuse and Glossy material
configurations. We also show the FPS obtained for respective network sizes.
Our approach renders in real-time for both diffuse and glossy configurations.

###  5.2. Results on SDF

We now present our rendering results on scenes with SDF geometry. As stated
earlier, implicit representations like SDFs do not have an inherent storage
schema like the Mesh, which makes it very hard to store transfer vectors,
hence we utilize the MLPs we trained to regress transfer. Rendering is done
using the GLSL implementation (Sect. 4.3.1). We sphere trace the SDF and
obtain transfer at the intersection point using our network. Since sphere
tracing needs to be done for all fragments, the scene geometry is set to two
triangles that cover the entire image. We show results on four analytic SDFs,
Mike-Monster , Rabbit from (Fizzler, 2018; Quilez, 2013) and Old-Car , Fish
from (Takikawa et al., 2022). We also show results with one Neural SDF,
Stanford-Bunny using the method of (Davies et al., 2021).

Renderings with glossy materials using our method are shown in Fig. 8. We
render these scenes with two different environment lighting. All highlights
from the environment map are reproduced on the glossy surface accurately. For
instance, the yellowish ground in the environment map reflects at the bottom
of the SDF and the bluish sky reflects at the top. We also show renderings of
one diffuse and three glossy results with increasing Phong exponents on the
Old-Car SDF in Fig. 1. All scenes produce real-time FPS while producing
plausible glossy highlights and soft shadows as shown in Tbl. 2, Figs. 8, 2.
The FPS for SDFs behaves similarly to triangle meshes, in that the $k=64,l=4$
network comfortably achieves real-time FPS while the $k=128,l=4$ network has a
lower FPS.

Scene | FPS
---|---
k=64, l=4 | k=128, l=4
Diffuse | Glossy | Diffuse | Glossy
Old-Car | 254.96 | 207.28 | 73.2 | 38.8
Mike-Monster | 294.28 | 173.6 | 61.0 | 24.2
Fish | 320.1 | 231.6 | 77.1 | 50.1
Rabbit | 310.21 | 205.94 | 73.2 | 37.9
Table 2. Performance on diffuse and glossy renders of analytic SDFs. Our
approach can render SDFs within the PRT framework in real time. Fish Mike-
Monster Rabbit

Lighting 1

Lighting 2

Figure 8. SDF Results: With the use of learnt transfer approach we extended PRT onto SDF where UV mapping or vertex storage is not defined. The learnt function approximates the transfer well. Results are shown on three SDFs: Mike-Monster , Fish , Rabbit with two different lighting environments. Figure 9. We show renderings of the Stanford-Bunny neural SDF learnt using two network architectures. We show diffuse and glossy renderings of the neural SDF by evaluating our learnt transfer on the surface. Note that the geometric representation of Stanford-Bunny is obtained from the (Davies et al., 2021) and we only contribute learnt transfer to it. Surface Transfer | k=64, l=4 | k=128, l=4
---|---|---
Diffuse | Glossy | Diffuse | Glossy
k=32, l=2 | 320 | 200 | 77 | 20.1
k=64, l=2 | 50 | 25 | 26.5 | 15.2
Table 3. FPS on the Stanford-Bunny Neural SDF (Fig. 9).

Finally, we show our rendering of the Stanford-Bunny scene, which is defined
as a Neural SDF as a proof of concept. We use the SDF optimized by (Davies et
al., 2021). The result is shown in Fig. 9 for diffuse and glossy material and
two different lighting conditions. We also show the FPS in Tbl. 3 for two
different SDF networks sizes (vertical) and two different sizes of our network
(horizontal). The renderings are plausible and achieve real-time framerates
for smaller network sizes while maintaining interactivity for larger network
sizes. The FPS could be further improved by optimizing the tracing of SDFs
(Takikawa et al., 2021), a research direction orthogonal to ours.

Implementation | $k=64$ | $k=128$ | $k=256$ | $k=512$
---|---|---|---|---
GLSL | 64.4 | 12.2 | N/A | N/A
CUDA | 55.1 | 29.1 | 20.7 | 8.0
Table 4. CUDA vs GLSL: Table contains the Performance in FPS of the GLSL and
CUDA implementation with various MLP architectures ranging from
$k\in(64,512),l=4$ tested on Plants scene with a glossy material. The GLSL
implementation outperforms the CUDA implementation for $k=64$, but degrades at
larger network sizes or fails to render ($k=256,k=512$). The CUDA
implementation renders in all cases and is real-time for $k=64$ while
maintaining interactivity for other cases. Please note we use the same number
of layers($l=4$) in all the architectures presented in the table

###  5.3. GLSL & CUDA: Comparison

We compare and contrast the GLSL and CUDA implementations of our learnt
representation. As mentioned before, the GLSL implementation limits the
network size to the available local shader memory. On the other hand, CUDA
allows the network to be as large as the entire GPU memory. We experimented
with four different network sizes for both implementations: $k=64$, $k=128$,
$k=256$ & $k=512$ with $l=4$. The FPS for these configurations on the Plants
scene are listed in Tbl. 4. The GLSL implementation outperforms for $k=64$
network, while the performance drops on higher sizes. It can be observed that
CUDA starts to take the lead from $k=128$ network. In fact, for $k=256$ and
above, GLSL implementation fails to render. Although the performance of the
CUDA implementation drops for higher network sizes, it is still able to load
and render the scene interactively.

###  5.4. Memory requirements

Thanks to our simple network structure, the entire transfer of a scene can be
represented as a set of weights. In vertex-based storage of transfer as in
Sloan et al. , the memory requirement has a one-to-one correspondence with the
mesh tessellation. With texture-based storage, this correspondence is reduced,
with the caveat of needing a good UV-mapping or possibly a large texture. With
our method, this correspondence is further reduced as the entire transfer is
now just a set of weights. For example, a mesh with $1M$ vertices requires
$\approx 70$MB data stored on vertices, while our method only needs around
$64$KB - $5$MB. On the other-hand grid-based storage requires 256MB of data
for a grid resolution of $512^{3}$.

##  6\. Discussion & conclusion

In this paper, we presented a learnt representation of transfer through small
MLPs in traditional PRT frameworks. Our main motivation was to alleviate the
dependence of transfer storage in PRT on geometry representations. We ensured
that our network learns a continuous representation of the transfer in a scene
by densely and equally distributing training samples over the surface. We
provided a strategy to handle large-scenes in Sec.4.4 and Fig. 5 while
maintaining real-time frame-rates. We demonstrated two implementations of our
learnt representation: GLSL and CUDA. We analyzed both implementations and
discussed their merits and drawbacks. We further demonstrated that both
implementations achieve real-time FPS on meshes as well as SDFs. SDF rendering
which we demonstrated with our approach was not easily possible within the PRT
framework. Utilizing our approach the works (Wu et al., 2020; Wang and
Ramamoorthi, 2018) can incorporate SDF geometries into their scenes.

One interesting challenge would be to empirically determine the optimal size
of the network for a given complex scene, we would like to work on it in the
future. We would also like to study the benefit of representing other surface
properties, like specularity and albedo with our proposed approach. Finally,
we would like to investigate the incorporation of inter-reflections with our
approach, without baking in BRDF as done in (Sloan et al., 2002), for real-
time global illumination with PRT with flexible surface representations.

Acknowledgement:  Dhawal would like to thank Pratikkumar Bulani for helping
set up the SDF scenes. Dhawal Sirikonda was partially supported by DST. Aakash
KT was supported by the Kohli Research Fellowship. We also acknowledge support
from the TCS Foundation through the Kohli Center on Intelligent Systems
(KCIS).

## References

  * (1)
  * Davies et al. (2021) Thomas Ryan Davies, Derek Nowrouzezahrai, and Alec Jacobson. 2021\.  On the Effectiveness of Weight-Encoded Neural Implicit 3D Shapes.  (2021).
  * Dhawal et al. (2022) Sirikonda Dhawal, Aakash KT, and P. J. Narayanan. 2022\.  PRTT: Precomputed Radiance Transfer Textures.  https://doi.org/10.48550/ARXIV.2203.12399
  * Dombi (2020) Szabolcs Dombi. 2020\.  ModernGL, high performance python bindings for OpenGL 3.3+.  Github.
  * Fizzler (2018) Fizzler. 2018.  Rabbit SH18.  Shadertoy.  https://www.shadertoy.com/view/XlccWH
  * Gafni et al. (2021) Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2021.  Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 8649–8658.  https://openaccess.thecvf.com/content/CVPR2021/html/Gafni_Dynamic_Neural_Radiance_Fields_for_Monocular_4D_Facial_Avatar_Reconstruction_CVPR_2021_paper.html
  * Greger et al. (1998) Gene Greger, Peter Shirley, Philip M. Hubbard, and Donald P. Greenberg. 1998.  The Irradiance Volume.  _IEEE Comput. Graph. Appl._ 18, 2 (mar 1998), 32–43.  https://doi.org/10.1109/38.656788
  * Hornik et al. (1989) Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989\.  Multilayer feedforward networks are universal approximators.  _Neural Networks_ 2, 5 (1989), 359–366.  https://doi.org/10.1016/0893-6080(89)90020-8
  * Iwanicki and Sloan (2009) Michal Iwanicki and Peter-Pike Sloan. 2009.  Normal Mapping with Low-Frequency Precomputed Visibility. In _SIGGRAPH 2009: Talks_ (New Orleans, Louisiana) _(SIGGRAPH ’09)_. ACM, New York, NY, USA, Article 52, 1 pages.  https://doi.org/10.1145/1597990.1598042
  * Kajiya (1986) James T. Kajiya. 1986\.  The Rendering Equation. In _Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’86)_. Association for Computing Machinery, New York, NY, USA, 143–150.  https://doi.org/10.1145/15922.15902
  * Klöckner et al. (2012) Andreas Klöckner, Nicolas Pinto, Yunsup Lee, B. Catanzaro, Paul Ivanov, and Ahmed Fasih. 2012.  PyCUDA and PyOpenCL: A Scripting-Based Approach to GPU Run-Time Code Generation.  _Parallel Comput._ 38, 3 (2012), 157–174.  https://doi.org/10.1016/j.parco.2011.09.001
  * Křivánek et al. (2004) Jaroslav Křivánek, Sumanta Pattanaik, and Jiří Žára. 2004.  Adaptive Mesh Subdivision for Precomputed Radiance Transfer. In _Proceedings of the 20th Spring Conference on Computer Graphics_ (Budmerice, Slovakia) _(SCCG ’04)_. Association for Computing Machinery, New York, NY, USA, 106–111.  https://doi.org/10.1145/1037210.1037226
  * Lehtinen et al. (2008) Jaakko Lehtinen, Matthias Zwicker, Emmanuel Turquin, Janne Kontkanen, Frédo Durand, François X. Sillion, and Timo Aila. 2008.  A Meshless Hierarchical Representation for Light Transport. In _ACM SIGGRAPH 2008 Papers_ (Los Angeles, California) _(SIGGRAPH ’08)_. Association for Computing Machinery, New York, NY, USA, Article 37, 9 pages.  https://doi.org/10.1145/1399504.1360636
  * Li et al. (2019) Yue Li, Pablo Wiedemann, and Kenny Mitchell. 2019.  Deep Precomputed Radiance Transfer for Deformable Objects.  _Proc. ACM Comput. Graph. Interact. Tech._ 2, 1, Article 3 (jun 2019), 16 pages.  https://doi.org/10.1145/3320284
  * Lyu et al. (2022) Linjie Lyu, Ayush Tewari, Thomas Leimkuehler, Marc Habermann, and Christian Theobalt. 2022.  Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination. In _European Conference on Computer Vision (ECCV)_.
  * Maas (2013) Andrew L. Maas. 2013\.  Rectifier Nonlinearities Improve Neural Network Acoustic Models.
  * McKenzie Chapter (2010) Harrison Lee McKenzie Chapter. 2010\.  Textured Hierarchical Precomputed Radiance Transfer.  (2010).  https://doi.org/10.15368/theses.2010.101
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In _ECCV_.
  * Ng et al. (2004) Ren Ng, Ravi Ramamoorthi, and Pat Hanrahan. 2004.  Triple Product Wavelet Integrals for All-Frequency Relighting.  _ACM Trans. Graph._ 23, 3 (Aug. 2004), 477–487.  https://doi.org/10.1145/1015706.1015749
  * Pantaleoni et al. (2010) Jacopo Pantaleoni, Luca Fascione, Martin Hill, and Timo Aila. 2010\.  PantaRay: Fast Ray-Traced Occlusion Caching of Massive Scenes.  _ACM Trans. Graph._ 29, 4, Article 37 (July 2010), 10 pages.  https://doi.org/10.1145/1778765.1778774
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems 32_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024–8035.
  * Phong (1975) Bui Tuong Phong. 1975\.  Illumination for Computer Generated Pictures.  _Commun. ACM_ 18, 6 (jun 1975), 311–317.  https://doi.org/10.1145/360825.360839
  * Quilez (2013) Inigo Quilez. 2013\.  Pixar Mike Monster Inc.  Shadertoy.  https://www.shadertoy.com/view/MsXGWr
  * Rahaman et al. (2019) Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. 2019\.  On the Spectral Bias of Neural Networks. In _Proceedings of the 36th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 97)_ , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 5301–5310.  https://proceedings.mlr.press/v97/rahaman19a.html
  * Rainer et al. (2022) Gilles Rainer, Adrien Bousseau, Tobias Ritschel, and George Drettakis. 2022.  Neural Precomputed Radiance Transfer.  _Computer Graphics Forum (Proceedings of the Eurographics conference)_ 41, 2 (April 2022).  http://www-sop.inria.fr/reves/Basilic/2022/RBRD22
  * Ramamoorthi (2009) Ravi Ramamoorthi. 2009\.  _Precomputation-Based Rendering_.  NOW Publishers Inc.
  * Ramamoorthi and Hanrahan (2001) Ravi Ramamoorthi and Pat Hanrahan. 2001.  An Efficient Representation for Irradiance Environment Maps. In _Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’01)_. Association for Computing Machinery, New York, NY, USA, 497–500.  https://doi.org/10.1145/383259.383317
  * Reiser et al. (2021) Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021\.  KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs. In _International Conference on Computer Vision (ICCV)_.
  * Ren et al. (2013) Peiran Ren, Jinpeng Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. 2013\.  Global Illumination with Radiance Regression Functions .  _ACM Transactions on Graphics (TOG) - SIGGRAPH 2013 Conference Proceedings_ 32 (July 2013).  https://www.microsoft.com/en-us/research/publication/global-illumination-radiance-regression-functions/
  * Sloan (2008) Peter-Pike Sloan. 2008\.  Stupid spherical harmonics (sh) tricks. In _Game developers conference_ , Vol. 9. 42.
  * Sloan et al. (2002) Peter-Pike Sloan, Jan Kautz, and John Snyder. 2002\.  Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Lighting Environments. In _Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques_ (San Antonio, Texas) _(SIGGRAPH ’02)_. Association for Computing Machinery, New York, NY, USA, 527–536.  https://doi.org/10.1145/566570.566612
  * Sloan et al. (2003) Peter-Pike J. Sloan, J. Hall, J. Hart, and John M. Snyder. 2003\.  Clustered principal components for precomputed radiance transfer.  _ACM SIGGRAPH 2003 Papers_ (2003).
  * Takikawa et al. (2022) Towaki Takikawa, Andrew Glassner, and Morgan McGuire. 2022\.  A Dataset and Explorer for 3D Signed Distance Functions.  _Journal of Computer Graphics Techniques (JCGT)_ 11, 2 (27 April 2022), 1–29.  http://jcgt.org/published/0011/02/01/
  * Takikawa et al. (2021) Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. 2021.  Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes.  (2021).
  * Tancik et al. (2020) Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. 2020.  Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains.  _NeurIPS_ (2020).
  * Turk (1990) Greg Turk. 1990\.  _Generating Random Points in Triangles_.  Academic Press Professional, Inc., USA, 24–28.
  * Wang and Ramamoorthi (2018) Jingwen Wang and Ravi Ramamoorthi. 2018.  Analytic Spherical Harmonic Coefficients for Polygonal Area Lights.  _ACM Trans. Graph._ 37, 4, Article 54 (July 2018), 11 pages.  https://doi.org/10.1145/3197517.3201291
  * Wu et al. (2020) Lifan Wu, Guangyan Cai, Shuang Zhao, and Ravi Ramamoorthi. 2020\.  Analytic Spherical Harmonic Gradients for Real-Time Rendering with Many Polygonal Area Lights.  _ACM Trans. Graph._ 39, 4, Article 134 (July 2020), 14 pages.  https://doi.org/10.1145/3386569.3392373
  * Zhang et al. (2021) Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron. 2021.  NeRFactor: Neural Factorization of Shape and Reflectance under an Unknown Illumination.  _ACM Trans. Graph._ 40, 6, Article 237 (dec 2021), 18 pages.  https://doi.org/10.1145/3478513.3480496