11institutetext: 1School of Electrical and Electronic Engineering, Yonsei
University
11email: {2015142131, nemotio, hydragon516, 2donghyung87<EMAIL_ADDRESS>
# SMURF: Continuous Dynamics for Motion-Deblurring Radiance Fields

Jungho Lee 11 Dogyoon Lee 11 Minhyeok Lee 11 Donghyeong Kim 11
Sangyoun Lee 11

###### Abstract

Neural radiance fields (NeRF) has attracted considerable attention for their
exceptional ability in synthesizing novel views with high fidelity. However,
the presence of motion blur, resulting from slight camera movements during
extended shutter exposures, poses a significant challenge, potentially
compromising the quality of the reconstructed 3D scenes. While recent studies
have addressed this issue, they do not consider the continuous dynamics of
camera movements during image acquisition, leading to inaccurate scene
reconstruction. Additionally, these methods are plagued by slow training and
rendering speed. To effectively handle these issues, we propose sequential
motion understanding radiance fields (SMURF), a novel approach that employs
neural ordinary differential equation (Neural-ODE) to model continuous camera
motion and leverages the explicit volumetric representation method for faster
training and robustness to motion-blurred input images. The core idea of the
SMURF is continuous motion blurring kernel (CMBK), a unique module designed to
model a continuous camera movements for processing blurry inputs. Our model,
rigorously evaluated against benchmark datasets, demonstrates state-of-the-art
performance both quantitatively and qualitatively. Source code will be
publicly available at https://github.com/Jho-Yonsei/SMURF.

###### Keywords:

Neural Rendering View Synthesis Motion Deblurring

##  1 Introduction

Reconstructing complex 3D scenes from 2D images of different views and re-
rendering novel view images represent a core problem in computer vision and
graphics, with significant applications in augmented reality (AR) and virtual
reality (VR). Over the past few years, photo-realistic novel view synthesis
has greatly advanced with the emergence of Neural Radiance Fields (NeRF) [33].
It takes 3D spatial coordinates and 2D viewing directions (i.e. 5D vector) as
inputs for mapping to the radiance and volume density, where the process is
parameterized through an implicit multi-layer perceptron (MLP) based model.
Exploiting classical volume rendering techniques [19], NeRF integrates the
output radiance and volume density along the emitted rays, making the volume
rendering process fully differentiable.

Generally, NeRF variants have reconstructed 3D scenes using well-captured,
noise-free images as inputs along with calibrated camera parameters. However,
images captured in real-world may include various types of noise (e.g. camera
motion blur, finite depth-of-field), which exert negative impacts on the
network to train the complex geometry of the scene. Recently, many works [31,
25, 56, 40, 26] have been conducted on NeRF that takes camera motion-blurred
images as input, which is caused by camera movement during the exposure time.
Deblur-NeRF [31] first proposes a method that models blurring kernel by
imitating the deconvolution method for blind image deblurring, to reconstruct
3D scenes from motion-blurred images and render sharp novel view images. Since
the inception of Deblur-NeRF, various methods [25, 56, 40, 26] for estimating
the blurring kernel have been proposed, but they have following limitations:
a) their blurring kernel does not continuously model the camera movement
during exposure time as shwon in Fig. 1. Since actual camera movement is
normally continuous, it can be represented as a non-linear continuous function
over time, and such continuous modeling allows for a more precise tracking of
the camera movement path, even when the movement is complex or irregular.
Previous works all use a method where the warped rays of the blurring kernel
are computed jointly, making the kernel for images with complex camera motion
imprecise. b) None of them overcome the tradeoff between performance, and the
speed of training and rendering. DP-NeRF [25] shows performance improvement
but suffers from slow training and rendering, whereas PDRF [40] shows fast
training but offers relatively lower performance as shown in Fig. 2.

Figure 1: Comparison of joint kernel generation and our sequential kernel
generation. A blurry image $I_{blur}$ is acquired as the camera moves over the
exposure time ($t_{0}\sim t_{\mathcal{N}}$), with images $I_{t}$ captured at
each camera pose being composited together. Previous methods generate warped
rays of the blurring kernel in a single step by passing through a
parameterized network $h_{p}$ without considering the temporal sequence of
camera motion. However, our approach iteratively estimates warped rays along
the sequential camera motion trajectory.

In this paper, we propose a sequential motion understanding radiance fields
(SMURF), which incorporates a novel continuous motion blur kernel (CMBK) for
modeling precise camera movements from blurry images. The CMBK estimates the
small change in pose regarding camera motion through the view information of
each scene and the 2D pixel location of the input ray. The output values of
the CMBK are not computed jointly as in previous studies but are designed to
be computed sequentially according to camera motion, as shown in Fig. 1.
Particularly, recognizing that the sequential camera movements share a single
continuous function, we employ a neural ordinary differential equation
(Neural-ODE) [7] to ensure that the sequentially computed camera movements
exhibits continuity. Specifically, to reflect the physics inherent in camera
motion, we apply continuous dynamics in the latent space to CMBK, and
transform the latent feature into changes in ray within the physical space.
Sequentially estimating the small pose change with a light shared function
implies that the CMBK does not overly depend on the learning parameters.
Additionally, we introduce two regularization strategies to prevent the
divergence of camera motion generated through CMBK. The first is residual
momentum, which ensures that the warped ray does not deviate significantly
from the previous ray, thereby preventing overfitting. The second is the
output suppression loss, which also serves as a regularization method to
ensure that the warped ray does not diverge significantly from the initial
ray.

Figure 2: Comparison of performance on synthetic scene datasets in terms of
training time. The performance comparison metrics are PSNR, SSIM, and LPIPS.

Our SMURF leverages explicit volumetric representation methods [18, 6, 34, 50]
as its backbone. Specifically, the tensor factorization-based approach,
Tensorial Radiance Fields (TensoRF) [6], facilitates compact and efficient 3D
scene reconstruction. Considering the fact that some parts may exhibit
significant blur while others are almost free of blur in a single image,
incomplete blurred information and complete sharp information merge into a few
adjacent voxels via blurring kernel and ensure a coherent 3D scene. Therefore,
we adopt the TensoRF as our backbone as it effectively mitigates the
uncertainty of information caused by motion blur and enables faster training
and high quality rendering compared to previous approaches. The outstanding
performance of SMURF relative to its fast speed is demonstrated in Fig. 2.
While there exists a deblurring approach [26] that uses voxel-based radiance
fields as a backbone and shares the common goal of fast training with ours, a
key difference lies in their use of Plenoxels [59] as the backbone.
Furthermore, while they adopt the different explicit representation method
from ours with another goal of efficient memory consumption, we diverge in our
objectives, as we aim to leverage the advantages of a 3D voxel-based method
when utilizing blurry images as input.

To demonstrate the effectiveness of the proposed SMURF both quantitatively and
qualitatively, we conduct extensive experiments on synthetic and real-world
scenes. Our experimental results elucidate the advantages of the CMBK in
comparison to previously presented blurring kernels.

Our main contributions are summarized as follows:

  * $\bullet$

We propose a continuous motion blur kernel (CMBK) to sequentially estimate
precise, continuous camera motion from blurry images.

  * $\bullet$

We propose two regularization strategies that guide the warped rays to prevent
divergence: residual momentum, and output suppression loss.

  * $\bullet$

We significantly alleviate the tradeoff between performance and training speed
exploiting CMBK and 3D tensor factorization-based representation.

  * $\bullet$

Our sequential motion understanding radiance fields (SMURF) exhibit higher
visual quality and quantitative performance compared to existing approaches.

##  2 Related Work

###  2.1 Neural Radiance Fields (NeRF)

The synthesis of photo-realistic novel views from images of different
viewpoints has attracted considerable attention with the advancement of NeRF
[33]. In particular, NeRF uses 3D spatial coordinates and 2D viewing
directions to map radiance and volume density, a process facilitated by
implicit neural representation (INR). Following the success of NeRF, various
sub-domains considering real-world scenarios have been explored. Many recent
studies have extensively applied NeRF across various fields, including dynamic
3D scene modeling [37, 28, 43, 54, 38, 27, 30, 61, 3], scene relighting [48,
2, 32, 41, 53], and 3D reconstruction [55, 51, 57]. Furthermore, recent
studies have applied NeRF for deblurring 3D scenes [31, 25, 40, 56], assuming
blurry input images from real-world conditions and aiming to produce clean
images. Additionally, due to the slow rendering caused by the costly MLP
layers required to evaluate each pixel’s density and color, some studies [23,
42, 36, 6, 34, 59, 50] design networks for faster training and rendering. To
address these challenges, various voxel-based explicit rendering methods [6,
34, 59, 50] have been proposed, offering competitive performance to NeRF while
significantly reducing training and rendering times. We adopt an explicit
representation method, TensoRF [6] as the backbone, which facilitates fast
training and achieves accurate and efficient novel-view rendering.

###  2.2 NeRFs for Blurry Images

Recently, several studies have been conducted to deblur 3D scenes and
synthesize clean novel view images using blurry input images from different
views. Deblur-NeRF [31] proposes, for the first time, an implicit blurring
kernel in 3D space inspired by blind deblurring methods for 2D vision [22, 17,
52, 35, 10]. This blurring kernel is intentionally trained close to the pixels
of the blurry image, and during rendering, a clean image is obtained without
the trained kernel. DP-NeRF [25] models a rigid kernel that assumes physical
scene priors when blurry images are captured through a camera, utilizing only
the scene’s view information regardless of pixel location. BAD-NeRF [56],
assuming a very short camera exposure time, designs a kernel that linearly
interpolates the camera motion trajectory in SE(3) space. However, these
approaches estimate the change in camera pose jointly, not modeling the fact
that light traces left by camera motion during exposure time appear
sequentially. Therefore, we aim to design a kernel where the elements of the
blurring kernel, namely the small changes in camera pose, are estimated
sequentially over time.

###  2.3 Neural Ordinary Differential Equations (Neural ODEs)

Neural-ODEs [7] are proposed to interpret neural networks within the framework
of ordinary differential equations, representing the underlying dynamics
inherent in hidden parameters. Neural-ODEs model a parameterized time-
continuous latent state and output a unique solution of the integral of
continuous dynamics, utilizing given initial values and various numerical
differential equation solvers (e.g. Euler’s method [12], Runge-Kutta method
[24], Dormand-Prince-Shampine method [11]). They are extensively used to
continuously connect the latent space embedded in videos [20, 39, 13, 16] or
model the continuous dynamics of irregularly sampled time-series data [44].
Inspired by the fact that camera motion is continuous, we exploit Neural-ODEs
to create a precise blurring kernel that models the hidden continuous dynamics
inherent in camera motion.

##  3 Method

In this section, we detail the optimization process of SMURF. In Sec. 3.1, we
explain the preliminaries about the 3D tensor factorization-based rendering
method and 3D scene blind deblurring for our methodology. Sec. 3.2 explains
how continuous dynamics are applied to our CMBK to generate warped rays. Sec.
3.3 discusses the objective function and the optimization process. The
framework of SMURF is illustrated in Fig. 3.

###  3.1 Preliminary

####  Tensorial Radiance Fields (TensoRF).

We follow the architecture of TensoRF [6], an explicit voxel-based volumetric
representation utilizing CANDECOMP/PARAFAC (CP) [4, 14] and block term
decomposition [9], which models an efficient view-dependent sparse voxel grid.
TensoRF optimizes two 3D grid tensors,
$\mathcal{G}_{\sigma},\mathcal{G}_{\mathbf{c}}\in\mathbb{R}^{FXYZ}$, for
estimating volume density and view-dependent appearance feature, employing the
vector-matrix (VM) decomposition that effectively extends CP decomposition.
Exploiting this method, the grid (3rd-order) tensor is decomposed into low-
rank 1D vectors and 2D matrices across three modes, reducing the space
complexity from $\mathcal{O}(n^{3})$ to $\mathcal{O}(n^{2})$ compared to
conventional explicit representations [59]. This grid is represented as
follows:

$\mathcal{G}=\sum_{r=1}^{R_{1}}\mathbf{b}_{r}^{F_{1}}\circ\mathbf{v}_{r}^{X}\circ\mathbf{M}_{r}^{YZ}+\sum_{r=1}^{R_{2}}\mathbf{b}_{r}^{F_{2}}\circ\mathbf{v}_{r}^{Y}\circ\mathbf{M}_{r}^{XZ}+\sum_{r=1}^{R_{3}}\mathbf{b}_{r}^{F_{3}}\circ\mathbf{v}_{r}^{Z}\circ\mathbf{M}_{r}^{XY},$
(1)

where $\circ$ denotes the outer product; $R_{1},R_{2}$, and $R_{3}$ indicates
the number of low-rank components. $\mathbf{b}^{F}\in\mathbb{R}^{F}$ and
$\mathbf{v}^{X}\in\mathbb{R}^{X}$are the vectors along the $F$ and $X$ axes in
each mode, and $\mathbf{M}^{YZ}\in\mathbb{R}^{YZ}$ is the matrix in the $XY$
plane for each mode. Once the grids are defined, the radiance field at a 3D
point $\mathbf{x}$ for computing volume density $\sigma$ and color
$\mathbf{c}$ is defined as follows:

$\sigma(\mathbf{x}),\mathbf{c}(\mathbf{x})=\mathcal{G}_{\sigma}(\mathbf{x}),\mathcal{S}(\mathcal{G}_{\mathbf{c}}(\mathbf{x}),d),$
(2)

where $\mathcal{S}$ denotes a parameterized shallow MLP that converts viewing
direction $d$ and appearance feature $\mathcal{G}_{\mathbf{c}}(\mathbf{x})$
into color. The features obtained from the grid,
$\mathcal{G}_{\sigma}(\mathbf{x})$ and $\mathcal{G}_{\mathbf{c}}(\mathbf{x})$,
are trilinearly interpolated from adjacent voxels.

To render an image for a given view, TensoRF uses a differentiable classic
volume rendering technique [19] with $\sigma$ and $\mathbf{c}$. The color
$\mathbf{c}_{p}$ for each pixel reached by ray $\mathbf{r}$ is computed as
follows:

$\mathbf{c}_{p}(\mathbf{r})=\sum_{i=1}^{N}T_{i}\left(1-\exp\left(-\sigma_{i}\delta_{i}\right)\right)\mathbf{c}_{i},~{}T_{i}=\exp\left(-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j}\right),$
(3)

where $T_{i}$ is the transmittance; $N$ and $\delta$ denote the number of
sampled points and the step size between adjacent samples on ray $\mathbf{r}$,
respectively.

Figure 3: Method overview of the SMURF. The CMBK encoder
$\mathcal{E}_{\theta}$ transforms the embedded 2D pixel location $\mathbf{p}$
of the initial ray and view index into a latent feature. This feature is
extended into an IVP with a parameterized derivative function $f_{\phi}$ in
the latent space. Then, it is sequentially solved using a Neural-ODE along
with given time $t$ and a chrono-view embedding function $\mathbf{\Psi}$,
obtaining latent features for all warped rays. These features are transformed
into changes of the ray (i.e., $\Delta\mathbf{p}$ and $\Delta\mathbf{o}$)
through a decoder $\mathcal{D}_{\xi}$, and we get the warped rays by applying
Eq. 11 to the initial ray. These rays are rendered into 2D pixel colors
through a 3D grid-based method, and a blurry color is acquired by summing up
the colors with weights from the decoder.

#### Blind Deblurring for 3D Scene.

For image blind deblurring [58, 5, 49], the blurring kernel $h$ is estimated
in a fully unsupervised manner, and this process is achieved by convolving $h$
with sharp images. Ma  _et al_. [31] apply this algorithm to NeRF, modeling an
adaptive kernel $h_{p}$ for each ray,

$\mathbf{c}_{blur}(\mathbf{r})=\mathbf{c}_{p}(\mathbf{r})*h_{p}(\mathbf{r}),$
(4)

where $\mathbf{c}_{blur}$ and $\mathbf{c}_{p}$ respectively denote the color
of the blurry pixel and the sharp pixel; $*$ indicates the convolution
operator.

The conventional image deblurring kernel takes a fixed grid of size $K\times
K$ centered around the location $p$. However, for 3D scene deblurring,
applying such a kernel based on NeRF requires computing $\mathbf{c}_{p}$ for
$K^{2}$ rays, which is inefficient in terms of memory and training time. To
optimize a kernel for the 3D scene, it is necessary to produce a sparse kernel
with fewer rays. Ma  _et al_. [31] designs the sprase kernel to acquire blurry
color, and for the temporally continuous camera motion, we extend the process
as follows:

$\mathbf{c}_{blur}(\mathbf{r})=\sum_{i=0}^{\mathcal{N}}w_{p}^{t_{i}}\mathbf{c}_{p}(\mathbf{r}^{{t}_{i}}),~{}\text{w.r.t.}~{}\sum_{i=0}^{\mathcal{N}}w_{p}^{t_{i}}=1,$
(5)

where $\mathcal{N}$ and $t_{i}$ respectively denote the number of warped rays
in camera motion and the instantaneous time during exposure; $w_{p}$ is the
corresponding weight at each ray’s location, and $\mathbf{r}^{t_{i}}$
represents the warped ray due to the camera movement. By setting the number of
$\mathcal{N}$ to be less than $K^{2}$, more efficient training is feasible. As
the warped rays are determined by parameterized learning, the blurring kernel
is deformable, not in a fixed grid manner.

###  3.2 Continuous Motion Blur Kernel

#### Continuous Latent Space Modeling for Camera Motion.

Camera-motion blur in images arises from camera movement during exposure time.
Such movement, often due to the motion of the hand holding the camera, allows
for the representation of camera pose changes as a temporally continuous
function, as shown in Fig. 1. Therefore, our goal is to model the continuous
movement of the camera. To assign this movement with continuity, we transform
the given information of the rays into latent features and apply them to a
Neural-ODE [7] to design a continuous latent space.

We embed information about the initial ray of camera motion into the latent
space using a parameterized encoder $\mathcal{E}_{\theta}$, utilizing the
scene’s view index $idx_{view}$ and 2D pixel location $\mathbf{p}$:

$\mathbf{z}(t_{0})=\mathcal{E}_{\theta}\left(\psi_{v}(idx_{view}),~{}\psi_{p}(\mathbf{p})\right),$
(6)

where $\mathbf{z}_{t_{0}}\in\mathbb{R}^{d}$ is the latent feature of initial
ray with hidden dimension $d$, and $\psi_{v}$ and $\psi_{p}$ denote embedding
functions for view and pixel information, respectively. Within a small step
limit of the latent feature $\mathbf{z}(t)$, the local continuous dynamic is
expressed as $\mathbf{z}(t+\epsilon)=\mathbf{z}(t)+\epsilon\cdot
d\mathbf{z}/dt$. To apply continuous dynamics to $\mathbf{z}(t)$, we model a
parameterized neural derivative function $f$ in the latent space. The
derivative of the continuous function is defined as follows:

$\frac{d\mathbf{z}(t)}{dt}=f(\mathbf{z}(t),t;\phi),$ (7)

where $\phi$ denotes the learnable parameters of $f$. With $\mathbf{z}(t_{0})$
and the derivative function $f$, we define an initial value problem (IVP), and
the features of subsequential rays in the latent space are obtained by the
integral of $f$ from $t_{0}$ to the desired time. This dynamic leads to the
format of ODE, and the process of obtaining latent features of the camera
motion trajectory using various ODE solvers [12, 24, 11] is expressed as
follows:

$\mathbf{z}\left(t_{n+1}\right)=\mathbf{z}\left(t_{n}\right)+\int_{t_{n}}^{t_{n+1}}f(\mathbf{z}(t),t;\phi)dt=\text{ODESolve}\left(\mathbf{z}(t_{n}),f,t_{n},t_{n+1},\phi\right),$
(8)

where $0\leq n<\mathcal{N}$, and $\mathcal{N}$ is the number of rays in the
camera motion. As a result of this approach, we obtain the latent features
$\mathbf{Z}\in\mathbb{R}^{\mathcal{N}\times d}$ for $\mathcal{N}$ rays.

Motion-blurred images all have different exposure times, but the given images
lack information about them. Assuming a common exposure time for all images
when obtaining latent features through the ODE solver may lead to suboptimal
local minima since all the images exhibit varying degrees of blur due to
different exposure times. Therefore, inspired by the difference in exposure
time for images of a single scene, we define a chrono-view embedding function
$\mathbf{\Psi}$ that simultaneously embeds given time $t$ and view index
$idx_{view}$. Then, $f$ is expressed as follows:

$f(\mathbf{z}(t),t;\phi)\rightarrow
f(\mathbf{z}(t),t,\mathbf{\Psi}(t;idx_{view});\phi).$ (9)

Exploiting $\mathbf{\Psi}$, when the conditions of $t_{n}$ and $idx_{view}$
are given to the ODE solver for the IVP, the subsequential latent feature is
defined as an unique solution by Picard’s existence theorem [29].

#### Motion-Blurring Kernel Generation.

The latent features $\mathbf{Z}$ of all rays on the camera motion trajectory,
obtained through continuous dynamics modeling, must be decoded into changes in
camera pose. We define a decoder $\mathcal{D}_{\xi}$, represented by a shallow
MLP parameterized by $\xi$, which outputs three components: the change in 2D
kernel location $\Delta\mathbf{p}$, the change in ray origin
$\Delta\mathbf{o}_{\mathbf{p}}$, and the corresponding weight $w_{p}$ as
defined in Eq. 5:

$(\Delta\mathbf{p}^{t},\Delta\mathbf{o}_{\mathbf{p}^{t}},w^{t}_{p})=\mathcal{D}_{\xi}(\mathbf{z}(t)).$
(10)

Then, $t$-th warped ray $\mathbf{r}^{t}$ is generated from the initial ray
$\mathbf{r}=\mathbf{o}+\tau\mathbf{d}$ by following equation:

$\mathbf{r}^{t}=(\mathbf{o}+\Delta{\mathbf{o}_{\mathbf{p}^{t}}})+\tau\mathbf{d}_{\mathbf{p}^{t}},~{}\mathbf{p}^{t}=\mathbf{p}+\Delta\mathbf{p}^{t},$
(11)

where $\mathbf{o}$ and $\mathbf{p}$ denote the ray origin and the 2D pixel
location of initial ray, respectively; $\mathbf{d}_{\mathbf{p}^{t}}$ stands
for the warped direction by $\mathbf{p}^{t}$. After acquiring sharp pixel
colors $\mathbf{c}_{p}(\mathbf{r}^{t})$ for all $\mathcal{N}$ warped rays with
inherent continuous camera movement, the blurry pixel color is computed using
Eq. 5.

#### Residual Momentum.

Latent features of the camera motion trajectory are predicted by CMBK and are
expected to be optimized sequentially. In this process, predicted origin and
direction of the ray $\mathbf{r}^{t}$ may diverge from the initial ray
$\mathbf{r}$, potentially leading to a suboptimal kernel. Ma  _et al_. [31]
prevents ray divergence by applying a hyperbolic tangent function to
$\Delta\mathbf{p}^{t}$ for regularization. However, such regularization is
inefficient due to the varying intensity of motion blur across all the images.
To address this issue, we apply a residual term to the latent derivative
function $f$, implementing regularization that ensures the predicted ray
$\mathbf{r}^{t_{i}}$ does not significantly deviate from the previous ray
$\mathbf{r}^{t_{i-1}}$:

$f_{\phi}(\mathbf{z}(t_{i-1}))\rightarrow\Lambda(f_{\phi}(\mathbf{z}(t_{i-1}))+\mathbf{z}(t_{i-1})),$
(12)

where the $\Lambda$ is a shallow MLP for regularization and
$f_{\phi}(\mathbf{z})$ is the simplified version of
$f(\mathbf{z},t,\mathbf{\Psi}(t;idx_{view});\phi)$. This approach prevents the
divergence of the camera trajectory, allowing rays to be warped regardless of
the motion blur intensity of the image. Note that while the proposed residual
momentum is implemented similarly to the methodology of residual connections
in ResNet [15], the underlying conceptual frameworks of the two approaches are
distinct.

#### Output Suppression Loss.

We encode the ray and the view information into the latent space using
$\mathcal{E}_{\theta}$, and decode it into changes of the ray in camera motion
trajectory using $\mathcal{D}_{\xi}$. In this process, since latent feature
the initial ray $\mathbf{z}(t_{0})$ serves as the initial value for the ODE,
there should be no change in the initial ray. Therefore, we apply an output
suppression loss as follows:

$\mathcal{L}_{supp}=\lambda_{supp}\left\|\mathcal{D}_{\xi}(\mathbf{z}(t_{0}))\right\|_{2},$
(13)

where $\lambda$ is the weight for the loss $\mathcal{L}_{supp}$. In this
process, the color weight $w_{p}^{t_{0}}$ is not included for the loss. This
concept is similar to the cycle consistency in CycleGAN [62]. Minimizing the
changes of initial ray to zero value, we ensure that not just the initial ray
but also the warped rays do not diverge, providing a regularization effect.

###  3.3 Optimization

Following Sec. 3.1, our goal is to minimize the difference between the blurry
pixel color $\hat{\mathbf{c}}_{blur}(\mathbf{r})$ obtained by the blurring
kernel and the ground truth pixel color of the motion-blurred image
$\mathbf{c}_{blur}(\mathbf{r})$. Hence, the photometric loss
$\mathcal{L}_{photo}$ is represented by
$\mathcal{L}_{photo}=\left\|\mathbf{c}_{blur}(\mathbf{r})-\hat{\mathbf{c}}_{blur}(\mathbf{r})\right\|_{2}^{2}$.
Furthermore, the parameterized 3D voxel grids $\mathcal{G}_{\sigma}$ and
$\mathcal{G}_{c}$ are regularized through the total variation [1] losses
$\mathcal{L}_{TV}^{\sigma}$ and $\mathcal{L}_{TV}^{c}$, respectively.
Additionally, to prevent divergence of the warped rays and ensure initial ray
consistency, Eq. 13 is applied, resulting in our combined objective as
follows:

$\mathcal{L}=\mathcal{L}_{photo}+\lambda_{TV}^{\sigma}\mathcal{L}_{TV}^{\sigma}+\lambda_{TV}^{c}\mathcal{L}_{TV}^{c}+\lambda_{supp}\mathcal{L}_{supp}$
(14)

where the $\lambda_{TV}^{\sigma}$ and the $\lambda_{TV}^{c}$ are the weights
for $\mathcal{L}_{TV}^{\sigma}$ and $\mathcal{L}_{TV}^{c}$, respectively.

##  4 Experiments

###  4.1 Implementation Details

#### Datasets.

In this experiment, we utilize the camera motion blur dataset published by
[31], which is divided into synthetic and real-world scenes. The synthetic
scene dataset comprises five scenes synthesized using Blender [8], with multi-
view cameras set up to render images with applied camera motion. The camera
motion trajectory is formed by linearly interpolating between the poses of the
first and last cameras, and the rendered multi-view images are combined in the
linear RGB space to obtain the final blurry images. The real-world scene
dataset consists of ten scenes captured with an actual camera. The blurry
images are obtained by physically shaking the camera during the exposure time,
and the camera poses are calculated using the images obtained with COLMAP [46,
45].

Methods | Synthetic Scene Dataset | Real-World Scene Dataset |  Training Time (h) |  Rendering Time (s)
---|---|---|---|---
PSNR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$) |  PSNR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$)
Naive NeRF [33] | 23.78 | 0.6807 | 0.3362 | 22.69 | 0.6347 | 0.3687 |  $\sim$ 6 |  $\sim$ 7
NeRF+MPR [60] | 25.11 | 0.7476 | 0.2148 | 23.38 | 0.6655 | 0.3140 |  $\sim$ 6 |  $\sim$ 7
NeRF+PVD [47] | 24.58 | 0.7190 | 0.2475 | 23.10 | 0.6389 | 0.3425 |  $\sim$ 6 |  $\sim$ 7
Deblur-NeRF [31] | 28.77 | 0.8593 | 0.1400 | 25.63 | 0.7645 | 0.1820 |  $\sim$ 25 |  $\sim$ 7
PDRF-10* [40] | 28.86 | 0.8795 | 0.1139 | 25.90 | 0.7734 | 0.1825 | 2.9 | 0.91
BAD-NeRF* [56] | 27.32 | 0.8178 | 0.1127 | 22.82 | 0.6315 | 0.2887 |  $\sim$ 20 |  $\sim$ 7
DP-NeRF [25] | 29.23 | 0.8674 | 0.1184 | 25.91 | 0.7751 | 0.1602 |  $\sim$ 26 |  $\sim$ 7
SMURF | 30.98 | 0.9147 | 0.0609 | 26.52 | 0.7986 | 0.1013 | 1.7 | 0.77

Table 1: Quantitative comparisons on synthetic and real-world scene dataset.
We evaluate the quantitative performance using three metrics, demonstrating
that our proposed model significantly ourperforms existing ones even with
faster training and rendering times. The red and orange cells respectively
indicate the highest and second-highest value. Figure 4: Qualitative
comparisons on synthetic scenes and real-world scenes. Without long training
times and sharp images, SMURF produces results most similar to the reference
images. SMURF models the detailed aspects that are not captured by previous
methods with long training time.

#### Training Details.

We implement our model on TensoRF [6], an explicit grid-based method, as our
backbone renderer. We upsample the voxel counts of grids
$\mathcal{G}_{\sigma}$ and $\mathcal{G}_{c}$ from $128^{3}$ to $480^{3}$, and
set the feature component $F$ of the grids to 36 and 96, respectively. To
implement the CMBK, we set the number of warped rays $\mathcal{N}$ on the
camera motion trajectory to 8 and adopt the Euler method [12] as the solver
for the Neural-ODE. Single scene training is conducted for 40k iterations,
with the learning rates for the grids decaying from $0.02$ to $0.002$, and the
learning rate for CMBK decaying from $0.001$ to $0.0001$. The weights for the
losses $\lambda_{TV}^{\sigma}$, $\lambda_{TV}^{c}$, and $\lambda_{supp}$ are
all set to 0.1. All our experiments are conducted on a single NVIDIA RTX 3090.

#### Evaluation Metrics.

The images rendered through the SMURF are quantitatively evaluated using three
metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index
Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS).

Synthetic Scene | Factory | CozyRoom | Pool | Tanabata | Trolley
---|---|---|---|---|---
PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS
Naive NeRF | 19.32 | 0.4563 | 0.5304 | 25.66 | 0.7941 | 0.2288 | 30.45 | 0.8354 | 0.1932 | 22.22 | 0.6807 | 0.3653 | 21.25 | 0.6370 | 0.3633
MPR+NeRF | 21.70 | 0.6153 | 0.3094 | 27.88 | 0.8502 | 0.1153 | 30.64 | 0.8385 | 0.1641 | 22.71 | 0.7199 | 0.2509 | 22.64 | 0.7141 | 0.2344
PVD+NeRF | 20.33 | 0.5386 | 0.3667 | 27.74 | 0.8296 | 0.1451 | 27.56 | 0.7626 | 0.2148 | 23.44 | 0.7293 | 0.2542 | 23.81 | 0.7351 | 0.2567
Deblur-NeRF | 25.60 | 0.7750 | 0.2687 | 32.08 | 0.9261 | 0.0477 | 31.61 | 0.8682 | 0.1246 | 27.11 | 0.8640 | 0.1228 | 27.45 | 0.8632 | 0.1363
PDRF-10* | 25.87 | 0.8316 | 0.1915 | 31.13 | 0.9225 | 0.0439 | 31.00 | 0.8583 | 0.1408 | 28.01 | 0.8931 | 0.1004 | 28.29 | 0.8921 | 0.0931
BAD-NeRF* | 24.43 | 0.7274 | 0.2134 | 29.77 | 0.8864 | 0.0616 | 31.51 | 0.8620 | 0.0802 | 25.32 | 0.8081 | 0.1077 | 25.58 | 0.8049 | 0.1008
DP-NeRF | 25.91 | 0.7787 | 0.2494 | 32.65 | 0.9317 | 0.0355 | 31.96 | 0.8768 | 0.0908 | 27.61 | 0.8748 | 0.1033 | 28.03 | 0.8752 | 0.1129
SMURF | 29.87 | 0.8958 | 0.1057 | 32.48 | 0.9285 | 0.0379 | 32.34 | 0.8884 | 0.0779 | 29.91 | 0.9300 | 0.0436 | 30.30 | 0.9307 | 0.0397

Table 2: Comparison of performance for individual scenes on the synthetic
dataset. SMURF exhibits higher performance across all scenes, with the
exception of “CozyRoom,” where it shows slightly lower performance relative to
others.

Real-World Scene | Ball | Basket | Buick | Coffee | Decoration
---|---|---|---|---|---
PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS
Naive NeRF | 24.08 | 0.6237 | 0.3992 | 23.72 | 0.7086 | 0.3223 | 21.59 | 0.6325 | 0.3502 | 26.48 | 0.8064 | 0.2896 | 22.39 | 0.6609 | 0.3633
Deblur-NeRF | 27.36 | 0.7656 | 0.2230 | 27.67 | 0.8449 | 0.1481 | 24.77 | 0.7700 | 0.1752 | 30.93 | 0.8981 | 0.1244 | 24.19 | 0.7707 | 0.1862
PDRF-10* | 27.37 | 0.7642 | 0.2093 | 28.36 | 0.8736 | 0.1179 | 25.73 | 0.7916 | 0.1582 | 31.79 | 0.9002 | 0.1133 | 23.55 | 0.7508 | 0.2145
BAD-NeRF* | 21.33 | 0.5096 | 0.4692 | 26.44 | 0.8080 | 0.1325 | 21.63 | 0.6429 | 0.2593 | 28.98 | 0.8369 | 0.1956 | 22.13 | 0.6316 | 0.2894
DP-NeRF | 27.20 | 0.7652 | 0.2088 | 27.74 | 0.8455 | 0.1294 | 25.70 | 0.7922 | 0.1405 | 31.19 | 0.9049 | 0.1002 | 24.31 | 0.7811 | 0.1639
SMURF | 27.50 | 0.7760 | 0.1298 | 28.95 | 0.8842 | 0.0619 | 27.10 | 0.8409 | 0.0839 | 31.33 | 0.8879 | 0.0874 | 24.90 | 0.8114 | 0.1190
Real-World Scene | Girl | Heron | Parterre | Puppet | Stair
PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS
Naive NeRF | 20.07 | 0.7075 | 0.3196 | 20.50 | 0.5217 | 0.4129 | 23.14 | 0.6201 | 0.4046 | 22.09 | 0.6093 | 0.3389 | 22.87 | 0.4561 | 0.4868
Deblur-NeRF | 22.27 | 0.7976 | 0.1687 | 22.63 | 0.6874 | 0.2099 | 25.82 | 0.7597 | 0.2161 | 25.24 | 0.7510 | 0.1577 | 25.39 | 0.6296 | 0.2102
PDRF-10* | 24.12 | 0.8328 | 0.1679 | 22.53 | 0.6880 | 0.2358 | 25.36 | 0.7601 | 0.2263 | 25.02 | 0.7496 | 0.1532 | 25.20 | 0.6235 | 0.2288
BAD-NeRF* | 18.10 | 0.5652 | 0.3933 | 22.18 | 0.6479 | 0.2226 | 23.44 | 0.6243 | 0.3151 | 22.48 | 0.6249 | 0.2762 | 21.52 | 0.4237 | 0.3341
DP-NeRF | 23.33 | 0.8139 | 0.1498 | 22.88 | 0.6930 | 0.1914 | 25.86 | 0.7665 | 0.1900 | 25.25 | 0.7536 | 0.1505 | 25.59 | 0.6349 | 0.1772
SMURF | 25.66 | 0.8592 | 0.0829 | 23.59 | 0.7317 | 0.1381 | 25.47 | 0.7825 | 0.1207 | 25.19 | 0.7702 | 0.1077 | 25.48 | 0.6421 | 0.0822

Table 3: Comparison of performance for individual scenes on the real-world
dataset. While SMURF shows slightly lower PSNR and SSIM in some scenes, it
shows significantly higher LPIPS across all scenes compared to previous
methods.

###  4.2 Novel View Synthesis Results

#### Quantitative Results.

We show the quantitative evaluation of our network, SMURF, in comparison to
various baselines on the two benchmark datasets proposed by Deblur-NeRF [31],
as shown in Tab. 1. NeRF+MPR and NeRF+PVD are trained with a naive NeRF, where
the single-image deblurring methods MPR [60] and PVD [47] to the input data.
Additionally, since PDRF-10 [40] does not specify LPIPS in their experiment,
we obtain performance of three evaluation metrics using their released code.
Furthermore, as BAD-NeRF [56] specifies results from experiments on a modified
benchmark dataset, for a fair evaluation, we apply the released code of BAD-
NeRF to the benchmark dataset. According to Tab. 1, despite significantly
reduced training and rendering times compared to previous methods, we
demonstrate superior quantitative performance across all the metrics.
Especially, LPIPS of SMURF demonstrates SMURF’s exceptional perceptual quality
across all datasets. Tab. 2 and Tab. 3 detail the individual performance of
scenes within the synthetic and real-world datasets, respectively. For
synthetic scenes, except for “Cozyroom,” all scenes show quantitatively high
performance, with this scene also displaying no significant difference when
compared to DP-NeRF. Additionally, for the real-world scenes, despite a few
scenes exhibiting somewhat lower PSNR, they demonstrate better perceptual
quality through superior LPIPS scores.

#### Qualitative Results.

We validate the effectiveness of SMURF’s quantitatively high performance
through qualitative evaluation via novel view rendering. According to Fig. 4,
we compare our results with previous 3D scene deblurring methods for two
synthetic scenes (“Factory” and “Tanabata”) and three real-world scenes
(“Girl”, “Buick”, and “Basket”). For the “Factory” scene, when comparing the
perceptual quality of the reconstructed lowest part of the stairs, it is
evident that BAD-NeRF [56] and SMURF best estimate the 3D geometry. However,
BAD-NeRF shows an inability to capture the overall color as accurately as
other methods. In the rendering results for the “Tanabata” scene, while other
methods fail to accurately estimate the position of the power lines, SMURF
restores them most similarly to the reference image. In the “Girl” and “Buick”
scenes, our method notably restores the visual quality of the shelf’s black
support rods and the car logo, respectively, most closely to the reference
images. For the “Basket” scene, the holes of the basket further demonstrate
the superior qualitative outcome of our results. For additional novel view
rendering results, refer to our Appendix.

###  4.3 Ablation Study

#### Chrono-View Embedding Function.

To demonstrate the effectiveness of the chrono-view embedding function
$\mathbf{\Psi}$ in making continuous latent space modeling robust to exposure
time, as discussed in Sec. 3, we conduct various ablative experiments. We
conduct experiments by dividing the embedding types into three categories. The
time embedding assumes that all images have the same exposure time without
incorporating view information, while the chrono-time embedding applies view
information, ensuring that all images have distinct exposure time information.
As shown in Tab. 4, the application of time embedding yields higher
performance in general, but chrono-time embedding shows higher performance
only in real-world scenes, while its effect in synthetic scenes is minimal.
This is attributed to the characteristics of the synthetic dataset created by
Deblur-NeRF [31] using Blender [8], where motion blur images are acquired by
linearly interpolating between camera poses to synthesize a total of 10
images. In other words, the synthetic scenes set a constant speed for camera
motion throughout the exposure time for all images, implying that all input
images share the same exposure time information. Our chrono-view embedding
function assumes that all images have varying exposure times, leading to its
minimal effect on the synthetic dataset. However, the real-world dataset
consists of images captured with an actual camera, where exposure times are
likely to vary unless shutter speed is fixed, and the speed of camera motion
is not constant during the exposure time. Therefore, applying this function to
the real-world scenes yield improved performance due to these variations.

Methods |  Embedding | Synthetic Scene Dataset | Real-World Scene Dataset
---|---|---|---
Type |  PSNR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$) |  PSNR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$)
SMURF | None | 30.54 | 0.8995 | 0.0709 | 25.74 | 0.7755 | 0.1194
SMURF | Time | 30.94 | 0.9158 | 0.0610 | 26.34 | 0.7881 | 0.1102
SMURF | Chrono-View | 30.98 | 0.9147 | 0.0609 | 26.52 | 0.7986 | 0.1013

Table 4: Ablations on chrono-view embedding function. We conduct an ablation
study by dividing the embedding types into three categories: no embedding,
time embedding, and chrono-time embedding.

#### Regularzation Strategies.

To validate the effectiveness of the regularization strategies presented in
Sec. 3, namely output suppression loss and residual momentum, we conduct
various experiments as shown in Tab. 5. The output suppression loss is
designed to suppress changes in the origin and direction of the initial ray on
the camera motion trajectory to zero, yielding higher performance when
applied. This is because the changes in rays are prevented from diverging
since the estimated poses also share the same derivative function $f$, similar
to the effect of the alignment loss proposed by Deblur-NeRF [31]. This
similarity suggests that aligning ray origins is robust to inaccuracies in
camera pose, which can be ambiguous due to motion blur. Consequently, Tab. 5
demonstrates that the performance of PSNR and SSIM in synthetic and real-world
scenes improves more significantly.

Methods |  O.S. Loss | Residual | Synthetic Scene Dataset | Real-World Scene Dataset
---|---|---|---|---
Momentum |  PSNR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$) |  PSNR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$)
SMURF |  |  | 26.17 | 0.8186 | 0.0759 | 25.27 | 0.7576 | 0.1121
SMURF |  | ✓ | 28.49 | 0.8727 | 0.0675 | 25.60 | 0.7748 | 0.1057
SMURF | ✓ |  | 30.72 | 0.9052 | 0.0684 | 25.99 | 0.7846 | 0.1181
SMURF | ✓ | ✓ | 30.98 | 0.9147 | 0.0609 | 26.52 | 0.7986 | 0.1013

Table 5: Ablations on regularization strategies. We conduct ablative
experiments on the two proposed normalization methods. “O.S. Loss” refers to
the output suppression loss.

Residual momentum offers a similar effect to output suppression loss. While
the change in camera poses is estimated through a parameterized differential
equation, the shape of the derivative obtained in an unsupervised manner could
diverge. Therefore, we use residual momentum to ensure the next pose does not
deviate significantly from the previous one. Tab. 5 shows that applying
another regularization strategy, residual momentum, shows better performance
in terms of LPIPS than output suppression loss. Combining two strategies, the
SMURF results in best performance across all metrics.

##  5 Conclusion

We have proposed SMURF, a novel approach that sequentially models accurate
camera motion for reconstructing sharp 3D scenes from motion-blurred images.
Unlike previous approaches that estimate camera motion in a single step, SMURF
incorporates, for the first time, a kernel for estimating sequential camera
motions, named the CMBK. This camera motion is represented with continuity by
solving continuous dynamics in the latent space using Neural-ODEs. To prevent
the divergence of rays estimated by CMBK beyond the motion blur range, we
apply regularization techniques: residual momentum and output suppression
loss. Furthermore, we model the 3D scene using tensor factorization-based
representation, which allows for the integration of incomplete blurred
information and complete sharp information within adjacent voxels via CMBK,
thereby reducing the uncertainty of blurry information. SMURF significantly
outperform previous works quantitatively with faster training and rendering,
and its qualitative evaluation is demonstrated through novel view rendering
results.

Appendix

##  Appendix A Details of CMBK

Figure 5: Detailed process for generating kernel. To highlight the continuous
dynamics in the latent space, all embedding functions are omitted from the
figure. $\mathbf{p}(x_{t_{0}})$ and $\mathbf{p}(y_{t_{0}})$ correspond to the
$x$ and $y$ coordinates of the pixel associated with the initial ray,
respectively.

We elaborate on the continuous dynamics of the proposed CMBK as shown in Fig.
5. We assume that camera motion encompasses inherent dynamics with a unique
solution. The assumed solution is represented by the lighter circles in the
left plots of Fig. 5 (b). Rather than implementing the inherent dynamics in a
simple physical space, we refine them within a latent space with parametric
learning. The continuous dynamics of CMBK involve transforming the pixel
coordinates corresponding to the ray into latent features via a parameterized
encoder $\mathcal{E}_{\theta}$, and a unique numerical solution [29] is
obtained by solving the initial value problem on the formed latent space
through a neural ordinary differential equation [7]. The solution obtained
represents the change in pixel of the warped ray relative to the initial ray’s
pixel. These changes define the pixels corresponding to the warped rays, and
we specify Eq. (11) from the main paper:

$\begin{array}[]{ccc}\vspace{+1mm}\mathbf{p}^{t_{i}}=\left(\mathbf{p}(x_{t_{i}}),~{}\mathbf{p}(y_{t_{i}})\right),\\\
\vspace{+1mm}\mathbf{p}(x_{t_{i+1}})=\mathbf{p}(x_{t_{i}})+\Delta\mathbf{p}(x_{t_{i+1}}),\\\
\mathbf{p}(y_{t_{i+1}})=\mathbf{p}(y_{t_{i}})+\Delta\mathbf{p}(y_{t_{i+1}}).\end{array}$
(15)

Following this principle, the change in the initial ray must necessarily be
zero, and to ensure this, the proposed method is the output suppression loss.

##  Appendix B Number of Warped Rays

Methods |  $\mathcal{N}$ | Synthetic Scene Dataset | Real-World Scene Dataset |  Training Time (h)
---|---|---|---|---
PSNR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$) |  PSNR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$)
SMURF | 4 | 29.52 | 0.8829 | 0.1014 | 25.81 | 0.7710 | 0.1388 | 1.16
SMURF | 5 | 30.30 | 0.9013 | 0.0926 | 25.91 | 0.7811 | 0.1253 | 1.27
SMURF | 6 | 29.85 | 0.8929 | 0.0851 | 25.98 | 0.7822 | 0.1155 | 1.43
SMURF | 7 | 30.23 | 0.8964 | 0.0751 | 26.40 | 0.7944 | 0.1052 | 1.56
SMURF | 8 | 30.98 | 0.9147 | 0.0609 | 26.52 | 0.7986 | 0.1013 | 1.72
SMURF | 9 | 30.41 | 0.9086 | 0.0575 | 26.24 | 0.7922 | 0.1021 | 1.88

Table 6: Performance and training time of SMURF according to the number of
warped rays. The red, orange, and yellow cells respectively indicate the
highest, second-highest, and third-highest value.

We conduct extensive experiments to analyze the performance of the proposed
CMBK based on the number of warped rays, $\mathcal{N}$. As shown in Tab. 6,
larger $\mathcal{N}$ requires the more pixels to be rendered, resulting in an
almost linear increase in training time. Moreover, Fig. 6 shows the
performance according to the number of warped rays. Across all datasets, an
increase in the number of warped rays tends to improve the PSNR and SSIM
metrics. LPIPS noticeably decreases with more rays, interpreting that a higher
number of warped rays ensures better perceptual quality. The performance for
individual scenes in the synthetic dataset is shown in Tab. 7, showing that
except for the “Cozyroom” scene, LPIPS decreases and overall performance peaks
when $\mathcal{N}$ is 9. Analysis for the “Cozyroom” scene is conducted in
Appendix D. As indicated in Tab. 8, for the real-world dataset, a larger
$\mathcal{N}$ generally guarantees higher performance across most scenes.
However, there are scenes where performance drops when $\mathcal{N}$ exceeds
9, suggesting that the optimal $\mathcal{N}$ might be 9. Even with
$\mathcal{N}$ set to 5, which is the same condition to DP-NeRF [25] and
Deblur-NeRF [31], SMURF outperforms them across all metrics. Furthermore,
SMURF achieves higher performance with fewer warped rays than PDRF-10 [40],
which set $\mathcal{N}$ at 10, validating the effectiveness of our proposed
ideas.

Figure 6: Variation in performance with the number of warped rays.

Synthetic |  $\mathcal{N}$ | Factory | CozyRoom | Pool | Tanabata | Trolley
---|---|---|---|---|---|---
PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS
SMURF | 4 | 25.13 | 0.7697 | 0.2127 | 32.46 | 0.9294 | 0.0407 | 33.04 | 0.8993 | 0.0825 | 28.38 | 0.9059 | 0.0825 | 28.57 | 0.9101 | 0.0890
SMURF | 5 | 27.33 | 0.8381 | 0.2075 | 32.72 | 0.9315 | 0.0375 | 32.47 | 0.8904 | 0.0797 | 29.34 | 0.9222 | 0.0631 | 29.65 | 0.9241 | 0.0755
SMURF | 6 | 26.74 | 0.8159 | 0.1897 | 31.74 | 0.9246 | 0.0381 | 32.56 | 0.8922 | 0.0778 | 29.22 | 0.9179 | 0.0604 | 28.99 | 0.9139 | 0.0599
SMURF | 7 | 26.15 | 0.8010 | 0.1679 | 32.12 | 0.9269 | 0.0378 | 32.59 | 0.8926 | 0.0773 | 29.69 | 0.9269 | 0.0490 | 30.58 | 0.9348 | 0.0438
SMURF | 8 | 29.87 | 0.8958 | 0.1057 | 32.48 | 0.9285 | 0.0379 | 32.34 | 0.8884 | 0.0779 | 29.91 | 0.9300 | 0.0436 | 30.30 | 0.9307 | 0.0397
SMURF | 9 | 30.52 | 0.9065 | 0.0807 | 31.43 | 0.9198 | 0.0428 | 32.10 | 0.8849 | 0.0771 | 28.93 | 0.9184 | 0.0448 | 29.05 | 0.9136 | 0.0423

Table 7: Performance and training time of SMURF for individual scenes in the
synthetic dataset, according to the number of warped rays.

Real-World |  $\mathcal{N}$ | Ball | Basket | Buick | Coffee | Decoration
---|---|---|---|---|---|---
PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS
SMURF | 4 | 26.47 | 0.7424 | 0.1754 | 28.30 | 0.8653 | 0.0847 | 26.57 | 0.8270 | 0.1066 | 30.66 | 0.8676 | 0.1228 | 24.59 | 0.7972 | 0.1471
SMURF | 5 | 26.63 | 0.7528 | 0.1549 | 27.96 | 0.8648 | 0.0741 | 26.03 | 0.8254 | 0.0947 | 30.93 | 0.8734 | 0.1172 | 24.22 | 0.7799 | 0.1497
SMURF | 6 | 27.31 | 0.7678 | 0.1434 | 27.19 | 0.8463 | 0.0773 | 26.75 | 0.8315 | 0.0905 | 30.76 | 0.8731 | 0.1004 | 24.16 | 0.7744 | 0.1465
SMURF | 7 | 27.68 | 0.7821 | 0.1325 | 29.24 | 0.8862 | 0.0617 | 27.05 | 0.8395 | 0.0867 | 30.67 | 0.8649 | 0.0931 | 24.97 | 0.8092 | 0.1220
SMURF | 8 | 27.50 | 0.7760 | 0.1298 | 28.95 | 0.8842 | 0.0619 | 27.10 | 0.8409 | 0.0839 | 31.33 | 0.8879 | 0.0874 | 24.90 | 0.8114 | 0.1190
SMURF | 9 | 27.16 | 0.7698 | 0.1315 | 28.52 | 0.8766 | 0.0631 | 26.92 | 0.8366 | 0.0813 | 31.41 | 0.8802 | 0.0870 | 24.12 | 0.7753 | 0.1405
Real-World |  $\mathcal{N}$ | Girl | Heron | Parterre | Puppet | Stair
PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS |  PSNR |  SSIM |  LPIPS
SMURF | 4 | 24.66 | 0.8410 | 0.1084 | 23.36 | 0.7144 | 0.1863 | 25.43 | 0.7644 | 0.1611 | 24.63 | 0.7496 | 0.1277 | 23.38 | 0.5406 | 0.1687
SMURF | 5 | 24.73 | 0.8354 | 0.1028 | 23.47 | 0.7244 | 0.1717 | 25.22 | 0.7640 | 0.1474 | 24.69 | 0.7514 | 0.1239 | 25.22 | 0.6391 | 0.1171
SMURF | 6 | 25.10 | 0.8473 | 0.0940 | 23.60 | 0.7386 | 0.1533 | 24.99 | 0.7569 | 0.1402 | 24.51 | 0.7493 | 0.1145 | 25.38 | 0.6371 | 0.0956
SMURF | 7 | 25.43 | 0.8570 | 0.0884 | 23.66 | 0.7352 | 0.1457 | 25.22 | 0.7744 | 0.1302 | 24.70 | 0.7587 | 0.1076 | 25.42 | 0.6370 | 0.0846
SMURF | 8 | 25.66 | 0.8592 | 0.0829 | 23.59 | 0.7317 | 0.1381 | 25.47 | 0.7825 | 0.1207 | 25.19 | 0.7702 | 0.1077 | 25.48 | 0.6421 | 0.0822
SMURF | 9 | 25.56 | 0.8567 | 0.0822 | 23.81 | 0.7828 | 0.1333 | 24.91 | 0.7583 | 0.1216 | 24.71 | 0.7554 | 0.1035 | 25.31 | 0.6302 | 0.0777

Table 8: Performance and training time of SMURF for individual scenes in the
real-world dataset, according to the number of warped rays.

##  Appendix C Additional Rendering Results

Additional rendering results are shown in Fig. 8 (for the synthetic dataset)
and Figs. 9 and 10 (for the real-world dataset), demonstrating that our SMURF
qualitatively offers the best quality when compared to the reference images.

##  Appendix D Analysis for Low-PSNR Scenes

Figure 7: Qualitative comparison of rendering results and error maps for the
Cozyroom scene. DP-TensoRF is a model that applies the kernel proposed by DP-
NeRF [25] to TensoRF [6].

In this section, we analyze individual scenes that showed slightly lower
performance from the main paper . Notably, we present the error maps for the
“Cozyroom” scene from the synthetic dataset for DP-NeRF [25], DP-TensoRF [25,
6], and SMURF in Fig. 7, where DP-TensoRF is a model that applies the blurring
kernel proposed by DP-NeRF to TensoRF. The error map for DP-NeRF shows bright
wall rendered cleanly without noise, whereas DP-TensoRF and SMURF exhibit
noise on the wall. This indicates that the cause of noise is not the proposed
CMBK but the inherent characteristics of the backbone model, TensoRF. As shown
in Fig. 8, aside from the noise on the wall, the rendering results of SMURF
show slightly sharper image quality in other areas except the wall. Moreover,
the “Coffee”, “Parterre”, and “Puppet" scenes from real-world dataset, SMURF
shows the best LPIPS score but somewhat lower PSNR. However, when comparing
the rendering results in Figs. 9 and 10, it is observable that the results of
SMURF are most similar to the reference images for these scenes.

##  Appendix E Limitations and Future Work

By adopting TensoRF [6], a 3D tensor factorization-based method, as our
backbone, we ensure high quantitative performance, superior perceptual
quality, and faster training. However, with the advent of 3D Gaussian
Splatting [21], which allows for GPU-based rasterization instead of optimizing
per ray, backbones that guarantee more faster training and rendering become
available. Although our backbone may be slower compared to 3D Gaussian
Splatting, applying the main idea of CMBK, continuous dynamics, to a
rasterization-based method is expected to result in faster training and
rendering. Furthermore, by demonstrating the applicability of continuous
dynamics to the 3D scene deblurring, we anticipate the possibility of
designing models that cover not only camera motion blur but also object motion
blur, which is caused by the movement of objects within the scene.

Figure 8: Qualitative comparison for individual scenes in the synthetic
dataset. Figure 9: Qualitative comparison for individual scenes in the real-
world dataset. Figure 10: Qualitative comparison for individual scenes in the
real-world dataset.

## References

  * [1] Beck, A., Teboulle, M.: Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE transactions on image processing 18(11), 2419–2434 (2009)
  * [2] Bi, S., Xu, Z., Srinivasan, P., Mildenhall, B., Sunkavalli, K., Hašan, M., Hold-Geoffroy, Y., Kriegman, D., Ramamoorthi, R.: Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824 (2020)
  * [3] Cao, A., Johnson, J.: Hexplane: A fast representation for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 130–141 (2023)
  * [4] Carroll, J.D., Chang, J.J.: Analysis of individual differences in multidimensional scaling via an n-way generalization of “eckart-young” decomposition. Psychometrika 35(3), 283–319 (1970)
  * [5] Chakrabarti, A.: A neural approach to blind motion deblurring. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14. pp. 221–235. Springer (2016)
  * [6] Chen, A., Xu, Z., Geiger, A., Yu, J., Su, H.: Tensorf: Tensorial radiance fields. In: European Conference on Computer Vision. pp. 333–350. Springer (2022)
  * [7] Chen, R.T., Rubanova, Y., Bettencourt, J., Duvenaud, D.K.: Neural ordinary differential equations. Advances in neural information processing systems 31 (2018)
  * [8] Community, B.O.: Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam (2018), http://www.blender.org
  * [9] De Lathauwer, L.: Decompositions of a higher-order tensor in block terms—part ii: Definitions and uniqueness. SIAM Journal on Matrix Analysis and Applications 30(3), 1033–1066 (2008)
  * [10] Deconvolution, B.I.: Blind image deconvolution: Theory and applications. USA, FL, Boca Raton: CRC (2007)
  * [11] Dormand, J.R., Prince, P.J.: A family of embedded runge-kutta formulae. Journal of computational and applied mathematics 6(1), 19–26 (1980)
  * [12] Euler, L.: Institutionum calculi integralis, vol. 4. impensis Academiae imperialis scientiarum (1845)
  * [13] Fernandes, S., Raj, S., Ortiz, E., Vintila, I., Salter, M., Urosevic, G., Jha, S.: Predicting heart rate variations of deepfake videos using neural ode. In: Proceedings of the IEEE/CVF international conference on computer vision workshops. pp. 0–0 (2019)
  * [14] Harshman, R.A., et al.: Foundations of the parafac procedure: Models and conditions for an" explanatory" multimodal factor analysis (1970)
  * [15] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [16] Hofherr, F., Koestler, L., Bernard, F., Cremers, D.: Neural implicit representations for physical parameter inference from a single video. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2093–2103 (2023)
  * [17] Joshi, N., Zitnick, C.L., Szeliski, R., Kriegman, D.J.: Image deblurring and denoising using color priors. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 1550–1557. IEEE (2009)
  * [18] Jun-Seong, K., Yu-Ji, K., Ye-Bin, M., Oh, T.H.: Hdr-plenoxels: Self-calibrating high dynamic range radiance fields. arXiv preprint arXiv:2208.06787 (2022)
  * [19] Kajiya, J.T., Von Herzen, B.P.: cing volume densities. ACM SIGGRAPH computer graphics 18(3), 165–174 (1984)
  * [20] Kanaa, D., Voleti, V., Kahou, S.E., Pal, C.: Simple video generation using neural odes. arXiv preprint arXiv:2109.03292 (2021)
  * [21] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4), 1–14 (2023)
  * [22] Krishnan, D., Tay, T., Fergus, R.: Blind deconvolution using a normalized sparsity measure. In: CVPR 2011. pp. 233–240. IEEE (2011)
  * [23] Kurz, A., Neff, T., Lv, Z., Zollhöfer, M., Steinberger, M.: Adanerf: Adaptive sampling for real-time rendering of neural radiance fields. In: European Conference on Computer Vision. pp. 254–270. Springer (2022)
  * [24] Kutta, W.: Beitrag zur näherungsweisen Integration totaler Differentialgleichungen. Teubner (1901)
  * [25] Lee, D., Lee, M., Shin, C., Lee, S.: Dp-nerf: Deblurred neural radiance field with physical scene priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12386–12396 (2023)
  * [26] Lee, D., Oh, J., Rim, J., Cho, S., Lee, K.M.: Exblurf: Efficient radiance fields for extreme motion blurred images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 17639–17648 (2023)
  * [27] Li, T., Slavcheva, M., Zollhoefer, M., Green, S., Lassner, C., Kim, C., Schmidt, T., Lovegrove, S., Goesele, M., Newcombe, R., et al.: Neural 3d video synthesis from multi-view video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5521–5531 (2022)
  * [28] Li, Z., Niklaus, S., Snavely, N., Wang, O.: Neural scene flow fields for space-time view synthesis of dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6498–6508 (2021)
  * [29] Lindelöf, E.: Sur l’application de la méthode des approximations successives aux équations différentielles ordinaires du premier ordre. Comptes rendus hebdomadaires des séances de l’Académie des sciences 116(3), 454–457 (1894)
  * [30] Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., Sheikh, Y.: Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.07751 (2019)
  * [31] Ma, L., Li, X., Liao, J., Zhang, Q., Wang, X., Wang, J., Sander, P.V.: Deblur-nerf: Neural radiance fields from blurry images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12861–12870 (2022)
  * [32] Martin-Brualla, R., Radwan, N., Sajjadi, M.S., Barron, J.T., Dosovitskiy, A., Duckworth, D.: Nerf in the wild: Neural radiance fields for unconstrained photo collections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7210–7219 (2021)
  * [33] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I. pp. 405–421 (2020)
  * [34] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) 41(4), 1–15 (2022)
  * [35] Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3883–3891 (2017)
  * [36] Neff, T., Stadlbauer, P., Parger, M., Kurz, A., Mueller, J.H., Chaitanya, C.R.A., Kaplanyan, A., Steinberger, M.: Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks. In: Computer Graphics Forum. vol. 40, pp. 45–59. Wiley Online Library (2021)
  * [37] Park, K., Sinha, U., Barron, J.T., Bouaziz, S., Goldman, D.B., Seitz, S.M., Martin-Brualla, R.: Nerfies: Deformable neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5865–5874 (2021)
  * [38] Park, K., Sinha, U., Hedman, P., Barron, J.T., Bouaziz, S., Goldman, D.B., Martin-Brualla, R., Seitz, S.M.: Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228 (2021)
  * [39] Park, S., Kim, K., Lee, J., Choo, J., Lee, J., Kim, S., Choi, E.: Vid-ode: Continuous-time video generation with neural ordinary differential equation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 2412–2422 (2021)
  * [40] Peng, C., Chellappa, R.: Pdrf: progressively deblurring radiance field for fast scene reconstruction from blurry images. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 2029–2037 (2023)
  * [41] Philip, J., Morgenthaler, S., Gharbi, M., Drettakis, G.: Free-viewpoint indoor neural relighting from multi-view stereo. ACM Transactions on Graphics (TOG) 40(5), 1–18 (2021)
  * [42] Piala, M., Clark, R.: Terminerf: Ray termination prediction for efficient neural rendering. In: 2021 International Conference on 3D Vision (3DV). pp. 1106–1114. IEEE (2021)
  * [43] Pumarola, A., Corona, E., Pons-Moll, G., Moreno-Noguer, F.: D-nerf: Neural radiance fields for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10318–10327 (2021)
  * [44] Rubanova, Y., Chen, R.T., Duvenaud, D.K.: Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems 32 (2019)
  * [45] Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4104–4113 (2016)
  * [46] Schönberger, J.L., Zheng, E., Frahm, J.M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In: European conference on computer vision. pp. 501–518. Springer (2016)
  * [47] Son, H., Lee, J., Lee, J., Cho, S., Lee, S.: Recurrent video deblurring with blur-invariant motion estimation and pixel volumes. ACM Transactions on Graphics (TOG) 40(5), 1–18 (2021)
  * [48] Srinivasan, P.P., Deng, B., Zhang, X., Tancik, M., Mildenhall, B., Barron, J.T.: Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7495–7504 (2021)
  * [49] Srinivasan, P.P., Ng, R., Ramamoorthi, R.: Light field blind motion deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3958–3966 (2017)
  * [50] Sun, C., Sun, M., Chen, H.T.: Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5459–5469 (2022)
  * [51] Sun, J., Xie, Y., Chen, L., Zhou, X., Bao, H.: Neuralrecon: Real-time coherent 3d reconstruction from monocular video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15598–15607 (2021)
  * [52] Sun, J., Cao, W., Xu, Z., Ponce, J.: Learning a convolutional neural network for non-uniform motion blur removal. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 769–777 (2015)
  * [53] Toschi, M., De Matteo, R., Spezialetti, R., De Gregorio, D., Di Stefano, L., Salti, S.: Relight my nerf: A dataset for novel view synthesis and relighting of real world objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20762–20772 (2023)
  * [54] Tretschk, E., Tewari, A., Golyanik, V., Zollhöfer, M., Lassner, C., Theobalt, C.: Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12959–12970 (2021)
  * [55] Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 (2021)
  * [56] Wang, P., Zhao, L., Ma, R., Liu, P.: Bad-nerf: Bundle adjusted deblur neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4170–4179 (2023)
  * [57] Wen, B., Tremblay, J., Blukis, V., Tyree, S., Müller, T., Evans, A., Fox, D., Kautz, J., Birchfield, S.: Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 606–617 (2023)
  * [58] Whyte, O., Sivic, J., Zisserman, A., Ponce, J.: Non-uniform deblurring for shaken images. International journal of computer vision 98, 168–186 (2012)
  * [59] Yu, A., Fridovich-Keil, S., Tancik, M., Chen, Q., Recht, B., Kanazawa, A.: Plenoxels: Radiance fields without neural networks. arXiv preprint arXiv:2112.05131 (2021)
  * [60] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14821–14831 (2021)
  * [61] Zhang, J., Liu, X., Ye, X., Zhao, F., Zhang, Y., Wu, M., Zhang, Y., Xu, L., Yu, J.: Editable free-viewpoint video using a layered neural representation. ACM Transactions on Graphics (TOG) 40(4), 1–18 (2021)
  * [62] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017)