# Point-DynRF: Point-based Dynamic Radiance Fields from a Monocular Video

Byeongjun Park Changick Kim
Korea Advanced Institute of Science and Technology (KAIST)
{pbj3810<EMAIL_ADDRESS>
###### Abstract

Dynamic radiance fields have emerged as a promising approach for generating
novel views from a monocular video. However, previous methods enforce the
geometric consistency to dynamic radiance fields only between adjacent input
frames, making it difficult to represent the global scene geometry and
degenerates at the viewpoint that is spatio-temporally distant from the input
camera trajectory. To solve this problem, we introduce point-based dynamic
radiance fields (Point-DynRF), a novel framework where the global geometric
information and the volume rendering process are trained by neural point
clouds and dynamic radiance fields, respectively. Specifically, we reconstruct
neural point clouds directly from geometric proxies and optimize both radiance
fields and the geometric proxies using our proposed losses, allowing them to
complement each other. We validate the effectiveness of our method with
experiments on the NVIDIA Dynamic Scenes Dataset and several causally captured
monocular video clips.

[13]

Figure 1: Point-based Dynamic Radiance Fields for Long-Term Novel View
Synthesis. Point-DynRF takes a monocular video following dynamic objects, as
shown in (a), and uses neural 3D points generated from the input video to
efficiently represent dynamic radiance fields. (b) We design a novel framework
that samples a subset point cloud (green boxes) at each time step from the
entire point cloud (a blue box) and regresses dynamic radiance fields only on
the scene surface where the subset point cloud are located. Especially with a
wide-range camera trajectory, Point-DynRF addresses the duplicating problem of
the state-of-the-art method (red boxes).

##  1 Introduction

Consider a monocular video recording of dynamic objects. While it is
challenging to distinguish between static and dynamic areas in a single frame,
analyzing the entire video sequence enables us to differentiate the background
from the moving objects. Moreover, we can also predict the background outside
a captured frame by assuming that the background scene remains constant over
time. This scene reasoning ability enables us to identify the moving objects
and integrate partially available scene information, which is crucial for
understanding in-the-wild videos and scaling the free-viewpoint rendering.

Existing novel view synthesis methods for monocular videos often use separate
modules for static and dynamic regions, where view-dependent radiance fields
are designed for static regions and time-dependent radiance fields for dynamic
regions [47, 22, 13, 31, 44, 39, 25, 23]. In this regard, recent deformable
NeRFs [31, 44, 12, 32] learn sufficient view dependencies from small camera
trajectories to represent the background, while representing the remaining
regions using time-dependent radiance fields. However, in the real world,
there are many cases where the camera does not follow a narrow trajectory, and
deformable NeRFs fail to distinguish between the background and dynamic
objects due to the lack of learning view dependencies.

On the other hand, flow-based methods [22, 13, 25, 23] use additional
supervisions from pre-trained depth [33], optical flow [38] and semantic
segmentation [16] estimation networks to constrain the radiance field since
identifying moving objects and estimating their motion in monocular videos are
challenging. By imposing geometric constraints on the radiance field, flow-
based methods can design dynamic radiance fields for large scenes. Despite its
scalability, we observe that flow-based methods quickly degenerate for
viewpoints in spatio-temporally distant from the input camera trajectory, and
the generated image is blurry and sometimes contain duplicated objects. This
is because time-dependent radiance fields are trained by the optical flow
supervision to satisfy geometric consistency between adjacent frames, which
fails to incorporate global geometric information of entire scene from wide-
range camera trajectories. Figure 1-(b) shows the problem of a state-of-the-
art dynamic view synthesis method [13] where a person is duplicated outside of
the input frame and the background is not preserved after the person walks by
because of the duplicated person.

Motivated by our observations, we introduce point-based dynamic radiance
fields (Point-DynRF) to represent the entire scene geometry and produce more
realistic long-term novel view synthesis results. Point-DynRF is built upon
the Point-NeRF [46] representation, which reconstructs 3D neural point clouds
and encodes the localized scene representation from neighboring neural points.
While Point-NeRF aims at static scenes, we extend it to consider the time
domain where different subsets of neural point clouds are sampled at each time
step to represent time-varying radiance fields. Specifically, we utilize a
pre-trained depth estimation network [33] and pre-defined foreground masks
[13] to initialize pixel-wise depth and rigidness of our neural point clouds,
respectively. Moreover, we propose a dynamic ray marching, where we march a
ray over a subset of the entire point cloud consisting of all background
points and the dynamic points corresponding to the rendering time. As each
subset of neural point clouds represents the actual scene surface of the
corresponding rendering time, our Point-DynRF can regress dynamic radiance
fields only on the scene surface at that rendering time and alleviate to
generate of duplicated dynamic objects.

To train Point-DynRF, we simply modify the training objective of DVS [13] and
jointly optimize the neural point clouds and dynamic radiance fields, rather
than solely supervising the radiance fields using initialized depth and
foreground masks. Specifically, we train Point-DynRF to align the initialized
learnable depth and foreground masks with the volume rendered depth and
dynamicsness maps. Through the joint optimization scheme, the global scene
geometry and dynamic radiance fields are further refined and complement each
other, addressing the degeneration problems of previous methods in long-term
dynamic view synthesis. Extensive experiments on the NVIDIA Dynamic Scenes
[47] and several monocular video clips show the efficiency and effectiveness
of our method.

Figure 2:  An overview of network architecture. Our framework consists of
three components. First, we initialize per-frame depth maps $D_{n}$ and
foreground masks $M_{n}$ for a given $N$ frames. Then, we back-project each
pixel of $N$ frames to reconstruct our neural 3D point clouds. Each neural
point $i$ contains its spatio-temporal locations $(\textbf{p}_{i},t_{i})$, a
point-wise rigidness $\gamma_{i}$, a randomly initialized neural feature
vector $f_{i}$ to represent the local scene representation. Then, we select a
subset point cloud at a rendering time step $t$ and assign sampling points
where the ray meets the neural points as they march. Finally, we regress a
volume density and a radiance on both view-dependent and time-dependent
radiance fields. The volume density and radiance for each sampling point in
the ray are integrated via volume rendering to output an RGB color.

##  2 Related Works

##### Neural representations for novel view synthesis.

Novel view synthesis aims to generate new views of a scene given multiple
posed images. To consider the arbitrary viewpoints in three-dimension,
multiple-view geometry is often utilized and combined with image-based
rendering methods to synthesize realistic novel views [10, 20, 34, 50, 9].
Moreover, deep neural networks have been explored to improve the visual
quality of novel views by using explicit geometric proxies, such as multi-
plane image [36, 49, 42], point cloud [1, 43, 40], and voxel [35, 12].

Recently, coordinate-based neural representations [8, 26, 27, 29] have
achieved outstanding results in modeling the scene as implicit scene
representations. In the context of novel view synthesis, Neural Radiance
Fields (NeRF) [27] has been proposed to model the scene as a continuous
volumetric field with neural networks. The success of NeRF is attributed to
the extension of neural representation design, which facilitates free-
viewpoint rendering with various applications, such as relighting [4],
appearance editing [24, 48], reflections [14], and generative models [7, 28,
5]. Despite its remarkable scalability, several methods [46, 19] focus on the
fact that NeRF samples a large number of unnecessary points for each ray.
Specifically, Point-NeRF [46] models a volumetric radiance field with 3D
neural point clouds, avoiding ray sampling in the empty space and encoding
localized scene representations. Our work extends Point-NeRF, encoding
different scene representations for static and dynamic regions by leveraging
its capability to encode localized scene representations.

##### Dynamic view synthesis for videos.

Dynamic view synthesis focuses on generating novel views with dynamically
moving objects at arbitrary viewpoints and time stamps. Several works have
been proposed to model time-varying scenes on multiple time-synchronized
videos [3, 37, 50, 21], sparse camera views [17, 11], stereo camera [2], and
specific domain [6, 15, 41]. However, modeling neural scene representation
from a monocular video is more challenging since it contains a single
viewpoint for each time stamp. This causes ambiguities that radiance can be
changed in either a view-dependent or a time-varying or both. To solve this
ambiguity, Yoon  _et al_. [47] combines an explicit depth estimation module to
leverage geometric transformations (i.e., warping) and to blend strategies for
synthesizing novel views of a dynamic scene, but it requires a time-consuming
preprocessing to generate manually annotated foreground masks. Recently, flow-
based methods [13, 22, 45, 25] directly regress 4D space-time radiance fields
by using additional geometric proxies, such as depth [33] and optical flow
[38] estimation networks. Geometric proxies are used as additional supervision
to learn their deformation module and constrain temporal changes of a dynamic
scene. Several methods [44, 31, 39, 32, 30, 12] propose deformable neural
radiance fields by modeling a canonical template radiance field and a
deformation field for each frame. Our work also uses geometric proxies for
point cloud initialization, but we optimize the dynamic radiance fields and
geometric proxies together based on the volume rendering process. Moreover,
point-based dynamic radiance fields allow us to incorporate the entire scene
geometry and regress the radiance fields from the actual scene surface for
each rendering time.

##  3 Method

Given a monocular video $V=\\{I_{1},I_{2},\dots,I_{N}\\}$ consisting of $N$
frames, our goal is to synthesize novel views at arbitrary viewpoints and time
steps. To achieve this, we design point-based dynamic radiance fields as shown
in Fig. 2. Our model is built on the Point-NeRF [46] representation and
extends it to consider time-varying radiance fields. We briefly describe the
volume rendering formulation in 3.1 and then explain how to extend Point-NeRF
to consider the time domain in 3.2. Finally, we illustrate the optimization
scheme of Point-DynRF in 3.3.

###  3.1 Volume rendering

We construct continuous volumetric fields for modeling dynamic scenes,
following the formulation in NeRF [27]. Given the camera center
$\textbf{o}\in\mathbb{R}^{3}$ and viewing direction
$\textbf{d}\in\mathbb{R}^{2}$, each pixel’s RGB color
$\textbf{C}\in\mathbb{R}^{3}$ is computed by marching a ray
$\textbf{r}(s)=\textbf{o}+s\textbf{d}$ through the pixel and approximate the
integration over radiance and its volume density
$\\{(r_{j},\sigma_{j})\in\mathbb{R}^{3}\times\mathbb{R}\mid j=1,\dots,M\\}$
for $M$ sampling points in the ray as:

$\textbf{C}(\textbf{r})=\sum_{j=1}^{M}{T_{j}(\alpha(\sigma_{j}\delta_{j}))r_{j}},$
(1) $T_{j}=\text{exp}(-\sum_{k=1}^{j-1}{\sigma_{k}\delta_{k}}),$ (2)

where $\alpha(x)=1-\text{exp}(-x)$ outputs the opacity at each sampling point,
$\delta_{j}$ is the distance between two adjacent sampling points and $T_{j}$
represents a volume transmittance.

###  3.2 Point-DynRF Representation

Point-NeRF [46] is pre-trained on a multi-view stereo dataset [18] or uses
only points located on the actual surface with high confidence from COLMAP. In
dynamic scenes, however, it fails to accurately regress the scene geometry
since dynamic objects disrupt to estimate point-to-point correspondences. To
solve this ambiguity, we propose Point-DynRF with associated neural point
clouds, which are initialized by imprecise depth maps and pre-defined
foreground masks, and jointly optimize scene geometry and dynamic radiance
fields.

##### Neural Point Clouds Reconstruction.

Our neural point clouds are reconstructed by depth maps
$\\{D_{1},...,D_{N}\\}$ and foreground masks $\\{M_{1},...,M_{N}\\}$. We first
initialize per-frame depths by using disparity maps $disp_{n}$ obtained from
DPT [33] and convert it to depth maps with per-frame scale $s_{n}$ and shift
$b_{n}$ values as:

$D_{n}(p)=s_{n}/(disp_{n}(p)+b_{n}).$ (3)

Note that we design a more stable network by optimizing scale, shift, and
disparity together rather than optimizing pixel-wise depth values
individually. Per-frame foreground masks are obtained as same as DVS [13], and
we directly parameterize our point-wise rigidness $\gamma$ with $1$ for the
background and $0$ for moving objects. Thus, we reconstruct neural point
clouds as $\mathbb{P}=\\{(\textbf{p}_{i},t_{i},\textbf{f}_{i},\gamma_{i})\mid
i=1,...,L\\}$, where each point $i$ is located at $\textbf{p}_{i}$ and
captured at time steps $t_{i}$ with a point-wise rigidness $\gamma_{i}$. We
also use a neural feature vector $\textbf{f}_{i}$, which are randomly
initialized and parameterized to encode local scene representations. Since
each neural point is a one-to-one match to each pixel of input frames,
training the $\textbf{p}_{i}$ and $\gamma_{i}$ of each neural point optimizes
the depth and foreground masks.

##### Dynamic Ray Marching.

Our dynamic radiance fields are regressed from a different subset of the
entire point cloud set $\mathbb{P}$ at each time step based on the sampling
time and the point-wise rigidness. Specifically, we select neural points where
their point-wise rigidness is higher than the threshold $\lambda=0.5$, or its
temporal location is the same as the sampling time as:

$\mathbb{P}_{t}=\\{(\textbf{p}_{i},t_{i},\textbf{f}_{i},\gamma_{i})\in\mathbb{P}\mid
t_{i}=t\ \text{or}\ \gamma_{i}>\lambda\\},$ (4)

where neural points with $\gamma_{i}$ is higher than $\lambda$ to be
background points to represent the static region whether the position of the
dynamic object changes with each subset. Moreover, dynamic neural points do
not represent the scene surface from different viewpoints, resulting in
avoiding unnecessary ray sampling and not duplicating objects.

##### Neural Point Aggregation.

After we select the subset of the neural point cloud, Point-DynRF aggregates
neural points to output the density and radiance for each shading point.
Specifically, we follow the Point-NeRF [46] to query $K=8$ neighbor neural
points for ray sampling, and we encode per-point local scene features with an
MLP layer $F$ for each shading point x as:

$f_{i,\textbf{x}}=F(\textbf{f}_{i},\textbf{x}-\textbf{p}_{i}).$ (5)

##### Volume Density Regression.

We use density regression MLP layers $G_{s}$ and $G_{d}$ for static and
dynamic regions, respectively. We first encode per-point time-invariant volume
density $\sigma^{s}$ and time-varying volume density $\sigma^{d}$ as:

$\sigma^{s}_{i,\textbf{x}}=G_{s}(f_{i,\textbf{x}}),$ (6)
$\sigma^{d}_{i,\textbf{x}}=G_{d}(f_{i,\textbf{x}},t).$ (7)

Then, the time-invariant volume density $\sigma^{s}_{\textbf{x}}$ and time-
variant volume density $\sigma^{d}_{\textbf{x}}$ at the sampling point x is
regressed as:

$\sigma^{s}_{\textbf{x}}=\sum_{i}\sigma^{s}_{i,\textbf{x}}\frac{w_{i}}{\sum
w_{i}},$ (8)
$\sigma^{d}_{\textbf{x}}=\sum_{i}\sigma^{d}_{i,\textbf{x}}\frac{w_{i}}{\sum
w_{i}},$ (9)

where $w_{i}=\frac{1}{\lvert p_{i}-x\rvert}$ is for a distance-based weighted
sum that gives higher weight to neural points closer to x.

##### Radiance Regression.

We regress a view-dependent radiance $r^{s}_{\textbf{x}}$ and a time-dependent
radiance $r^{d}_{\textbf{x}}$ by using MLP layers $R_{s}$ and $R_{d}$,
respectively, as:

$r^{s}_{\textbf{x}}=R_{s}(\sum_{i}\frac{w_{i}}{\sum
w_{i}}f_{i,\textbf{x}},d),$ (10)
$r^{d}_{\textbf{x}}=R_{d}(\sum_{i}\frac{w_{i}}{\sum
w_{i}}f_{i,\textbf{x}},t),$ (11)

where $d$ and $t$ is the viewing direction and sampling time, respectively.

##### Blending Weight Regression.

We directly regress blending weights from the point-wise rigidness
$\gamma_{i}$ of neighboring points as:

$b_{\textbf{x}}=\mathbbm{1}[\sum_{i}(\frac{w_{i}}{\sum
w_{i}}(1-\gamma_{i}))>\lambda],$ (12)

where $\mathbbm{1}$ equals to one if the condition is true. We define the
blending weight as $0$ or $1$ so that either static or dynamic radiance fields
dominate at each shading point. To optimize $\gamma_{i}$, we use the gradient
clamping used in Point-NeRF to $\sum_{i}(\frac{w_{i}}{\sum
w_{i}}(1-\gamma_{i}))$ if
$\text{MAX}(\sigma^{s}_{\textbf{x}},\sigma^{d}_{\textbf{x}})$ is larger than a
threshold $0.7$ and there exists at least one dynamic point.

###  3.3 Training Objectiveness

In this section, we briefly demonstrate how we jointly optimize dynamic
radiance fields and neural 3D point clouds. Specifically, we introduce
reconstruction losses to learn combined NeRF, static NeRF, and dynamic NeRF in
Sec. 3.3.1, scene geometry losses to reconstruct accurate neural points in
Sec. 3.3.2 and joint optimization of Point-DynRF and neural 3D points in Sec.
3.3.3.

####  3.3.1 Reconstruction Loss

##### Combined NeRF

We apply a reconstruction loss to dynamic radiance fields, which are a blend
of view-dependent and time-dependent radiance fields. To this end, we combine
two radiance fields with blending weights as:

$\textbf{C}(\textbf{r},t,\mathbb{P}_{t})=\sum_{j=1}^{M}{T_{j}}\left(\alpha(\sigma^{s}_{j}\delta_{j})(1-b_{j})r^{s}_{j}+\alpha(\sigma^{d}_{j}\delta_{j})b_{j}r^{d}_{j}\right),$
(13)
$T_{j}=\text{exp}\left(-\sum_{k=1}^{j-1}{((\sigma^{s}_{k}(1-b_{k})+\sigma^{d}_{k}b_{k})\delta_{k}})\right),$
(14)

where $\textbf{C}(\textbf{r},t,\mathbb{P}_{t})$ is a volume rendered RGB value
from a ray $\textbf{r}(s)=\textbf{o}+s\textbf{d}$, rendering time $t$, and a
subset point cloud $\mathbb{P}_{t}$. To ensure that the dynamic radiance
fields accurately reconstruct the input video sequence, we jointly train
view-/time-dependent radiance fields by applying a reconstruction loss
$L^{full}_{rec}$ as:

$L^{full}_{rec}=\sum^{N}_{i=1}\sum_{uv}\lVert\textbf{C}(\textbf{r}^{i}_{uv},i,\mathbb{P}_{i})-I^{i}_{u,v}\rVert^{2}_{2},$
(15)

where $\textbf{r}^{i}_{uv}$ is a ray for pixel coordinates $(u,v)$ in $i$-th
frame and $I^{i}_{u,v}$ is a ground-truth RGB value for pixel coordinates
$(u,v)$ in $i$-th frame.

##### Static and Dynamic NeRF

We leverage point-based neural scene representations to learn time-invariant
radiance fields (Static NeRF) and time-variant radiance fields (Dynamic NeRF),
respectively. If we sample a subset point cloud $\mathbb{P}_{t,s}$ consisting
of only background points as:

$\mathbb{P}_{t,s}=\\{(\textbf{p}_{i},t_{i},\textbf{f}_{i},\gamma_{i})\in\mathbb{P}\mid\gamma_{i}>\lambda\\},$
(16)

a volume rendered image contain only the background with no dynamic objects.
Likewise, if we sample a subset point cloud $\mathbb{P}_{t,d}$ captured at a
specific time $t$ as:

$\mathbb{P}_{t,d}=\\{(\textbf{p}_{i},t_{i},\textbf{f}_{i},\gamma_{i})\in\mathbb{P}\mid
t_{i}=t\\},$ (17)

Point-DynRF can render an image restricted to only the neural points at that
moment. Figure 3 shows which subset point clouds are used by combined NeRF,
Static NeRF, and Dynamic NeRF. Thus, each radiance field is regressed by using
Eq. 1 as:

$\textbf{C}^{s}(\textbf{r},t,\mathbb{P}_{t,s})=\sum_{j=1}^{M}{T^{s}_{j}(\alpha(\sigma^{s}_{j}\delta_{j}))r^{s}_{j}},$
(18)
$\textbf{C}^{d}(\textbf{r},t,\mathbb{P}_{t,d})=\sum_{j=1}^{M}{T^{d}_{j}(\alpha(\sigma^{d}_{j}\delta_{j}))r^{d}_{j}}.$
(19)

Then, we apply reconstruction losses to each radiance field as:

$L^{s}_{rec}=\sum_{i,u,v}\lVert\textbf{C}^{s}(\textbf{r}^{i}_{uv},t,\mathbb{P}_{t,s})-I^{i}_{u,v}\rVert^{2}_{2}*\mathbbm{1}[M^{i}_{u,v}>\lambda],$
(20)

$L^{d}_{rec}=\sum_{i,u,v}\lVert\textbf{C}^{s}(\textbf{r}^{i}_{uv},t,\mathbb{P}_{t,d})-I^{i}_{u,v}\rVert^{2}_{2},$
(21)

where we only apply $L^{s}_{rec}$ to background regions by using a foreground
mask, and $\mathbbm{1}[M^{i}_{u,v}>\lambda]$ indicates whether a rigidness
value of pixel coordinates $(u,v)$ in $i$-th frame is higher than the
threshold $\lambda$. Finally, our reconstruction loss is formulated as:

$L_{rec}=\lambda^{full}_{rec}L^{full}_{rec}+\lambda^{s}_{rec}L^{s}_{rec}+\lambda^{d}_{rec}L^{d}_{rec}.$
(22)

Figure 3:  An overview of point cloud subsets for each NeRF.

####  3.3.2 Scene Geometry Loss

Initialized depth maps well represent the scene geometry but contain scale
ambiguities with other frames. Therefore, we use optical flow maps $f_{gt}$
from RAFT [38] to supervise scale $s_{t}$ and shift $b_{t}$ by applying a flow
loss $L_{flow}$ only for background pixels as:

$[u^{\prime},v^{\prime},z^{\prime}]^{T}=T^{-1}_{t^{\prime}}T_{t}D_{t}[u,v,1]^{T},$
(23)
$L_{flow}=\sum_{uv}\lVert(\frac{u^{\prime}}{z^{\prime}}-u,\frac{v^{\prime}}{z^{\prime}}-v)-f_{gt}\rVert*\mathbbm{1}[M^{i}_{u,v}>\lambda],$
(24)

where $t^{\prime}$ indicates a time step for adjacent frames and $T_{t}$ is
known camera parameters at $t$. Note that we detach the gradient from back-
propagating to the disparity so that only the scale and shift values can be
trained from the flow loss.

Moreover, we observe two cases that point-based ray marching can miss the ray
as shown in Fig. 4. While learning the scene geometry, some pixels may have
large depth values to satisfy the geometric consistency. As a result,
neighboring pixels in the image plane are also outside the query boundary,
resulting in the ray can not be marched. Therefore, we introduce
$L^{s}_{miss}$, which is an $\ell_{2}$-loss to minimize the depth value
corresponding to the pixel for which the ray is not marched. Also, rays may
not be marched for different render times in the fixed-viewpoint due the
dynamic ray sampling. To solve this problem, we introduce $L^{d}_{miss}$,
which is also an $\ell_{2}$-loss to maximize the rigidness of a green point in
Fig. 4-(b) to be one. Note that $L^{s}_{miss}$ and $L^{s}_{miss}$ are
introduced to deal with outlier cases, since missing rays are rarely present
in the entire training process. Consequently, a scene geometry loss $L_{geo}$
is formulated as:

$L_{geo}=\lambda_{flow}L_{flow}+\lambda^{s}_{miss}L^{s}_{miss}+\lambda^{d}_{miss}L^{d}_{miss}.$
(25)

Figure 4: Missing rays. Assuming a ray is marched when it has three neighbor
points for a shading point. (a) If the depth value is too large, the distance
between neighboring pixels in 3D world coordinates is larger than the querying
radius and fails to march the ray. Moreover, the subset point cloud is changed
as the render time varies, causing rays can be marched (b) or sometimes not
(c).

####  3.3.3 Joint Optimization

We further introduce loss functions that optimize the dynamic radiance field
and neural points together. Our joint optimization losses are formulated in
the same manner as DVS [13]. However, we make a modification by introducing
learnable per-frame depth and foreground masks, in contrast to the supervised
learning approach of matching the volume-rendered depth
$\textbf{\~{D}}(\textbf{r},t,\mathbb{P}_{t})$ and dynamicsness map
$\textbf{\~{M}}(\textbf{r},t,\mathbb{P}_{t})$ to the initialized depth and
foreground mask.

##### Depth Adjust Loss

We apply a depth adjust loss $L_{depth}$ to train the depth map of $i$-th
frame $D_{i}$ to match the expected depth
$\textbf{\~{D}}(\textbf{r},t,\mathbb{P}_{t})$ as:

$L_{depth}=\sum^{N}_{i=1}\sum_{uv}\lVert\textbf{\~{D}}(\textbf{r}^{i}_{uv},i,\mathbb{P}_{i})-D^{i}_{u,v}\rVert^{2}_{2},$
(26)

where $D^{i}_{u,v}$ is a depth value of pixel coordinates $(u,v)$ in $i$-th
frame.

##### Mask Adjust Loss

Similar to expected depth maps, we use volume rendering for the blending
weight to get the dynamicness map
$\textbf{\~{M}}(\textbf{r}^{i}_{uv},i,\mathbb{P}_{i})$ and propose a mask
adjust loss $L_{mask}$ to match the per-frame foreground mask.

$L_{mask}=\sum^{N}_{i=1}\sum_{uv}\lVert\textbf{\~{M}}(\textbf{r}^{i}_{uv},i,\mathbb{P}_{i})-M^{i}_{u,v}\rVert^{2}_{2}.$
(27)

Table 1: Quantitative results on NVIDIA Dynamic Scene dataset [47]. Image
quality is measured by PSNR and LPIPS. Furthermore, we show the average
performance over all view changes at the end. Best results in each metric are
in bold, and second best are underlined.

Methods | PSNR$(\uparrow)$ / SSIM$(\uparrow)$ / LPIPS$(\downarrow$)
---|---
Jumping | Skating | Truck | Balloon1 | Balloon2 | Playground | Avg
NeRF [27] \+ time | 16.6 / 0.42 / 0.48 | 19.1 / 0.46 / 0.54 | 17.1 / 0.39 / 0.40 | 17.5 / 0.40 / 0.29 | 19.8 / 0.54 / 0.22 | 13.7 / 0.18 / 0.44 | 17.3 / 0.40 / 0.40
D-NeRF [32] | 21.0 / 0.68 / 0.21 | 20.8 / 0.62 / 0.35 | 22.9 / 0.71 / 0.15 | 18.0 / 0.44 / 0.28 | 19.8 / 0.52 / 0.30 | 19.4 / 0.65 / 0.17 | 20.4 / 0.59 / 0.24
NR-NeRF [39] | 19.4 / 0.61 / 0.29 | 23.2 / 0.72 / 0.23 | 18.8 / 0.44 / 0.45 | 17.0 / 0.34 / 0.35 | 22.0 / 0.70 / 0.21 | 14.3 / 0.19 / 0.33 | 19.2 / 0.50 / 0.33
HyperNeRF [31] | 17.1 / 0.45 / 0.32 | 20.6 / 0.58 / 0.19 | 19.4 / 0.43 / 0.21 | 12.8 / 0.13 / 0.56 | 15.4 / 0.20 / 0.44 | 12.3 / 0.11 / 0.52 | 16.3 / 0.32 / 0.37
TiNeuVox [12] | 19.7 / 0.60 / 0.26 | 21.9 / 0.68 / 0.16 | 22.9 / 0.63 / 0.19 | 16.2 / 0.34 / 0.37 | 18.1 / 0.41 / 0.29 | 12.6 / 0.14 / 0.46 | 18.6 / 0.47 / 0.29
NSFF [22] |  23.9 / 0.80 / 0.15 | 28.8 / 0.88 / 0.13 | 25.4 / 0.76 / 0.17 | 21.5 / 0.69 / 0.22 | 23.8 / 0.73 / 0.23 | 20.8 / 0.70 / 0.22 | 24.1 / 0.76 / 0.18
DVS [13] | 23.4 / 0.83 / 0.10 |  31.9 / 0.94 / 0.04 | 27.9 / 0.86 / 0.09 |  21.6 / 0.75 / 0.11 |  26.6 / 0.85 / 0.05 |  23.7 / 0.85 / 0.08 |  25.9 / 0.85 / 0.08
RoDynRF [25] |  24.3 / 0.84 / 0.08 | 27.5 / 0.93 / 0.06 |  28.3 / 0.89 / 0.07 | 21.4 / 0.76 / 0.11 | 25.6 / 0.84 / 0.06 |  24.3 / 0.89 / 0.05 | 25.2 / 0.86 / 0.07
Point-DynRF (Ours) | 23.6 / 0.90 / 0.14 |  29.6 / 0.96 / 0.04 |  28.5 / 0.94 / 0.08 |  21.7 / 0.88 / 0.12 |  26.2 / 0.92 / 0.06 | 22.2 / 0.91 / 0.09 |  25.3 / 0.92 / 0.08
|  |  |  |  |  |  |

NSFF [22]

HyperNeRF [31]

DVS [13]

TiNeuVox [12]

RoDynRF [25]

Ours

Ground Truth

Figure 5: Comparison to baselines on NVIDIA Dynamic Scene Dataset [47].

##  4 Experiments

###  4.1 Experimental Settings

##### Dataset.

We evaluate our method on the Dynamic Scene Dataset [47]. We also use the same
evaluation protocol in DVS [13], which evaluate the quality of the synthesized
novel views through PSNR, SSIM and LPIPS metrics with ground truth images.
Note that we exclude the Umbrella sequences since COLMAP estimates inaccurate
camera poses, failing to regress the scene geometry accurately. Instead, we
evaluate our method on several causally captured monocular video clips, which
are more realistic videos and have a wide range of camera trajectories.
Causally captured videos provide various scene contexts that can be happened
in real-world scenarios, and COLMAP accurately estimates camera poses.

###  4.2 Comparison to Baselines

We now compare our method with the state-of-the-art methods on the NVIDIA
Dynamic Scene dataset [47]. Table 1 shows quantitative results, and Point-
DynRF demonstrates competitive performance with previous methods across most
scenes. Specifically, Point-DynRF outperforms all previous methods on the SSIM
metric for all scenes. However, due to the inaccurate camera pose estimated by
COLMAP, the construction of neural points in Point-DynRF is not optimal. As a
result, the rendered position of the dynamic object by Point-DynRF may differ
from the ground truth. In the Playground scene depicted in Fig. 5, Point-DynRF
generates a visually pleasing view but the position of the object is slightly
shifted behind compared to the ground-truth. In the Skating scene, Point-DynRF
generate realistic images, while flow-based methods like NSFF [22], DVS [13],
and RoDynRF [25] produce blurry images, and deformable NeRFs such as HyperNeRF
[31] and TiNeuVox [12] struggle to represent the scene.

Figure 6:  Long-Term Novel View Generation. For a fixed camera viewpoint at
time $t_{0}$, the first column shows the novel view at time $t_{0}$ and the
second column shows the novel view at time $t_{1}$. DynIBaR is over-fitted on
the input camera trajectory (green box) Figure 7: Extremely Wide-Range Camera
Trajectory. DVS produces duplicated dynamic objects in distant spatio-temporal
locations from the input camera trajectory (green boxes). Moreover, previous
methods are quickly degenerated at the spatio-temporally distant viewpoints.

###  4.3 Long-Term View Synthesis

We evaluate our method and flow-based dynamic view synthesis methods DVS [13],
RoDynRF [25] and DynIBaR [23] on real-world scenarios with a wide-range camera
trajectory. Point-DynRF can generate realistic novel views for viewpoints far
from the input camera trajectory in both space and time because it leverages
global scene geometry (i.e., neural 3D points). Figure 6 shows the long-term
view synthesis results where DVS [13] has quickly degenerated for unseen
viewing directions and produces artifacts in the background regions. Moreover,
DynIBaR [23] fails to generate a dynamic object since it is highly over-fitted
on the input trajectory. On the other hand, our Point-DynRF effectively
captures both the moving object and the background.

We also observe that DVS [13] infinitely duplicates the moving object and
RoDynRF [25] is also degenerated when a camera moves in an extremely single
direction, as shown in Fig. 7. This is due to geometric constraints focused on
the input camera trajectory, which fails to represent the global scene
geometry. Notably, our Point-DynRF generates more detailed dynamic regions as
well as static background regions. These results confirm the superiority of
our dynamic ray sampling and joint optimization scheme, which incorporates the
entire scene geometry.

Figure 8: Refinement of scale ambiguity. The initial point cloud may not
capture the complete scene geometry, but after optimization, the refined point
cloud is free from scale ambiguity.

(a)

(b)

(c)

(d)

Figure 9: Refinement of depth map. For a input frame (a) and initialized
disparity map (b), optimized disparity map (c) well represent the boundary of
a dynamic object. Also, expected depth map (d) is well aligned with the
disparity map.

###  4.4 Effect of Accurate Scene Geometry

To verify the effectiveness of our proposed losses for training the scene
geometry, we visualize initialized and refined point clouds as well as a depth
map on the Skating scene as shown in Fig. 8-9. The results demonstrate that
our joint optimization effectively regresses the scene geometry and address
the scale ambiguity problem in monocular videos, resulting in a dynamic
radiance field that accurately reflects this geometry.

###  4.5 Training and Rendering Time

Table 2 shows the training and rendering time on NVIDIA Dynamic Scene Dataset
[47] for recent dynamic view synthesis methods. Since Point-DynRF avoids the
unnecessary ray marching for empty space, the training process converges
faster, leading to a reduction in overall training time. In the rendering
process, however, searching neighbor neural points for each shading point
requires additional computational costs, and the rendering time is slower than
DVS [13] and RoDynRF [25].

Table 2: Comparison of Training and Rendering Time. Methods denoted by
$\dagger$ refer to reported performance in the paper.

Method | Training (GPU hours) | Rendering (s/img)
---|---|---
HyperNeRF [31] | 32 | 15
DVS [13] | 36 | 8
$\text{RoDynRF}^{\dagger}$ [25] | 28 | 8
$\text{DynIBaR}^{\dagger}$ [23] | 48 | 20
Ours | 20 | 11

###  4.6 Ablation Study for Point-DynRF Design

We conduct an ablation study for each component of Point-DynRF as shown in
Table 3. The results show the quantitative results, and we verify all
components contribute to the design of our Point-DynRF. Especially from the
results on $L_{flow}$ and $L_{depth}$, we confirm that the accuracy of the
neural points has a direct impact on the performance of dynamic radiance
fields. Also, the results on $\mathbb{P}_{t}$ confirm that our dynamic ray
marching scheme significantly improves the performance. Dynamic ray marching
ensures that the dynamicsness map for the novel view is well matched to the
actual scene, as shown in Fig. 10.

Table 3: Ablation Study of our proposed losses. We report the PSNR, SSIM and
LPIPS on the average of NVIDIA Dynamic Scene Dataset [47].

| PSNR ($\uparrow$) | SSIM ($\uparrow$) | LPIPS ($\downarrow$)
---|---|---|---
Ours w/o $\mathbb{P}_{t}$ | 23.62 (-1.68) | 0.755 (-0.161) | 0.148 (+0.067)
Ours w/o $L^{s}_{rec}$ | 24.38 (-0.92) | 0.843 (-0.073) | 0.121 (+0.040)
Ours w/o $L^{d}_{rec}$ | 25.08 (-0.22) | 0.901 (-0.015) | 0.097 (+0.016)
Ours w/o $L_{flow}$ | 24.13 (-1.07) | 0.872 (-0.042) | 0.099 (+0.018)
Ours w/o $L_{depth}$ | 24.45 (-0.85) | 0.856 (-0.070) | 0.100 (+0.019)
Ours w/o $L_{mask}$ | 24.66 (-0.64) | 0.884 (-0.032) | 0.111 (+0.030)
Ours | 25.30 | 0.916 | 0.081
|  |  |

Figure 10:  Qualitative Ablation of Dynamic Ray Marching. Without the dynamic
ray marching, dynamic points at other times interfere with dynamic radiance
fields at the rendering time.

##  5 Conclusion

We propose a novel framework called point-based dynamic radiance fields for
long-term dynamic view synthesis from monocular videos. In our approach, we
employ neural point clouds to encode geometric information and dynamic
radiance fields to handle the volume rendering process. Our framework, Point-
DynRF, optimizes the neural point clouds and dynamic radiance fields jointly,
leveraging direct regression from neural 3D points. This allows us to
effectively utilize the global scene geometry, which sets our method apart
from previous approaches relying on correspondences between neighboring
frames, limiting their ability to incorporate the overall scene geometry. We
believe that our work contributes significantly to the field of dynamic view
synthesis, enabling realistic rendering in various real-world scenarios.

##  Appendix A Overview

In this supplementary material, we further demonstrate our experimental setup
and provide additional results that the scene geometry is well regressed.
First, we explain the total loss formulation in our training process in Sec.
B. Then, we describe implementation details with image near-far bound
determination by neural points in Sec. C and provide additional results for
dynamicsness map of novel views in Sec. D. Finally, we demonstrate failure
cases in Sec. E.

##  Appendix B Losses

Our optimization process involves utilizing the loss functions $L_{rec}$,
$L_{geo}$, $L_{depth}$, and $L_{mask}$. These loss functions are either
modifications of those used in DVS [13] or newly introduced in this paper. To
train Point-DynRF more stable, we also incorporate with a depth order loss
$L_{order}$ introduced in DVS [13] and a sparsity loss $L_{sparse}$ introduced
in Point-NeRF [46].

##### Depth Order Loss

While the depth adjust loss helps optimize the overall scene geometry, there
are inherent challenges in accurately determining the distance between dynamic
objects and the background. Therefore, we use depth order loss $L_{order}$ to
allow the dynamic radiance fields to be regularized via a frame-by-frame depth
map. Since regularizing the dynamic radiance fields with per-frame depth maps
has scale and shift ambiguities as mentioned earlier, we leverage the volume
rendering process of Dynamic NeRF to propose $L_{order}$ as:

$L_{order}=\sum^{N}_{i=1}\sum_{uv}\lVert\textbf{\~{D}}(\textbf{r}^{i}_{uv},i,\mathbb{P}_{i})-\textbf{\~{D}}^{d}(\textbf{r}^{i}_{uv},i,\mathbb{P}_{i,d})\rVert^{2}_{2}.$
(28)

##### Sparsity Loss

Following the point-based representation, we apply a sparsity loss
$L_{sparse}$ on the point-wise rigidness to enforce it to be close to zero or
one as:

$L_{sparse}=\sum_{i}(\log(\gamma_{i})+\log(1-\gamma_{i})).$ (29)

##### Total Training Loss Formulation

We formulate a reconstruction loss $L_{rec}$, a scene geometry loss $L_{geo}$,
a depth adjust loss $L_{depth}$, a depth order loss $L_{order}$, a mask adjust
loss $L_{mask}$ and a sparsity loss $L_{sparse}$, to train our Point-DynRF and
neural points. Specifically, we define $\lambda^{full}_{rec}=3$,
$\lambda^{s}_{rec}=1$, $\lambda^{d}_{rec}=1$ for the reconstruction loss. For
the scene geometry loss, we define $\lambda_{flow}=0.1$,
$\lambda^{s}_{miss}=1$, $\lambda^{d}_{miss}=1$. Finally, we define
$\lambda_{depth}=0.1$, $\lambda_{order}=0.1$, $\lambda_{mask}=0.1$, and
$\lambda_{sparse}=0.0002$ to formulate the final loss as:

$\displaystyle L_{total}$
$\displaystyle=L_{rec}+L_{geo}+\lambda_{depth}L_{depth}+\lambda_{order}L_{order}+$
$\displaystyle\qquad\lambda_{mask}L_{mask}+\lambda_{sparse}L_{order}.$

##  Appendix C Implementation Details.

We randomly sampled $1024$ rays in a batch, and each ray was assigned up to
$32$ sampling points. We used COLMAP to estimate the camera poses and resized
all images into a resolution of $480\times 272$. Also, we initialized our
scale and shift parameters by using near and far bounds from COLMAP. We
trained Point-DynRF for 250$k$ iterations, and training takes about $20$ hours
on a single NVIDIA Geforce RTX $3090$ GPU.

Figure 11:  Image Near-Far Bound Determination.

NeRF $+$ time [27]

NR-NeRF [39]

NSFF [22]

DVS [13]

Ours

Ground Truth

Figure 12: Comparison to baselines on NVIDIA Dynamic Scene Dataset [47].

##### Near-Far Boundary Determination

As our Point-DynRF is built on Point-NeRF [46] representation, dynamic
radiance fields are regressed in 3D world coordinates, not in NDC space used
by previous methods. Moreover, we need to render the far background as well,
so we set the image near-far bound dynamically associated with the neural
points. Specifically, we set the image near boundary to be the depth for the
nearest neural point multiplied by $0.9$, and the image far boundary to be the
depth for the farthest neural point multiplied by $1.1$. Figure 11 shows the
convergence of the image near-far boundary of the scenes in the Dynamic Scene
Dataset [47] during training. This result confirms that the scene geometry is
stably trained and refined the initialized scene geometry well.

Figure 13:  Dynamicsness Maps for novel views.

##  Appendix D Additional Results

##### Additional Qualitative Results.

We further provide additional qualitative results on Dynamic Scene Dataset
[47] . Point-DynRF generates more realistic images compared to previous
methods, and the human face in the third row of Fig. 12 confirms that Point-
DynRF produces much sharper images, while other methods either fail to
synthesize or produce blurry images. We also provide a video result of a
causally captured monocular video that our Point-DynRF generates realistic
images while the state-of-the-art method DVS [13] suffers from duplicated
dynamic objects when rendering from a fixed viewpoint.

Our foreground masks $(M_{1},\dots,M_{N})$ are also optimized during the
training, so we provide dynamicsness maps for novel views, as shown in Fig 13.
For each novel view, our Point-DynRF can render blending weights by using the
volume rendering process. These dynamicsness maps for novel views confirm that
our Point-DynRF well represents dynamic regions in the scene, and we can see
that the static representation in the center of the person in the Playground
Sequence is due to the fact that all the sequences in the input video for that
region are learned as dynamic regions and represented as background by the
miss ray marching scheme.

##  Appendix E Failure Cases

Figure 14:  Failure Case.

While Point-DynRF optimizes well the ambiguous initial geometry and foreground
masks, it fails to represent the scene if the neural point clouds are
unnaturally initialized. A combination of inaccurate camera pose, depth map,
and foreground masks sometimes unnaturally initialize neural point clouds
where background points are closer to the camera than dynamic points as shown
in Fig. 14. In this failure case, Point-DynRF falls short of distinguishing
background points in front of the dynamic objects even addressing the scale
ambiguity, and novel views also contain artifacts on these background points.

## References

  * [1] Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky.  Neural point-based graphics.  In European Conference on Computer Vision, pages 696–712. Springer, 2020.
  * [2] Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, and James Tompkin.  Matryodshka: Real-time 6dof video view synthesis using multi-sphere images.  In European Conference on Computer Vision, pages 441–459. Springer, 2020.
  * [3] Aayush Bansal, Minh Vo, Yaser Sheikh, Deva Ramanan, and Srinivasa Narasimhan.  4d visualization of dynamic events from unconstrained multi-view videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5366–5375, 2020.
  * [4] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch.  Nerd: Neural reflectance decomposition from image collections.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12684–12694, 2021.
  * [5] Shengqu Cai, Anton Obukhov, Dengxin Dai, and Luc Van Gool.  Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3981–3990, 2022.
  * [6] Joel Carranza, Christian Theobalt, Marcus A Magnor, and Hans-Peter Seidel.  Free-viewpoint video of human actors.  ACM transactions on graphics (TOG), 22(3):569–577, 2003.
  * [7] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5799–5809, 2021.
  * [8] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939–5948, 2019.
  * [9] Paul Debevec, Yizhou Yu, and George Borshukov.  Efficient view-dependent image-based rendering with projective texture-mapping.  In Eurographics Workshop on Rendering Techniques, pages 105–116. Springer, 1998.
  * [10] Paul E Debevec, Camillo J Taylor, and Jitendra Malik.  Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach.  In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 11–20, 1996.
  * [11] Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu.  Neural radiance flow for 4d view synthesis and video processing.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14304–14314. IEEE Computer Society, 2021.
  * [12] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian.  Fast dynamic radiance fields with time-aware neural voxels.  In SIGGRAPH Asia 2022 Conference Papers, 2022.
  * [13] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.  Dynamic view synthesis from dynamic monocular video.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5712–5721, 2021.
  * [14] Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, and Song-Hai Zhang.  Nerfren: Neural radiance fields with reflections.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18409–18418, 2022.
  * [15] Marc Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt.  Livecap: Real-time human performance capture from monocular video.  ACM Transactions On Graphics (TOG), 38(2):1–17, 2019.
  * [16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [17] Zeng Huang, Tianye Li, Weikai Chen, Yajie Zhao, Jun Xing, Chloe LeGendre, Linjie Luo, Chongyang Ma, and Hao Li.  Deep volumetric video from very sparse multi-view performance capture.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [18] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs.  Large scale multi-view stereopsis evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 406–413, 2014.
  * [19] Andreas Kurz, Thomas Neff, Zhaoyang Lv, Michael Zollhöfer, and Markus Steinberger.  Adanerf: Adaptive sampling for real-time rendering of neural radiance fields.  In European Conference on Computer Vision, pages 254–270. Springer, 2022.
  * [20] Marc Levoy and Pat Hanrahan.  Light field rendering.  In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 31–42, 1996.
  * [21] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al.  Neural 3d video synthesis from multi-view video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5521–5531, 2022.
  * [22] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498–6508, 2021.
  * [23] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely.  Dynibar: Neural dynamic image-based rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4273–4284, 2023.
  * [24] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell.  Editing conditional radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5773–5783, 2021.
  * [25] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, and Jia-Bin Huang.  Robust dynamic radiance fields.  arXiv preprint arXiv:2301.02239, 2023.
  * [26] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4460–4470, 2019.
  * [27] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [28] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453–11464, 2021.
  * [29] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165–174, 2019.
  * [30] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865–5874, 2021.
  * [31] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz.  Hypernerf: a higher-dimensional representation for topologically varying neural radiance fields.  ACM Transactions on Graphics (TOG), 40(6):1–12, 2021.
  * [32] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  * [33] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12179–12188, 2021.
  * [34] Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski.  A comparison and evaluation of multi-view stereo reconstruction algorithms.  In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06), volume 1, pages 519–528. IEEE, 2006.
  * [35] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer.  Deepvoxels: Learning persistent 3d feature embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437–2446, 2019.
  * [36] Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely.  Pushing the boundaries of view extrapolation with multiplane images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 175–184, 2019.
  * [37] Timo Stich, Christian Linz, Georgia Albuquerque, and Marcus Magnor.  View and time interpolation in image space.  In Computer Graphics Forum, volume 27, pages 1781–1787. Wiley Online Library, 2008.
  * [38] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In European conference on computer vision, pages 402–419. Springer, 2020.
  * [39] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt.  Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12959–12970, 2021.
  * [40] Cen Wang, Minye Wu, Ziyu Wang, Liao Wang, Hao Sheng, and Jingyi Yu.  Neural opacity point cloud.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(7):1570–1581, 2020.
  * [41] Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher-Shlizerman.  Humannerf: Free-viewpoint rendering of moving people from monocular video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16210–16220, 2022.
  * [42] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn.  Nex: Real-time view synthesis with neural basis expansion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8534–8543, 2021.
  * [43] Minye Wu, Yuehao Wang, Qiang Hu, and Jingyi Yu.  Multi-view neural human rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1682–1691, 2020.
  * [44] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli.  D2nerf: Self-supervised decoupling of dynamic and static objects from a monocular video.  arXiv preprint arXiv:2205.15838, 2022.
  * [45] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim.  Space-time neural irradiance fields for free-viewpoint video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9421–9431, 2021.
  * [46] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann.  Point-nerf: Point-based neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5438–5448, 2022.
  * [47] Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz.  Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5336–5345, 2020.
  * [48] Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao.  Nerf-editing: geometry editing of neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18353–18364, 2022.
  * [49] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  arXiv preprint arXiv:1805.09817, 2018.
  * [50] C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski.  High-quality video view interpolation using a layered representation.  ACM transactions on graphics (TOG), 23(3):600–608, 2004.


Dated: ]

# Suppressed fluctuations as the origin of the static magnetic order in
strained Sr2RuO4

Bongjae Kim1<EMAIL_ADDRESS>Sergii Khmelevskyi2 Cesare Franchini3,4
I. I. Mazin5,6<EMAIL_ADDRESS>1 Department of Physics, Kunsan National
University, Gunsan, 54150, Korea  2 Center for Computational Materials
Science, Institute for Applied Physics, Vienna University of Technology,
Wiedner Hauptstrasse $8$ \- $10$, $1040$ Vienna, Austria  3 University of
Vienna, Faculty of Physics and Center for Computational Materials Science,
Vienna A-1090, Austria  4 Dipartimento di Fisica e Astronomia, Università di
Bologna, 40127 Bologna, Italy  5 Department of Physics and Astronomy, George
Mason University, Fairfax, VA 22030  6 Quantum Science and Engineering Center,
George Mason University, Fairfax, VA 22030

([)

###### Abstract

Combining first principle density functional calculations and Moriya’s self-
consistent renormalization theory, we explain the recently reported
counterintuitive appearance of an ordered magnetic state in uniaxially
strained Sr2RuO4 beyond the Lifshits transition. We show that strain weakens
the quantum spin ﬂuctuations, which destroy the static order, more strongly
than the tendency to magnetism. A different rate of decrease of the spin
fluctuations vs. magnetic stabilization energy promotes the onset of a static
magnetic order beyond a critical strain.

††preprint:

_Introduction._ After many years of pursuing triplet superconductivity in
Sr2RuO4, recent studies provide evidence in favor of a singlet state [1, 2].
However, while most theorists believe that spin fluctuations are important for
superconductivity, there is no consensus on the detailed mechanism or the
pairing symmetry. Some of the proposals include $d_{x^{2}-y^{2}}$ [4, 3, 5],
$s+id_{xy}$ [6], $d_{xz}+id_{yz}$ [7, 8], or $g_{xy(x^{2}-y^{2})}$ [5, 9].
Experiments under uniaxial strain have been instrumental in the quest for
elucidating the pairing symmetry [10, 3]. One of the most impressive pieces of
evidence was the observation of a maximum in $T_{c}$ and $H_{C2}$ at the
critical strain corresponding to the Lifshitz transition [3, 11]. In systems
where the Fermi level ($E_{F}$) lies close to a Van Hove singularity (VHs),
small perturbations can induce a Lifshitz transition involving a tuned shift
of the Van Hove point across $E_{F}$ associated with a change of the Fermi
surface topology [12, 13, 14]. The latest addition in this direction is
establishing an ordered static magnetic state under strain beyond the Lifshitz
transition [15].

This result is counterintuitive because the magnetism is nearly universally
suppressed by pressure as the atomic distances, $a$, become smaller (See Refs
[16, 17] for representative examples): the bandwidth scales as $1/a^{2}$ and
DOS as $a^{2}$. Hence, $I(0)N(0)$, which determines the tendency towards
ferromagnetism in Stoner picture, becomes smaller and enhanced itinerancy
eventually wins over the spin exchange splitting. Here, $I(0)$ and $N(0)$ are
Stoner factor and density of states at Fermi level, respectively.
Understanding this paradox may bring up new and novel progress in
understanding the physics of this compound, which has a nontrivial magnetic
energy landscape [18]. The emergence of magnetism under pressure, as stated,
is very uncommon and usually associated with materials close to the itinerant
magnetism, where the long-range order is suppressed by the fluctuations (cf.
Fe-based superconductors, Ref. [19]). Of course, this is the average behavior
and around the Lifshitz transition, this picture is not valid. Further, the
leading instability in Sr2RuO4 is not ferromagnetic but at an incommensurate
$\mathbf{q}$, where the susceptibility is not affected by the Lifshitz
transition.

Such anomalous pressure effect on magnetism can be understood within Moriya’s
self-consistent renormalization (SCR) theory [20]. It stipulates that the
magnetization in itinerant magnets is soft and fluctuates in amplitude.
Assuming Gaussian fluctuations with the mean square amplitude, $\xi,$ it was
shown that if the total energy is expanded with the magnetic order parameter
$M$ as

$E=a+bM^{2}+cM^{4}+dM^{6}+...,.$ (1)

According to the fluctuation-renormalized expansion, the corresponding
coefficients change as

$\displaystyle\tilde{b}$
$\displaystyle=b+\frac{5}{3}c\xi^{2}+\frac{35}{9}d\xi^{4}+...$
$\displaystyle\tilde{c}$ $\displaystyle=c+\frac{14}{3}d\xi^{2}+....$ (2)

and so on. Obviously, this increases the value of the inverse spin
susceptibility, $2b,$ and its sign changes from negative to positive for
$\xi^{2}\gtrsim|3b/5c|$. Hence, the spin fluctuation in an itinerant magnet
can lead to _(i)_ the reduction of spin susceptibility in paramagnetic
materials, _(ii)_ the decrease of the average magnetic moment, or _(iii)_ the
suppression of the long-range magnetic order in a system where mean-field
theories, such as density functional theory (DFT), predicts to be an ordered
magnet.

SCR theory has been successfully applied to various (near-)ferromagnetic
materials, such as Pd [21], Ni3Ga and Ni3Al [22], ZrZn2 [23], and notable
other systems where the SCR theory is somewhat more straightforward. Also, it
was called up in connection with Fe-based superconductors [24, 25]. Moriya, in
his book [20], emphasizes that, while the entire frequency and momentum
dependence of spin susceptibility, via the fluctuation-dissipation theorem,
determines the value of $\xi,$ a significant role is played by the phase
space, $i.e.$, the fraction of the Brillouin zone (BZ) that is close to a
magnetic instability.

This point was clearly demonstrated in Ref. [22], where $\xi$ was estimated
from first principle calculations for two similar compounds, Ni3Al and Ni3Ga.
Within DFT, both systems are ferromagnetic, but, in reality, only the former
is, while Ni3Ga is a strongly renormalized paramagnet. Counterintuitively, the
calculated magnetic moment and magnetic stabilization energy were higher in
Ni3Ga. This apparent paradox was resolved in Ref. [22] by the fact that the
instability in Ni3Ga, while stronger, is also considerably less localized in
the momentum space (the characteristic volume of the unstable part of the BZ
is more than twice larger), which leads to a larger $\xi,$ and a stronger
suppression. As a result, from the view of mean-field theory, the magnetic
instability is entirely suppressed in the $more$ ferromagnetic material, while
in the $less$ one, the magnetism survives [22].

In this Letter, we argue that Sr2RuO4 represents a similar case with the spin-
density wave (SDW) type instability: The tendency toward SDW-
antiferromagnetism is stronger for the unstrained material, but the static
order is not established due to the even more substantial spin fluctuations.
We show that when uniaxial stress is applied, the fluctuations are suppressed
more strongly than the tendency towards magnetism which causes the emergence
of a static magnetic phase.

_Methods._ We employed the Vienna _ab initio_ simulation package (VASP) [26,
27] within the Generalized gradient approximation (GGA) by Perdew-Burke-
Ernzerhof functional [28]. The energy cut for the plane waves of 600 eV was
used with a Monkhorst-Pack $k$-mesh of 17 $\times$ 17 $\times$ 10 for the
primitive unit cell. For the spin spiral calculations with various
$\mathbf{q}$-values, we employed the generalized Bloch theorem [29], allowing
spiral calculations for an arbitrary wave vector without using the supercells.
We have further used a very stringent convergence criteria of $10^{-7}$ eV in
most cases.

To obtain a series of strained structures, we fixed the $a$ lattice parameter
and fully optimized $b$ and $c$ parameters and the internal positions. Due to
the well-known fact that GGA overestimates the equilibrium volume at a given
pressure, the critical stress (but not the critical strain) in our
calculations is likely overestimated (we get the Lifshits transition at
$\sigma=1.5$ GPa, see Fig. 1, about twice the value estimated in Ref. [15]),
but the important part is that we trace the evolution of magnetic properties
well past this transition.

_Results._ Sr2RuO4 does not order magnetically down to low temperatures;
however, it was predicted from DFT calculations [30] and later confirmed by
the neutron diffraction that it features strong spin fluctuations with the
wave vector $\mathbf{q}\sim(0.3,0.3,0)$ [31, 32] (here and below, we give the
wave vectors in the orthorhombic reciprocal lattice units, $i.e.,$ $2\pi/a$).
This incommensurate SDW feature, from the nesting of 1D $\alpha$ and $\beta$
Fermi surfaces, can be well-captured by DFT in the unstrained case using
approximate commensurate $\mathbf{q}$ [34]. While we did not attempt to locate
the exact position of the incommensurate SDW instability under strain, we
could monitor its evolution using our commensurate $\mathbf{q}$-space grid. In
our case, we obtained $\mathbf{q}_{SDW}$ at $(0.29,0.29,0)$ as the ground
state for the unstrained case, which gives excellent agreement with the
neutron diffraction study (from now on, we will omit $q_{z}=0$ for brevity).

Figure 1:  Uniaxial strain dependent evolution of (a) total energy and (b) Ru
local magnetic moment of Sr2RuO4 for various magnetic $\mathbf{q}$ ordering
vectors. The calculated SDW wave vector is $\mathbf{q}_{SDW}=(0.29,0.29,0)$.
For the total energy evaluation, we set $\Delta E$ as the energy difference of
each configuration with nonmagnetic one. The compressively strained $a$
lattice parameter and corresponding stress is indicated in the $x$-axis of the
plot. (c) Position of the VHs as a function of strain. For a critical strain
at 3.840 Å, corresponding to a pressure of 1.5GPa (see panel a and b), the VHs
crosses $E_{F}$ establishing a Lifshitz transition. Figure 2: (a) Ru local
magnetic moment and (b) energy calculation for
($\mathbf{q_{x}}$,$\mathbf{q_{y}}$) plane for three representative stress
cases: unstrained, 2.3 GPa and 5.7 GPa. 2.3 GPa corresponds to the case beyond
the Lifshitz transition, and 5.7 GPa to the highly strained limit. Here, the
$\mathbf{q}$-grid calculation is performed with varying $\mathbf{q_{x}}$ and
$\mathbf{q_{y}}$ values with the interval of
$\Delta\mathbf{q}=0.02\frac{2\pi}{a}$ at $\mathbf{q_{z}}=0$. (c) The
corresponding Fermi surface plots. The diagonal dashed arrows indicate the
nesting vectors for unstrained and 5.7 GPa cases.

First, to inspect the correlation between the leading magnetic instability and
uniaxial pressure in Sr2RuO4, we have calculated the total energy and Ru local
magnetic moment for representative magnetic orderings in a wide range of
stress levels and pressures. In addition to the $\mathbf{q}_{SDW}$ phase, we
have considered the $\mathbf{q}=(1/2,1/4)$ order suggested by Rømer $et$ $al$
[33] and the checkerboard ordering $\mathbf{q}=(1/2,1/2)$, as well as
$\mathbf{q}=(1/2,0)$ and $(0,1/2)$ configurations [34], based on the
observation that only the first nearest neighbor exchange responds to uniaxial
stress [34]. In Fig. 1, we display the energy difference of each configuration
with the nonmagnetic one ($\Delta E$) and Ru local moment for uniaxially
strained Sr2RuO4. We find that the SDW phase is the most stable state up to
3.6 GPa. As shown in Fig 1(c), the VHs crosses the Fermi energy at around 1.5
GPa, and the SDW phase remains the ground state well beyond that. Uniaxial
strain breaks the $C_{4}$ symmetry and bifurcates the first nearest neighbor
exchange interactions between Ru ions [34]. Hence, $\mathbf{q}=(1/4,1/2)$ and
$\mathbf{q}=(1/2,1/4)$ are progressively split in energy. We see that at the
highly-strained limit of 5.7 GPa, $\mathbf{q}=(1/2,1/4)$ is lower than
$\mathbf{q}_{SDW}$ in energy, which, however, does not indicate the
$\mathbf{q}=(1/2,1/4)$ is the ground state, as discussed in detail later.

Interestingly, the size of the Ru moment strongly varies for different
$\mathbf{q}$ values. The $\mathbf{q}_{SDW}$ moment is the largest for all
studied ranges, while the one for $\mathbf{q}=(1/2,0)$ is negligibly small.
The sizes of the moment for $\mathbf{q}=(1/4,1/2)$ and $\mathbf{q}=(1/2,1/4)$
also bifurcate, and, at 3.6 GPa, the moment suddenly vanishes for
$\mathbf{q}=(1/4,1/2)$.

For the unstrained case, while strong $\mathbf{q}_{SDW}$ tendency exists, the
long-range magnetic order does not set in due to the strong spin-fluctuations.
As the system undergoes uniaxial strain, $\mu$SR experiments have found the
stabilization of a magnetic order beyond the Lifshitz transition point [15].
However, the actual magnetic pattern is unknown, albeit an incommensurate SDW
was suggested [15]. On the other hand, preliminary nuclear magnetic resonance
(NMR) data, while also suggestive with magnetic order, do not see expected
broadening at low temperatures [35].

Our calculations, consistently with $\mu$SR, show that the DFT (that is to
say, mean field) ground state is always an incommensurate SDW, even though its
wave vector slowly shifts from $(0.29,0.29)$ toward $\mathbf{q}=(1/2,1/4)$
[Fig. 2(b)]. As expected, while SDW is always stable at the mean field level,
the stabilization energy [Fig. 2(b)] and the equilibrium magnetic moment [Fig.
2(a)] are considerably reduced by the strain.

The next step is to estimate the effect of the fluctuations. For materials
close to ferromagnetism, first principle calculations of the $\xi$ parameter
(Eq. 2) are tedious but feasible [21, 22]. For those close to
antiferromagnetism, we do not have a similar recipe. Therefore, we have
adapted “a poor man’s” approach, estimating $\xi^{2}$ as $\left\langle
M^{2}(\mathbf{q})\right\rangle,$ which is obtained by averaging the squared Ru
local magnetic moment over the entire Brillouin zone. Ideally, knowing the
frequency- and momentum-dependent spin susceptibility,
$\chi(\mathbf{q},\omega)$, one could evaluate $\xi^{2}$ [20]. Lacking this
information, we assume that the frequency dependence does not affect the trend
(similarly, the $\omega$ dependence was neglected, for instance, in Ref. [36],
where $\xi^{2}$ was estimated for some ferromagnets in the high-temperature
limit). Since we, on the contrary, are interested in zero-point fluctuation,
we can further assume that fluctuations at any $\mathbf{q}$ are excited with
the same probability and the amplitude proportional to $M(\mathbf{q})$, from
where our “poor man” formula follows. While this estimate may be
quantitatively inexact, it should provide us with the correct trend under
strain.

To this end, we have calculated, for three stresses of 0.0 GPa, 2.3 GPa, and
5.7 GPa, the moments and magnetic stabilization energies on a full 2D-grid in
$\mathbf{q_{x}}$ and $\mathbf{q_{y}}$, while keeping $\mathbf{q_{z}}=0$. The
results are shown in Fig. 2 (a) and (b). For the unstrained case, a large
moment ($>0.8\mu_{B}$) can be found in a broad region centered at
$\mathbf{q}_{SDW}=(0.29,0.29)$. The size of the moment strongly depends on
$\mathbf{q}$, and almost vanishes close to $\mathbf{q}=(1/2,0)$ and
$\mathbf{q}=(0,1/2)$. The large q range where a strong instability occurs
demonstrates the highly fluctuating nature of the spin moment in Sr2RuO4. As
the uniaxial stress is imposed, the lowest energy position is gradually moved
asymmetrically to $\mathbf{q}=(0.32,0.24)$, for 2.3 GPa, and
$\mathbf{q}=(0.34,0.20)$ for 5.7 GPa (See Fig. 2(b)). According to our
calculation, the magnetic instability is of SDW-type, not the commensurate one
$\mathbf{q}=(1/2,1/4)$ suggested in Ref. [33] and not checkerboard
$\mathbf{q}=(1/2,1/2)$. While there is an apparent tendency of the ground-
state $\mathbf{q}$ to shift from the original SDW one,
$\mathbf{q}_{SDW}=(0.29,0.29)$, toward $\mathbf{q}=(1/2,1/4)$, the magnetism
of the system remains incommensurate. Our calculations show that, while
proximity to an SDW instability in unstrained Sr2RuO4 is definitely related to
nesting [23], the exact position is not given by the nesting vector (which is
related to the imaginary, not the real part of susceptibility[37]), and in
fact the evolution of the SDW vector with strain cannot be attributed to the
evolution of nesting (Fig. 2) [38].

Estimating $\xi^{2}\approx\left\langle M^{2}(\mathbf{q)}\right\rangle$, we
obtain for 0.0 GPa, 2.3 GPa and 5.7 GPa stress, respectively, 0.43, 0.36, and
0.22 ${\mu_{B}}^{2}$, demonstrating the progressive suppression of the spin
fluctuations. The apparent shrinking of the area where the moment survives
(Fig. 2(a)) demonstrates the dramatic reduction of the phase space available
for fluctuations under strain.

While the leading instability has, of course, its origin in the momentum
space, it is instructive to look at it also from the real space point of view.
As discussed in our previous paper [34], the Fermi-surface driven instability,
when mapped onto the nearest-neighbor Heisenberg Hamiltonian, results in three
sizeable exchange parameters, corresponding to the ferromagnetic coupling
along the 110 bond, $J_{\rm 110}$ and two antiferromagnetic couplings, along
the 100 and 200 bonds, so that $|J_{\rm 200}|>|J_{\rm 110}|>|J_{\rm 100}|$.
Note that single $J_{\rm 100}$ generate a ${\bf q}=(1/2,0)$ order, and $J_{\rm
200}$ a ${\bf q}=(1/4,0)$ order, _etc_. In Ref. [34] we estimated, using the
real-space Disordered Local Moments method, the effect of uniaxial strain
(without reoptimizing the atomic position) on the parameters $J$s, and found
that only $J_{\rm 100}$ is affected, by splitting into different $J_{\rm 100}$
and $J_{\rm 010}$. For sufficiently large strain, $J_{\rm 100}$ dictates the
antiferromagnetic order along $x$, that is, $q_{x}=1/2$, while (still the
largest) $J_{\rm 200}$ forces $q_{y}$ to get close to 1/4, thus promoting the
$(1/2,1/4)$ order. However, as discussed above, direct calculations show that
this limit is never achieved in the considered stress range, albeit the
leading instability shifts from (1/3,1/3) to this general direction. Thus, the
evolution of the instability q vector can be roughly described as the
competition between the Fermi surface driven instability and the nearest
neighbor superexchange.

Figure 3:  A toy model illustrating that within the SCR theory, the overall
suppression of magnetism may, paradoxically, lead to the establishment of
long-range magnetic order. The figure shows a model magnetic Hamiltonian, that
is, the energy as a function of a magnetic order parameter $M$. The mean-field
theory, in this particular model and range of parameters (see the main text),
yields a magnetic ground state for both unstrained and strained cases (solid
lines). On the other hand, with SCR theory, where the spin-fluctuations are
included per Eq. 2, one obtains a magnetic ground state only for the strained
case.

We can illustrate the physics of the recovery of magnetism under uniaxial
strain on a simple toy model (Fig. 3). Let us assume that the energy of the
SDW in the unstrained state in the mean field theory (MFT) is a simple
quadratic polynomial of its amplitude: $E(m)=-m^{2}+2m^{4}.$ The MFT ground
state is then $m=1/2.$ If we take Gaussian spin fluctuations of $\xi=0.6$,
then, per Eq.2, these will destabilize the static order and stabilize a
nonmagnetic ground state. Let us now assume that under strain, the quadratic
coefficient got reduced by a factor of two (a reduced tendency towards
magnetism), $E_{strain}(m)=m^{2}/2+2m^{4}.$ This would shift the MFT solution
to $m=1/2\sqrt{2},$ and the magnetic stabilization energy will be reduced from
$1/8$ ($E(m=1/2)$) to $1/32$ ($E_{strain}(m=1/2\sqrt{2})$). If we now assume
that $\xi$ has also been reduced by a factor of two, from $0.6$ to $0.3,$ the
SCR solution will still be magnetic, $m\approx 0.42,$
$E_{strain}\approx-0.06.$.

_Summary and discussions._ Our DFT calculations show that, as expected, the
stress generally weakens the tendency toward magnetic order in Sr2RuO4. That
is to say, the magnetic ground state is less stable in the uniaxially strained
case than in the unstrained one on the mean-field level. However, at the same
time, the propensity of magnetic order to be destroyed by quantum spin
fluctuations becomes weaker and allows the establishment of long-range order
in the strained system. The competition between the two effects can be
understood as follows: The mean square amplitude of spin fluctuations $\langle
M(\mathbf{q})^{2}\rangle$ decreases much faster with applied strain than
static magnetic moment formation energy. In the parlance of Moriya’s SCR
theory, the coefficient $b$ in Eq. 1 is negative, indicating the magnetic
tendency, and becomes less negative with increasing the uniaxial strain.
However, the strength of the spin fluctuation quantified by the parameter
$\xi$, $also$ decreases with strain and, apparently, varies faster than $b.$
As a result, up to some critical stress, determined experimentally as $\sim$
0.8 GPa, $\tilde{b}=b+\frac{5}{3}c\xi^{2}$ remains positive and becomes
negative afterward.

A corollary of this picture is that under further straining, the ordered
magnetism will be suppressed again due to the further reduction of the
magnetic tendency - a prediction that should be possible to verify
experimentally.

It is of note that the position of the MFT instability (or, equivalently, of
the maximum in spin susceptibility) shifts with strain from its unstrained
position of $\mathbf{q}=(0.29,0.29)$ toward $\mathbf{q}=(1/2,1/4),$ a new
suggested nesting for strained case [33]. But, still, $\mathbf{q}_{SDW}$
remains strongly incommensurate, $\mathbf{q\approx}(0.34,0.20),$ even at the
strain of several GPa, and this evolution does not reflect the changes in any
types of nestings. The origin of the deviation in $\mathbf{q}_{SDW}$, and its
correlation with the nesting vector is also a topic of further studies. While
direct verification of this prediction by neutron scattering is questionable,
there might be observable indirect manifestations.

Our investigations on the unusual emergence of magnetism in Sr2RuO4 can offer
crucial insights into other unconventional superconductors where the pairing
mechanism is attributed to the spin-fluctuations.

_Acknowledgments._ We thank Stuart Brown for fruitful discussions. This work
was supported by the National Research Foundation of Korea(NRF) funded by the
Ministry of Science and ICT (No. 2022M3H4A1A04074153). B.K. also acknowledges
support by NRF grant No. 2021R1C1C1007017 and KISTI supercomputing Center
(Project No. KSC-2021-CRE-0605). B.K, S.K., and C.F is supported by Korea-
Austria Joint Mobility Projects (NRF-2019K1A3A1A18116063). I.I.M. acknowledges
support from the U.S. Department of Energy through the grant No. DE-SC0021089.

## References

  * [1] A. Chronister, A. Pustogow, N. Kikugawa, D. A. Sokolov, F. Jerzembeck, C. W. Hicks, A. P. Mackenzie, E. D. Bauer, and S. E. Brown, Evidence for even parity unconventional superconductivity in Sr2RuO4, Proc. Natl. Acad. Sci. USA 118, e2025313118 (2021).
  * [2] A. N. Petsch, M. Zhu, M. Enderle, Z. Q. Mao, Y. Maeno, I. I. Mazin, and S. M. Hayden, Reduction of the Spin Susceptibility in the Superconducting State of Sr2RuO4 Observed by Polarized Neutron Scattering, Phys. Rev. Lett. 125, 217004 (2020)
  * [3] A. Steppke, L. Zhao, M. E. Barber, T. Scaffidi, F. Jerzembeck, H. Rosner, A. S. Gibbs, Y. Maeno, S. H. Simon, A. P. Mackenzie, and C. W. Hicks, Strong peak in Tc of Sr2RuO4 under uniaxial pressure, Science 355, 6321 (2017).
  * [4] E. Hassinger, P. Bourgeois-Hope, H. Taniguchi, S. René de Cotret, G. Grissonnanche, M. S. Anwar, Y. Maeno, N. Doiron-Leyraud, and L. Taillefer, Vertical Line Nodes in the Superconducting Gap Structure of Sr2RuO4, Phys. Rev. X 7, 011032 (2017).
  * [5] O. Gingras, N. Allaglo, R. Nourafkan, M. Côté, and A.-M. S. Tremblay, Superconductivity in correlated multiorbital systems with spin-orbit coupling: Coexistence of even- and odd-frequency pairing, and the case of Sr2RuO4, Phys. Rev. B 106, 064513 (2022).
  * [6] A. T. Rømer, P. J. Hirschfeld, and B. M. Andersen, Superconducting state of Sr2RuO4 in the presence of longer-range Coulomb interactions, Phys. Rev. B 104, 064507 (2021).
  * [7] I. Žutić and I. Mazin, Phase-Sensitive Tests of the Pairing State Symmetry in Sr2RuO4, Phys. Rev. Lett. 95 217004 (2005).
  * [8] Han Gyeol Suh, Henri Menke, P. M. R. Brydon, Carsten Timm, Aline Ramires, and Daniel F. Agterberg, Stabilizing even-parity chiral superconductivity in Sr2RuO4, Phys. Rev. Res. 2, 032023(R) (2020).
  * [9] S. A. Kivelson, A. C. Yuan, B. Ramshaw, R. Thomale, A proposal for reconciling diverse experiments on the superconducting state in Sr2RuO4, npj Quantum Mater. 5, 43 (2020).
  * [10] C. W. Hicks, D. O. Brodsky, E. A. Yelland, A. S. Gibbs, J. A. N. Bruin, M. E. Barber, S. D. Edkins, K. Nishimura, S. Yonezawa, Y. Maeno, and A. P. Mackenzie, Strong increase of Tc of Sr2RuO4 under both tensile and compressive strain. Science 344, 283 (2014).
  * [11] Y. Luo, A. Pustogow, P. Guzman, A. P. Dioguardi, S. M. Thomas, F. Ronning, N. Kikugawa, D. A. Sokolov, F. Jerzembeck, A. P. Mackenzie, C. W. Hicks, E. D. Bauer, I. I. Mazin, and S. E. Brown, Normal state 17O NMR studies of Sr2RuO4 under uniaxial stress, Phys. Rev. X 9, 021044 (2019).
  * [12] I. M. Lifshitz, Anomalies of electron characteristics of a metal in the high pressure region, Sov. Phys. JETP 11, 1130 (1960).
  * [13] M. E. Barber, A. S. Gibbs, Y. Maeno, A. P. Mackenzie, and C. W. Hicks, Resistivity in the Vicinity of a van Hove Singularity: Sr2RuO4 under Uniaxial Pressure, Phys. Rev. Lett. 120 076602 (2018).
  * [14] V. Sunko, E. Abarca Morales, I. Marković, M. E. Barber, D. Milosavljević, F. Mazzola, D. A. Sokolov, N. Kikugawa, C. Cacho, P. Dudin, H. Rosner, C. W. Hicks, P. D. C. King, and A. P. Mackenzie, Direct observation of a uniaxial stress-driven Lifshitz transition in Sr2RuO4, npj Quantum Mater. 4, 46 (2019)
  * [15] V. Grinenko, S. Ghosh, R. Sarkar, J.-C. Orain, A. Nikitin, M. Elender, D. Das, Z. Guguchia, F. Brückner, M. E. Barber, J. Park, N. Kikugawa, D. A. Sokolov, J. S. Bobowski, T. Miyoshi, Y. Maeno, A. P. Mackenzie, H. Luetkens, C. W. Hicks, and H.-H. Klauss, Split superconducting and time-reversal symmetry-breaking transitions in Sr2RuO4 under stress, Nat. Phys. 17, 748 (2021).
  * [16] Katsuya Shimizu, Tomohiro Kimura, Shigeyuki Furomoto, Keiki Takeda, Kazuyoshi Kontani, Yoshichika Onuki, and Kiichi Amaya, Superconductivity in the non-magnetic state of iron under pressure, Nature 412, 316 (2001).
  * [17] Ronald E. Cohen, I. I. Mazin, and Donald G. Isaak, Magnetic Collapse in Transition Metal Oxides at High Pressure: Implications for the Earth, Science 275, 654 (1997).
  * [18] Bongjae Kim, Minjae Kim, Chang-Jong Kang, Jae-Ho Han, and Kyoo Kim, Competing spin-fluctuations in Sr2RuO4 and their tuning through epitaxial strain, arXiv:2205.11711
  * [19] Rustem Khasanov , Zurab Guguchia, Elvezio Morenzoni, Chris Baines, Aifeng Wang, Xianhui Chen, Zbigniew Bukowski, and Fazel Tafti, Pressure-induced magnetism in the iron-based superconductors AFe2As2 (A=K, Cs, Rb), Phys. Rev. B 102, 140502(R) (2020)
  * [20] T. Moriya, Spin fluctuations in itinerant electron magnetism (Berlin, Springer, 1985).
  * [21] P. Larson, I.I. Mazin, D.J. Singh, Magnetism, Critical Fluctuations and Susceptibility Re normalization in Pd, Phys. Rev. B 69, 064429 (2004)
  * [22] A. Aguayo, I. I. Mazin, and D. J. Singh, Why Ni3Al Is an Itinerant Ferromagnet but Ni3Ga Is Not, Phys. Rev. Lett. 92, 147201 (2004)
  * [23] I. I. Mazin and D.J. Singh, Spin Fluctuations and the Magnetic Phase Diagram of ZrZn2, Phys. Rev. B 69, 020402(R), 2004
  * [24] I. I. Mazin, M. D. Johannes, L. Boeri, K. Koepernik, and D. J. Singh, Problems with reconciling density functional theory calculations with experiment in ferropnictides Phys. Rev. B 78, 085104 (2008).
  * [25] L. Ortenzi, H. Gretarsson, S. Kasahara, Y. Matsuda, T. Shibauchi, K. D. Finkelstein, W. Wu, S.R. Julian, Young-June Kim, I. I. Mazin, L. Boeri, Structural origin of the anomalous temperature dependence of the local magnetic moments in the CaFe2As2 family of materials. Phys. Rev. Lett. 114, 047001 (2015).
  * [26] G. Kresse and J. Hafner, Phys. Rev. B 47, 558 (1993).
  * [27] G. Kresse and J. Furthmüller, Phys. Rev. B 54, 11169 (1996).
  * [28] J. P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996).
  * [29] L. M. Sandratskii, Journal of Physics: Condensed Matter 3, 8565 (1991).
  * [30] I.I. Mazin and D.J. Singh, Competitions in layered ruthenates: ferro- vs. antiferromagnetism and triplet vs. singlet pairing, Phys. Rev. Lett. 82, 4324 (1999).
  * [31] Y. Sidis, M. Braden, P. Bourges, B. Hennion, S. Nishizaki, Y. Maeno, and Y. Mori, Evidence for Incommensurate Spin Fluctuations in Sr2RuO4, Phys. Rev. Lett. 83, 3320 (1999)
  * [32] K. Iida, M. Kofu, N. Katayama, J. Lee, R. Kajimoto, Y. Inamura, M. Nakamura, M. Arai, Y. Yosida, M. Fujita, K. Yamada, and S.-H. Lee, Inelastic neutron scattering study of the magnetic fluctuations in Sr2RuO4, Phys. Rev. B 84, 060402(R) (2011).
  * [33] A. T. Rømer, Andreas Kreisel, Marvin A. Müller, P. J. Hirschfeld, Ilya M. Eremin, and B. M. Andersen, Theory of strain-induced magnetic order and splitting of TC and $T_{TRSB}$ in Sr2RuO4, Phys. Rev. B 102, 054506 (2020).
  * [34] B. Kim, S. Khmelevskyi, I. I. Mazin, D. F. Agterberg, and C. Franchini, Anisotropy of magnetic interactions and symmetry of the order parameter in unconventional superconductor Sr2RuO4, npj Quantum Mater. 2, 37 (2017).
  * [35] S. E. Brown, private communication.
  * [36] M. Uhl, L. M. Sandratskii, and J. Kübler, Spin fluctuations in ${\gamma}$-Fe and in Fe3Pt Invar from local-density-functional calculations, Phys. Rev. B 50, 291 (1994)
  * [37] M.D. Johannes and I.I. Mazin, Fermi surface nesting and the origin of Charge Density Waves in metals, Phys. Rev. B 77, 165135 (2008)
  * [38] See Supplementary Materials for detailed Fermi surfaces for each pressure and related nesting vectors.