# CPT-Interp: Continuous sPatial and Temporal Motion Modeling for 4D Medical
Image Interpolation

Xia Li1,2, Runzhao Yang1,3, Xiangtai Li4, Antony Lomax2,5, Ye Zhang2,🖂,
Joachim Buhmann1
1 Department of Computer Science, ETH Zurich
2 Center for Proton Therapy, Paul Scherrer Institut
3 Department of Automation, Tsinghua University
4 S-Lab, Nanyang Technological University
5 Department of Physics, ETH Zurich
<EMAIL_ADDRESS>psi.ch}<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Motion information from 4D medical imaging offers critical insights into
dynamic changes in patient anatomy for clinical assessments and radiotherapy
planning and, thereby, enhances the capabilities of 3D image analysis.
However, inherent physical and technical constraints of imaging hardware often
necessitate a compromise between temporal resolution and image quality. Frame
interpolation emerges as a pivotal solution to this challenge. Previous
methods often suffer from discretion when they estimate the intermediate
motion and execute the forward warping. In this study, we draw inspiration
from fluid mechanics to propose a novel approach for continuously modeling
patient anatomic motion using implicit neural representation. It ensures both
spatial and temporal continuity, effectively bridging Eulerian and Lagrangian
specifications together to naturally facilitate continuous frame
interpolation. Our experiments across multiple datasets underscore the
method’s superior accuracy and speed. Furthermore, as a case-specific
optimization (training-free) approach, it circumvents the need for extensive
datasets and addresses model generalization issues. 111The code will be made
publicly available upon acceptance.

(a) Pure discrete modeling.

(b) Temporal continuous modeling.

(c) Temporal and spatial continuity.

Figure 1: Visualization of motion under different continuity settings. (a) In
pure discrete modeling, the motion arrow can only start from a discrete
location on the image grid and linearly point toward the target location. (b)
Temporal continuity allows for smooth trajectories, encoding the dynamic of
the motion. (c) The proposed Continuous sPatial and Temporal modelling (CPT)
allows the motion to start from any continuous location while following a
smooth curve.

##  1 Introduction

Four-dimensional (4D) medical imaging, which integrates a temporal dimension
with three-dimensional (3D) imaging, has revolutionized clinical diagnostics
and radiotherapy planning [54, 22, 33, 55]. It enables the dynamic observation
of anatomical changes and is critical for precise disease management and
treatment accuracy. In 4D Computed Tomography (4DCT), phase sorting and
binning are employed to categorize images according to different phases of a
patient’s respiratory cycle, effectively reducing motion artifacts. However,
this method inherently limits temporal resolution, necessitating a compromise
between capturing rapid physiological events and maintaining high image
quality [67, 58, 37]. Such compromises often lead to increased radiation
exposure, posing significant clinical concerns. Similarly, 4D Magnetic
Resonance Imaging (4DMR) faces challenges with slow data acquisition rates,
leading to motion artifacts like motion blur during prolonged scanning
sessions [31, 65, 49]. Achieving an optimal balance between temporal and
spatial resolution is crucial for accurately monitoring the anatomic changes
of fast-moving organs such as the heart, lungs, or abdomen, and it remains a
significant challenge in 4DMR.

Frame interpolation is used as a post-processing technique to mitigate the
trade-off between fast monitoring and high spatial resolution. It has been
developed in the domain of video processing to synthesize intermediate frames
effectively, thereby improving the fluidity and realism of video playback [57,
23, 62, 5]. Directly applying these video frame interpolation (VFI) algorithms
to 4D medical imaging encounters several formidable obstacles [28]. Firstly,
mainstream VFI algorithms are data-driven approaches that typically rely on
extensive training data to learn the natural video prior, yet they still
suffer from generalization problems when applied to novel domains [14]. This
phenomenon severely aggravates the medical image domain with the diversity of
clinical conditions and sequence variations. Secondly, elevating 2D+t modeling
to 3D+t modeling has proven to pose a significant challenge [38] due to the
substantially increased parameters and training time. Thirdly, previous
algorithms usually assume linear motion patterns and suffer from the forward
warping problem [39]. Limited by the discrete representation, numerical
approximations in the calculation severely impact interpolation precision.

Drawing insights from fluid mechanics [13], we conceptualize tissue motion as
a non-linear continuous process. To effectively capture this dynamic, we adopt
spatial and temporal continuous modeling, as illustrated in Fig. 1.
Specifically, we employ the implicit neural representation (INR) [46] for
spatial modeling, which technique provides a continuous representation of the
Displacement Vector Field (DVF) [59]. This approach circumvents the
limitations inherent in discrete grid-based representation and requires only a
compact set of parameters, effectively addressing the challenge of increased
parameters. Building upon this foundation, our model shifts its focus from
merely estimating the DVF to learning the Velocity Vector Field (VVF),
introducing an element of temporal continuity. By integrating both spatial and
temporal continuity, our framework seamlessly bridges the Eulerian and
Lagrangian specifications. This integration facilitates the modeling of non-
linear motion and naturally bypasses forward warping, effectively overcoming
the associated challenges. Moreover, our approach does not rely on large-scale
pre-training and allows for case-specific optimization. This feature
alleviates the burden of collecting large-scale medical datasets.

We have efficiently implemented and rigorously evaluated our proposed method
using two benchmarks [4, 21]. The experimental results confirm its superior
performance, surpassing all previous methods. Moreover, our approach employs
straightforward operators and achieves remarkable speeds, completing tasks in
less than 20 seconds. This efficiency significantly surpasses that of
traditional optimization-based methods. Such rapid processing holds
substantial potential for clinical application, highlighting the practicality
and effectiveness of our method in real-world medical settings.

##  2 Related Works

Deformable Image Registration. Deformable image registration (DIR) is a
critical component in various medical imaging applications [44, 40, 30, 48, 3,
10], enabling the alignment of images from different time points, modalities,
or subjects. DIR involves estimating the dense displacement vector field (DVF)
between sequential images. It is challenging due to the lack of definitive
ground truth [15] and the high degree of freedom [47] in DVF optimization.
Traditional methods like optical flow [41, 64] and elastic models [32, 11]
have evolved into more advanced techniques such as the Demons algorithm [53,
52] and B-splines [45, 29], which, despite improved performance, still require
extensive computational resources [12] and expert tuning [16]. Recently, deep
learning approaches have significantly advanced DIR. Methods like VoxelMorph
[1] use unsupervised learning to improve accuracy and speed, while SynthMorph
[20] extends these capabilities to multi-modal image registration. Transmorph
[7] enhances spatial correspondence modeling using transformer models.
However, these deep learning methods still face significant limitations,
including the need for extensive training datasets [66] and difficulties in
generalizing to new data or imaging modalities. Implicit Neural Representation
(INR) [36, 46] has revolutionized DIR by combining iterative optimization
benefits with neural network efficiency, avoiding the need for heavy
supervised pre-training [43]. IDIR [59] pioneered this trend by using INR to
fit the DVF accurately, and ccDIR [51] enhanced it with a cycle consistency
loss to achieve better diffeomorphism. Building on these methods, CPT-DIR [34]
models the registration process as a spatial-temporal continuous flow using an
MLP network, effectively addressing sliding boundary issues and large
deformations while adapting quickly to new cases without extensive hyper-
parameter tuning. In this study, we draw inspiration from CPT-DIR [34] to
continuously model the tissue motion using INR.

Video Frame Interpolation. Video Frame Interpolation (VFI) is used to create
intermediate frames between consecutive video frames. Traditional VFI methods,
which rely on optical flow for motion estimation [57, 23, 62, 5], often
produce artifacts in challenging scenarios like abrupt brightness changes,
large motions, and occlusions. Recent advancements in deep learning have
introduced more robust approaches [50, 9, 8, 25, 39, 42, 61, 69] to improve
motion estimation accuracy and interpolation quality. Despite advancements,
applying VFI algorithms to 4D medical imaging is challenging. VFI methods
require extensive training data, which is problematic in the diverse medical
imaging domain [14]. Transitioning from 2D+t to 3D+t increases complexity and
computational demands [38], necessitating more parameters and longer training
times. Additionally, many VFI algorithms assume linear motion patterns and
suffer from forward warping problems [39], leading to numerical approximation
errors and reduced interpolation precision [28]. Thus, current VFI techniques
are not suitable for 4D medical image interpolation, underscoring the need for
specialized solutions tailored to this field’s challenges.

4D Medical Image Interpolation. To address the above challenges, frame
interpolation methods specifically focused on 4D medical images are driven
[18]. SVIN [17] employs a dual-network strategy for capturing and
interpolating volumetric motion; however, its application is limited by its
reliance on extensive radiation, lengthy imaging processes, and the
availability of ground-truth intermediate images for training. MPVF [56] uses
multi-pyramid voxel flows to address periodic motion in organ structures, but
struggles with non-periodic motions due to its discrete representation of
dynamic biological processes. UVI-Net [28] avoids using intermediate frames
through a cycle consistency model, enhancing image fidelity from limited data.
However, its reliance on linear motion assumptions can lead to spatial
distortion and inaccuracies in capturing complex physiological movements.
While these techniques are pioneering, they underscore the ongoing challenge
of accurately modelling physiological changes’ continuous and complex nature
over time. Our method addresses these shortcomings by spatially and temporally
modelling the continuous motion of tissues.

##  3 Method

Overview. We first review how previous methods work on deformable image
registration (DIR) and video frame interpolation (VFI) and analyze the
limitations caused by discrete motion modeling (Sec. 3.1). Next, we elaborate
on our approach to modeling the motion continuously, both spatially and
temporally (Sec. 3.2 and Sec. 3.3). We describe how it bridges the Eulerian
and the Lagrangian specifications and the corresponding benefits. Finally, we
derive its usage to continuous frame interpolation (Sec. 3.4).

###  3.1 Conventional Frame interpolation by Image Registration

(a) Example of nonlinear motion.

(b) Example of linear motion.

(c) Problematic forward warping.

Figure 2: (a) In the nonlinear motion shown, $t*\phi_{0\to 1}$ does not give
an accurate $\phi_{0\to t}$. They start at the same point, but there are
errors in direction and magnitude. (b) In the linear motion shown,
$(1-t)*\phi_{0\to 1}$ does not give an accurate $\phi_{t\to 1}$ either. They
are in the same direction, but the modeling errors are in the starting point
and magnitude. (c) In the overlapping scenario shown, even with an accurate
$\phi_{0\to t}$, ambiguities arise during the forward warping of $I_{0}$ : i)
What values should be assigned to the two upper pixels that are not targeted
by the warping? ii) What values should be assigned to the two lower pixels
that are mapped from multiple sources?

Both deformable image registration and frame interpolation rely on motion
modeling, typically achieved by estimating the displacement field $\phi_{0\to
1}$, which is linear in time. Given a pair of 3D images $I_{0}$ (fixed) and
$I_{1}$ (moving), where voxel $z_{0}\in\mathbb{Z}^{3}$, a DIR algorithm
estimates the DVF $\phi_{0\to 1}$ that maps the coordinates in $I_{0}$ to
their corresponding locations in $I_{1}$, aligning the twos images by
minimizing the difference between the reference image $I_{0}$ and the warped
moving image $I_{1}\circ\phi_{0\to 1}$. Since this problem is ill-posed in
general, regularization terms are introduced to constrain the solution space
of $\phi_{0\to 1}$. The optimization problem for DIR can be formulated as:

$\phi^{*}=\underset{\phi}{\arg\min}D(I_{0},I_{1}\circ\phi_{0\to 1})+\lambda
R(\phi_{0\to 1}),$ (1)

where $D$ measures the image similarity and is usually implemented as the Sum
of Squared Differences (SSD), Normalized Cross Correlation (NCC), or
Normalized Gradient Flow (NGF). Additionally, $R$ with strength $\lambda$ is
the regularization term; possible choices for $R$ include the classic L1 loss,
TV loss, etc. Conventional methods directly optimize the DVF map $\phi_{0\to
1}$ or use condensed representations like B-splines [68], while deep learning-
based methods [1, 20, 7, 45, 53] learn a mapping function $f:I_{0}\times
I_{1}\to\phi_{0\to 1}$ from a large set of data pairs.

In both paradigms, $\phi_{0\to 1}$ is represented discretely and
directionally. This representation implies that $\phi_{0\to
1}:\mathbb{Z}^{3}\to\mathbb{R}^{3}$ maps a coordinate $z_{0}$ on the grid
(where $z_{0}\in\mathbb{Z}^{3}$) of $I_{0}$ to a continuous location $x_{1}$
in $I_{1}$. The trilinear interpolation operator is required to calculate the
displacement vector for a continuous location $x_{0}\in\mathbb{R}^{3}$.

With the estimated DVF $\phi_{0\to 1}$, we naturally have linear motion
modeling. Built upon this, frame interpolation estimates the intermediate
frame $I_{t},t\in(0,1)$. In previous approaches, two directions have been
pursued: Forward Warping: The DVF for the intermediate frame is approximated
by $\phi_{0\to t}\approx t\cdot\phi_{0\to 1}$, and a trained neural network
handles the challenging task of forward warping $I_{0}\bullet\phi_{0\to t}$.
This model introduces significant errors: a) As shown in Fig. 2, $\phi_{0\to
t}\approx t\cdot\phi_{0\to 1}$ is not precise when the motion is nonlinear; b)
Forward warping suffers from holes and multiple sources mapping [39], as
illustrated in Fig. 2. Backward Warping: Researchers adopt the backward
warping $I_{1}\circ\phi_{t\to 1}$, but this method suffers from inaccuracies
in approximating $\phi_{t\to 1}$. As shown in Fig. 2, $\phi_{t\to
1}\neq(1-t)\cdot\phi_{0\to 1}$, even for the linear motion. This discrepancy
arises because the vector $(1-t)\cdot\phi_{0\to 1}$ starts from a continuous
location $x_{t}\in\mathbb{R}^{3}$, while $\phi_{t\to 1}$ for warping requires
the vector to start from a discrete grid-point $z_{t}\in\mathbb{Z}^{3}$.

###  3.2 Spatial Continuous Modeling

(a) Siren network for spatial continuity.

(b) ODE solver for temporal continuity.

Figure 3: The proposed Continuous sPatial and Temporal (CPT) modeling. Given
continuous input locations $x_{0}$, the Siren network $h_{\omega}$ estimates
their corresponding velocities $\upsilon_{t}$ at time $t$. The full trajectory
is then calculated by the ODE solver over $\upsilon_{t},t\in[0,1]$, yielding
the displacement vector $\varphi_{0\to 1}$ for optimizing the deformable image
registration (DIR). The CPT motion modeling then naturally enables continuous
image interpolation.

We believe the above dilemma faced in deformable image registration (DIR) and
video frame interpolation (VFI) estimation arises from the discrete modeling
in both space and time. Only through the spatial continuous modeling can we
precisely estimate the continuous DVF $\varphi_{0\to
1}:\mathbb{R}^{3}\to\mathbb{R}^{3}$. To achieve this goal, we leverage the
adaptivity of INR, which has achieved great success in 3D reconstruction.
Instead of using INR to reconstruct the 3D image, we use it to model the
displacement, resulting in $\varphi_{0\to 1}(x)=g_{\theta}(x)+x$, where
$g_{\theta}$ is implemented as a Siren[46] network, as in IDIR[59] and CPT-
DIR[34]. The optimization target for DIR then becomes:

$\theta^{*}=\underset{\theta}{\arg\min}D(I_{0},I_{1}\circ\varphi_{0\to
1})+\lambda_{1}R_{1}(\varphi_{0\to 1})+\lambda_{2}R_{2}(\theta),$ (2)

where regularization is also applied to the network parameters $\theta$. The
process is illustrated in Fig. 3, where the network is $g_{\theta}$ instead in
this modeling. This approach avoids the numerical accuracy sacrifice from
trilinear interpolation as above. Additionally, as both input and output are
continuous coordinates, the DVF reversing approximations become feasible. In
conclusion, spatial continuity eliminates the discrete and directional
limitations of conventional representations. However, achieving continuous
frame interpolation remains elusive, as accurately estimating $\varphi_{t\to
1}$ is still challenging.

###  3.3 Temporal Continuous Modeling

In LDDMM-based DIR methods, large deformations are decomposed into smaller
steps to ease the burden of correspondence searching. Although we have not
attempted to solve the problem of large deformations, we have found a solution
for temporal continuity by decomposition. To achieve this, we estimate the
velocity vector field (VVF) $\upsilon$ instead of the DVF. The VVF can be
estimated as a function of $t$ as $\upsilon_{t}(x)=h_{\omega}(x,t)$ or
independent of $t$ as $\upsilon(x)=h_{\omega}(x)$, where $\omega$ is the
parameter for the Siren network $h$. The DVF is then calculated from the
temporal integration of the VVF:

$\varphi_{0\to 1}(x_{0})=x_{0}+\int_{0}^{1}h_{\omega}(x_{t},t)dt,$ (3)

where $x_{t}$ represents the particle or tissue’s location at time $t$. For
practical implementation, this needs to be estimated discretely in time, using
the Euler method as:

$\varphi_{0\to 1}(x_{0})=x_{0}+\sum_{t=0}^{1}h_{\omega}(x_{t},t)\frac{1}{T},$
(4)

where $T$ is the maximum number of steps for the ODE solver; typically, more
steps yield less loss. As illustrated in Fig. 3, this can also be recursively
calculated as:

$\varphi_{0\to 1}(x_{0})=x_{1};\quad
x_{t+\frac{1}{T}}=x_{t}+h_{\omega}(x_{t},t)\frac{1}{T}.$ (5)

Since the VVF can be reversed directly, we can easily estimate the reversed
DVF $\phi_{1\to 0}(x_{1})=x_{1}-\sum_{t=1}^{0}h_{\omega}(x_{t},t)\frac{1}{T}$.
Thus, the optimization target for DIR becomes:

$\begin{split}\omega^{*}&=\underset{\omega}{\arg\min}D(I_{0},I_{1}\circ\varphi_{0\to
1})+\lambda_{1}R_{1}(\varphi_{0\to
1})+\lambda_{2}\sum_{t=0}^{1}R_{2}(\upsilon_{t})+\lambda_{3}R_{3}(\omega)\\\
&+D(I_{1},I_{0}\circ\varphi_{1\to 0})+\lambda_{1}R_{1}(\varphi_{1\to
0}),\end{split}$ (6)

which also constrains the diffeomorphism property of the mapping.

Our proposed method distinguishes us from ODE-based [63, 60, 35] and LDDMM-
based [2] methods, which still rely on an Eulerian specification that models
only from the coordinates’ perspective. These methods require discrete steps
to integrate over time, involving successive warping of the VVF, such as
$\phi_{0\to 1}\approx\phi_{0\to 0.5}\circ\phi_{0.5\to 1}$. The more steps
involved, the more trilinear interpolation is needed, decreasing precision. In
contrast, our implementation is the first to seamlessly bridge the Eulerian
and Lagrangian specifications, accommodating both the coordinates’ perspective
and the parcels’ perspective. The first perspective allows its easy formation
as a mapping network, while the second provides the possibility for
parcel/tissue tracking.

###  3.4 Continuous Frame Interpolation

Equipped with properties of spatial and temporal continuity, our motion model
can derive the displacement vector for any time $t_{0}\in(0,1)$ to any time
$t_{1}\in(0,1)$ from any location $x_{0}$ using an ODE solver (such as the
Euler method [19])

$\varphi_{t_{0}\to t_{1}}(x_{t_{0}})\approx
x_{t_{0}}+\sum_{t=t_{0}}^{t_{1}}\upsilon_{t}\frac{1}{T},$ (7)

where the loss of numerical precision only comes from $\delta t$.

These formulations enable us to estimate $\varphi_{t\to 0}$ and $\varphi_{t\to
1}$ more precisely than previous methods. We avoid the cumbersome forward
warping because we don’t utilize $\varphi_{0\to t}$ and $\varphi_{1\to t}$.
Instead, we calculate frame interpolation at time $t$ using the weighted
average of $I_{1}\circ\varphi_{t\to 1}$ and $\cdot I_{0}\circ\varphi_{t\to 0}$
as:

$I_{t}=w_{1}\cdot I_{1}\circ\varphi_{t\to 1}+w_{0}\cdot
I_{0}\circ\varphi_{t\to 0}.$ (8)

The naive approach would simply average the weights ($w_{0}$ and $w_{1}$).
However, we choose to adopt linear weights ($w_{1}=t,w_{0}=(1-t)$). It is
worth noting that in our approach, no post-processing or so-called refinement
network is adopted.

##  4 Experiments

###  4.1 Datasets and Evaluation Metrics

We evaluate the performance of CPT-Interp on two benchmarks: ACDC [4] and
4D-Lung [21] datasets. Additionally, we evaluate the models trained from
4D-Lung over the DIRLab dataset [6] to evaluate the generalization ability.
The ACDC dataset is a cardiac dataset with 4DMR images, averaging $8.80\pm
2.48$ frames between the end-systolic and end-diastolic phases. The last 10
cases are used as the testing set. The 4D-Lung dataset contains 4DCT images,
with 4 phases between the end-inspiratory and the end-expiratory phases. We
follow the prepossessing in UVI-Net [28] and use the last 14 cases as the
testing test. The DIRLab dataset comprises 4D CT scans from 10 patients
undergoing treatment for esophageal malignancies. Each patient dataset
consists of 10 respiratory phases, spanning from 0% to 90% of the respiratory
cycle in increments of 10%. For the evaluation metrics, we include Peak
Signal-to-Noise Ratio (PSNR), Normalized Cross Correlation (NCC), Structural
Similarity Index Measure (SSIM), and Normalized Mean Squared Error (NMSE).

###  4.2 Implementation Details

We implement $h_{\omega}$ as a 4-layer Siren network, with the hyper-parameter
$\omega_{0}$ (defined in Siren [46]) set t0 48 and the layer width set to
$256$. The output is multiplied by the size of the voxel to guarantee the
initial estimation of the $\varphi$ is within [-voxel, voxel]. The $T$ used in
the ODE solver is set to the number of intermediates frames plus 1, and
$\beta$ used in weighted merging is set to $4$. We linearly warm up the
learning rate for the first 500 steps, adopt a cosine learning rate scheduler
to stabilize the convergence, and train the network for 3,000 steps, each with
a mini-batch of 10,000 voxels. To speed up optimization, we only used the
regularization term $R_{2}$ in Eq. 6 as $\sum_{t}\|\upsilon_{t}\|^{2}$ and
neglect $R_{1}$ and $R_{3}$, with $\lambda_{2}$ set as 1e-4. We implement CPT-
Interp with Pytorch on an NVIDIA RTX 4090 GPU.

Table 1: Quantitative comparison of frame interpolation results. All metrics are averaged over all intermediate frames and repeated three times. “TTO” indicates testing-time optimization. NMSE is presented in units of $10^{-2}$. The best results are bolded, and the second-best are underlined. Supervised | Training | Method |  Cardiac[4] |  Lung[21]
---|---|---|---|---
|  PSNR$\uparrow$ |  NCC$\uparrow$ |  SSIM$\uparrow$ |  NMSE$\downarrow$ |  PSNR$\uparrow$ |  NCC$\uparrow$ |  SSIM$\uparrow$ |  NMSE$\downarrow$
Sup | Need |  SVIN [17] | 32.51 | 0.559 | 0.972 | 2.930 | 30.99 | 0.312 | 0.973 | 0.852
Sup | Need |  MPVF [56] | 33.15 | 0.561 | 0.971 | 2.435 | 31.18 | 0.310 | 0.972 | 0.761
Unsup | Need |  VM [1] | 31.02 | 0.555 | 0.966 | 4.254 | 32.29 | 0.316 | 0.977 | 0.641
Unsup | Need |  TM [7] | 30.45 | 0.547 | 0.958 | 4.826 | 30.92 | 0.313 | 0.973 | 0.786
Unsup | Need |  Fourier-Net+ [24] | 29.98 | 0.544 | 0.957 | 5.503 | 30.26 | 0.308 | 0.971 | 0.959
Unsup | Need |  R2Net [26] | 28.59 | 0.509 | 0.930 | 7.281 | 29.34 | 0.294 | 0.962 | 1.061
Unsup | Need |  DDM [27] | 29.71 | 0.541 | 0.956 | 5.007 | 30.37 | 0.308 | 0.971 | 0.905
Unsup | Need |  UVI-Net [28] | 33.59 | 0.565 | 0.978 | 2.384 | 34.00 | 0.320 | 0.980 | 0.552
TTO | Free |  IDIR [59] | 31.56 | 0.557 | 0.968 | 3.806 | 32.91 | 0.321 | 0.980 | 0.586
TTO | Free | CPT-Interp | 33.85 | 0.580 | 0.979 | 2.125 | 34.03 | 0.321 | 0.981 | 0.518
Table 2: Comparisons across different levels of continuity. “Spat Cont.” indicates spatial continuity only modeling, and “Temp Cont.” indicates the temporal continuous modeling. Spat Cont. | Temp Cont. | One-way | Two-way
---|---|---|---
PSNR $\uparrow$ |  NCC$\uparrow$ |  SSIM$\uparrow$ |  NMSE$\downarrow$ |  PSNR$\uparrow$ |  NCC$\uparrow$ |  SSIM$\uparrow$ |  NMSE$\downarrow$
- | - | 30.71 | 0.554 | 0.959 | 4.335 | 33.23 | 0.575 | 0.975 | 2.476
✓ | - | 31.89 | 0.566 | 0.969 | 3.138 | 33.30 | 0.577 | 0.977 | 2.407
- | ✓ | 30.70 | 0.554 | 0.959 | 4.339 | 33.61 | 0.578 | 0.977 | 2.279
✓ | ✓ | 32.21 | 0.568 | 0.970 | 2.849 | 33.85 | 0.580 | 0.979 | 2.125

###  4.3 Comparisons with Previous Methods

We compare CPT-Interp with previous methods on 4D medical image interpolation,
including supervised methods like SVIN [17] and MPVF [56], unsupervised
methods like VM [1], UVI-Net [28] and etc., as well as the testing-time
optimization (TTO) method like IDIR [59]. As shown in Table 1, CPT-Interp
outperforms all previous methods across all metrics. Additionally, CPT-Interp
requires no further post-processing or refinement network. Moreover, the TTO
only takes 1.96s in ACDC, and inference for all intermediate frames takes
approximately 1.23s, which is much faster than the instance-specific
optimization time in UVI-Net (70 s) and TTO in IDIR (1 min).

###  4.4 Ablation Studies

In this subsection, we analyze the contribution of each component and quantify
the choices of hyper-parameters. All the following results are from 10 testing
cases from the ACDC dataset.

Significance of Continuity. We conduct an ablation study to evaluate the core
of our work-continuity. We considered the effects of both spatial and temporal
continuity by setting up four comparative scenarios. For spatial discrete
models, we use 1st-order B-splines [68] as the representation, keeping the
other settings the same as ours. For the temporal discrete model, we simply
approximate $\phi_{t\to 1}\approx(1-t)\cdot\phi_{0\to 1}$ during inference. As
shown in Table. 2, the joint continuous one “Spat + Temp” achieves the best
performance across all metrics. This comparison validates the significance of
continuity in both space and time domains. Moreover, temporal continuity
brings less improvement only when spatial continuity is present. This supports
our analysis in Sec. 3.3 that spatial continuity lays the foundation for
temporal continuity, providing a combined Eulerian and Lagrangian
specification. Without spatial continuity, temporal continuity is less
effective.

Optimization and Inference. As formulated in Eqs. 6&8, we optimize the model
from both the forward and the backward directions to interpolate the frames,
leveraging the complementary nature of bidirectional information. We conduct
an ablation study on this assumption and list the results in Table. 3. As
compared between rows 1 & 2, two-way training marginally improves over one-way
training when only adopting one-way inference. Additionally, two-way inference
outperforms one-way inference in any situation, verifying our assumption. For
further evaluation, we use linear merging as the default way for two-way
inference.

Table 3: Ablation studies on the optimization and inference strategy over the ACDC dataset. “One-way” means only the forward DVF $\varphi_{0\to 1}$ is optimized or integrated for optimization and inference, respectively. While “Two-way” means DVFs of both directions are optimized/integrated. No. | Optimization | Time (s) | Inference | Merge | Time (s) |  PSNR $\uparrow$ |  NCC $\uparrow$ |  SSIM $\uparrow$ |  NMSE $\downarrow$
---|---|---|---|---|---|---|---|---|---
1 | One-way |  1.96$\pm$0.53 | One-way | - |  0.64$\pm$0.33 | 32.17 | 0.568 | 0.970 | 2.878
2 | Two-way | 3.78$\pm$1.05 | One-way | - |  0.65$\pm$0.36 | 32.21 | 0.568 | 0.970 | 2.849
4-9 3 | Two-way | Average |  1.23$\pm$0.66 | 33.64 | 0.576 | 0.976 | 2.470
4 | Linear |  1.23$\pm$0.66 | 33.85 | 0.580 | 0.979 | 2.125

(a) Sensitivity of $\lambda$ on one-way inference.

(b) Sensitivity of $\lambda$ on two-way inference.

(c) Sensitivity of $\omega_{0}$ on one-way inference.

(d) Sensitivity of $\omega_{0}$ on two-way inference.

(e) Sensitivity of opt- steps on one-way inference.

(f) Sensitivity of opt- steps on two-way inference.

Figure 4: Ablation studies on the hyper-parameters ($\lambda$, $\omega_{0}$,
and optimization steps) over the ACDC dataset. Results of one-way inference
are shown on the left, and two-way inference results are shown on the right.
The black dotted line denotes the selected hyper-parameters for the final
comparisons.

Sensitivity of Hyper-parameters. We examine the selection of hyper-parameters
and plot the results in Fig. 4. By comparing sub-figures on the left side to
those on the right side, it is evident that two-way inference is less
sensitive to hyper-parameters. This can easily explained by the complementary
nature of bidirectional inference. The hyper-parameter $\lambda$ controls the
strength of the regularization term. In our settings, it constrains the norm
of velocities, indirectly determining the property of motion trajectories: a
too small $\lambda$ may bring a zigzag trajectory, while a too large $\lambda$
may lead to an overly straight line, degrade to the linear motion. Besides,
$\omega_{0}$ in Siren [46] determines the frequency width of the simulated
signal; a small value leads to a low-frequency, over-smooth DVF map, while a
high $\omega_{0}$ may over-parameterize the DVF representation. As validated
in the figure, curves over these two hyper-parameters have single peaks. The
number of optimization steps defines the trade-off between time and accuracy.
The performance over two-way inference goes to a plateau when optimization
steps are over 50. We select 200 steps for all our experiments and ablation
studies on ACDC.

Generalization Ability. As a case-specific optimization approach, CPT-Interp
does not suffer from limitations of training dataset size, nor is it
constrained by domain gaps when adapting to a new dataset. To verify its
generalization advantage, we adopt the unseen DIRLab dataset [6] to test
models trained or hyper-parameters tunned on 4D-Lung. For VM [1] and UVI-Net
[27], the models trained on 4DLung are directly tested. While for IDIR [59]
and CPT-Interp, the same hyper-parameters used in 4D-Lung are directly adopted
for testing-time optimization over the DIRLab dataset. We resized the input
DIRLab images to $128\times 128\times 128$ to align the image sizes with those
of 4D-Lung. The results are shown in Table. 4. The “Size 128” offers an easy
benchmark, while evaluation over the “original size” sets a more strict
setting that matches clinical usage requirements. Due to precision loss during
resizing, VM’s and UVI-Net’s performance drops greatly after resizing back to
the original size. IDIR and CPT-Interp, both as spatial continuous models, are
not limited by image sizes.

Table 4: Evaluation of the generalization ability on the DIRLab dataset. The evaluation is conducted under two settings: 1) Images are resized to $128\times 128\times 128$ to match 4DLung’s setting; 2) Images are resized back to their original sizes for a more stringent evaluation. Method | Direction | Size 128 | Original size
---|---|---|---
3-10 |  PSNR $\uparrow$ |  NCC $\uparrow$ |  SSIM $\uparrow$ |  NMSE $\downarrow$ |  PSNR $\uparrow$ |  NCC $\uparrow$ |  SSIM $\uparrow$ |  NMSE $\downarrow$
VM [1] | Two-way | 31.14 | 0.779 | 0.940 | 0.274 | 27.18 | 0.489 | 0.848 | 0.652
UVI-Net [27] | Two-way | 32.57 | 0.797 | 0.949 | 0.172 | 27.89 | 0.505 | 0.858 | 0.548
IDIR [59] | Two-way | 32.71 | 0.823 | 0.956 | 0.149 | 31.35 | 0.632 | 0.912 | 0.202
CPT-Interp | One-way | 33.38 | 0.832 | 0.962 | 0.145 | 31.92 | 0.641 | 0.918 | 0.196
CPT-Interp | Two-way | 33.76 | 0.840 | 0.966 | 0.131 | 32.34 | 0.655 | 0.926 | 0.176

Visualizations. To qualitatively evaluate the performance of our CPT-Interp
method, we visualize the error maps for the interpolated frames compared to
the ground truth in Fig. 5. In each row, the first four columns represent the
error maps from four approaches, with the ground truth frames shown in the
last column. As illustrated, CPT-Interp consistently produces error maps with
fewer high-error regions compared to the other methods, indicating a closer
approximation to the ground truth.

Figure 5: Visualization of the error maps for the interpolated frames. Results
of VM [1], IDIR [59], UVI-Net [28] and CPT-Interp are compared over ACDC
(upper) and 4D-Lung (lower) datasets.

##  5 Conclusion

In this work, we introduce a novel Continuous sPatial and Temporal (CPT)
modeling approach for 4D medical image interpolation, addressing the
limitations of discrete motion modeling in both spatial and temporal domains.
By leveraging implicit neural representations and integrating an ODE solver,
our method achieves superior accuracy and speed without requiring extensive
training datasets or domain-specific fine-tuning. Through comprehensive
experiments and ablation studies, we demonstrated the contributions of spatial
and temporal continuity, as well as the robustness of our method across
different datasets. CPT-Interp not only outperforms state-of-the-art methods
but also exhibits remarkable generalization ability, making it a promising
solution for clinical applications.

Limitations and Broader Impacts. CPT-Interp, as a case-specific optimization
model, avoids training and generalization issues. In this study, we tuned
hyper-parameters at the dataset level using the validation sets. Future work
is required to explore strategies to automatically select case-specific hyper-
parameters without relying on a validation set or ground-truth intermediate
frames. As a high-performance medical image interpolation approach that does
not require training, it enables detailed dynamic modeling of patients. This
can lead to more accurate diagnoses and improved radiotherapy treatments, and
improve clinical workflows by providing precise motion analysis.

## Acknowledgments and Disclosure of Funding

This project is supported by the interdisciplinary doctoral grant (iDoc
2021-360) from the Personalized Health and Related Technologies (PHRT) of the
ETH domain, Switzerland

## References

  * [1] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca.  Voxelmorph: a learning framework for deformable medical image registration.  TIP, 38(8):1788–1800, 2019.
  * [2] M Faisal Beg, Michael I Miller, Alain Trouvé, and Laurent Younes.  Computing large deformation metric mappings via geodesic flows of diffeomorphisms.  IJCV, 61:139–157, 2005.
  * [3] Edward T Bender, Nicholas Hardcastle, and Wolfgang A Tome.  On the dosimetric effect and reduction of inverse consistency and transitivity errors in deformable image registration for dose accumulation.  Med. Phys., 39(1):272–280, 2012.
  * [4] Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, et al.  Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved?  TMI, 37(11):2514–2525, 2018.
  * [5] Andrés Bruhn, Joachim Weickert, Christian Feddern, Timo Kohlberger, and Christoph Schnorr.  Variational optical flow computation in real time.  TIP, 14(5):608–615, 2005.
  * [6] Richard Castillo, Edward Castillo, Rudy Guerra, Valen E Johnson, Travis McPhail, Amit K Garg, and Thomas Guerrero.  A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets.  Phys. Med. Biol., 54(7):1849, 2009.
  * [7] Junyu Chen, Eric C Frey, Yufan He, William P Segars, Ye Li, and Yong Du.  Transmorph: Transformer for unsupervised medical image registration.  Med. Image Anal., 82:102615, 2022.
  * [8] Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang.  Videoinr: Learning video implicit neural representation for continuous space-time super-resolution.  In CVPR, pages 2047–2057, 2022.
  * [9] Xianhang Cheng and Zhenzhong Chen.  A multi-scale position feature transform network for video frame interpolation.  TCSVT, 30(11):3968–3981, 2019.
  * [10] Indrin J Chetty and Mihaela Rosu-Bubulac.  Deformable registration for dose accumulation.  In Seminars in radiation oncology, volume 29, pages 198–208. Elsevier, 2019.
  * [11] Christos Davatzikos.  Spatial transformation and registration of brain images using elastically deformable models.  Computer Vision and Image Understanding, 66(2):207–222, 1997.
  * [12] Bob D De Vos, Floris F Berendsen, Max A Viergever, Hessam Sokooti, Marius Staring, and Ivana Išgum.  A deep learning framework for unsupervised affine and deformable image registration.  Med. Image Anal., 52:128–143, 2019.
  * [13] Gregory Falkovich.  Fluid mechanics: A short course for physicists.  Cambridge University Press, 2011.
  * [14] Edward Ferdian, Avan Suinesiaputra, David J Dubowitz, Debbie Zhao, Alan Wang, Brett Cowan, and Alistair A Young.  4dflownet: super-resolution 4d flow mri using deep learning and computational fluid dynamics.  Frontiers in Physics, 8:138, 2020.
  * [15] Bernd Fischer and Jan Modersitzki.  Ill-posed medicine—an introduction to image registration.  Inverse problems, 24(3):034008, 2008.
  * [16] Yabo Fu, Yang Lei, Tonghe Wang, Walter J Curran, Tian Liu, and Xiaofeng Yang.  Deep learning in medical image registration: a review.  Phys. Med. Biol., 65(20):20TR01, 2020.
  * [17] Yuyu Guo, Lei Bi, Euijoon Ahn, Dagan Feng, Qian Wang, and Jinman Kim.  A spatiotemporal volumetric interpolation network for 4d dynamic medical image.  In CVPR, pages 4726–4735, 2020.
  * [18] Yuyu Guo, Lei Bi, Dongming Wei, Liyun Chen, Zhengbin Zhu, Dagan Feng, Ruiyan Zhang, Qian Wang, and Jinman Kim.  Unsupervised landmark detection-based spatiotemporal motion estimation for 4-d dynamic medical images.  IEEE Trans. Cybern, 53(6):3532–3545, 2021.
  * [19] Francis Begnaud Hildebrand.  Introduction to numerical analysis.  Courier Corporation, 1987.
  * [20] Malte Hoffmann, Benjamin Billot, Douglas N Greve, Juan Eugenio Iglesias, Bruce Fischl, and Adrian V Dalca.  Synthmorph: learning contrast-invariant registration without acquired images.  TIP, 41(3):543–558, 2021.
  * [21] Geoffrey D Hugo, Elisabeth Weiss, William C Sleeman, Salim Balik, Paul J Keall, Jun Lu, and Jeffrey F Williamson.  Data from 4d lung imaging of nsclc patients.  The Cancer Imaging Archive, 10:K9, 2016.
  * [22] Adam Hutchinson and Pete Bridge.  4dct radiotherapy for nsclc: a review of planning methods.  Journal of Radiotherapy in Practice, 14(1):70–79, 2015.
  * [23] Seong-Gyun Jeong, Chul Lee, and Chang-Su Kim.  Motion-compensated frame interpolation based on multihypothesis motion estimation and texture optimization.  TIP, 22(11):4497–4509, 2013.
  * [24] Xi Jia, Alexander Thorley, Alberto Gomez, Wenqi Lu, Dipak Kotecha, and Jinming Duan.  Fourier-net+: Leveraging band-limited representation for efficient 3d medical image registration.  arXiv preprint arXiv:2307.02997, 2023.
  * [25] Xin Jin, Longhai Wu, Jie Chen, Youxin Chen, Jayoon Koo, and Cheul-hee Hahm.  A unified pyramid recurrent network for video frame interpolation.  In CVPR, pages 1578–1587, 2023.
  * [26] Ankita Joshi and Yi Hong.  R2net: Efficient and flexible diffeomorphic image registration using lipschitz continuous residual networks.  Med. Image Anal., 89:102917, 2023.
  * [27] Boah Kim and Jong Chul Ye.  Diffusion deformable model for 4d temporal medical image generation.  In MICCAI, pages 539–548. Springer, 2022.
  * [28] JungEun Kim, Hangyul Yoon, Geondo Park, Kyungsu Kim, and Eunho Yang.  Data-efficient unsupervised interpolation without any intermediate frame for 4d medical images.  arXiv preprint arXiv:2404.01464, 2024.
  * [29] Stefan Klein, Marius Staring, and Josien PW Pluim.  Evaluation of optimization methods for nonrigid medical image registration using mutual information and b-splines.  TIP, 16(12):2879–2890, 2007.
  * [30] Akila Kumarasiri, Farzan Siddiqui, Chang Liu, Raphael Yechieli, Mira Shah, Deepak Pradhan, Hualiang Zhong, Indrin J Chetty, and Jinkoo Kim.  Deformable image registration based automatic ct-to-ct contour propagation for head and neck adaptive radiotherapy in the routine clinical setting.  Med. Phys., 41(12):121712, 2014.
  * [31] Thomas Küstner, Jiazhen Pan, Christopher Gilliam, Haikun Qi, Gastao Cruz, Kerstin Hammernik, Bin Yang, Thierry Blu, Daniel Rueckert, René Botnar, et al.  Deep-learning based motion-corrected image reconstruction in 4d magnetic resonance imaging of the body trunk.  In 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 976–985. IEEE, 2020.
  * [32] Jan Kybic and Michael Unser.  Fast parametric elastic image registration.  TIP, 12(11):1427–1442, 2003.
  * [33] Guang Li, Deborah Citrin, Kevin Camphausen, Boris Mueller, Chandra Burman, Borys Mychalczak, Robert W Miller, and Yulin Song.  Advances in 4d medical imaging and 4d radiation therapy.  Technology in cancer research & treatment, 7(1):67–81, 2008.
  * [34] Xia Li, Muheng Li, Antony Lomax, Joachim Buhmann, and Ye Zhang.  Continuous spatial-temporal deformable image registration (cpt-dir) for motion modelling in radiotherapy: beyond classic voxel-based methods.  arXiv preprint arXiv:2405.00430, 2024.
  * [35] Xiao Liang, Shan Lin, Fei Liu, Dimitri Schreiber, and Michael Yip.  Orrn: An ode-based recursive registration network for deformable respiratory motion estimation with lung 4dct images.  TBME, 2023.
  * [36] Amirali Molaei, Amirhossein Aminimehr, Armin Tavakoli, Amirhossein Kazerouni, Bobby Azad, Reza Azad, and Dorit Merhof.  Implicit neural representation in medical imaging: A comparative survey.  In ICCV, pages 2381–2391, 2023.
  * [37] Cyril Mory, Guillaume Janssens, and Simon Rit.  Motion-aware temporal regularization for improved 4d cone-beam computed tomography.  Phys. Med. Biol., 61(18):6856, 2016.
  * [38] Reabal Najjar.  Redefining radiology: a review of artificial intelligence integration in medical imaging.  Diagnostics, 13(17):2760, 2023.
  * [39] Simon Niklaus and Feng Liu.  Softmax splatting for video frame interpolation.  In CVPR, pages 5437–5446, 2020.
  * [40] Seungjong Oh and Siyong Kim.  Deformable image registration in radiation therapy.  Radiation oncology journal, 35(2):101, 2017.
  * [41] Karsten Østergaard Noe, Baudouin Denis De Senneville, Ulrik Vindelev Elstrøm, Kari Tanderup, and Thomas Sangild Sørensen.  Acceleration and validation of optical flow based deformable registration for image-guided radiotherapy.  Acta Oncologica, 47(7):1286–1293, 2008.
  * [42] Junheum Park, Jintae Kim, and Chang-Su Kim.  Biformer: Learning bilateral motion estimation via bilateral transformer for 4k video frame interpolation.  In CVPR, pages 1568–1577, 2023.
  * [43] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In ICCV, pages 5865–5874, 2021.
  * [44] Bastien Rigaud, Antoine Simon, Joël Castelli, Caroline Lafond, Oscar Acosta, Pascal Haigron, Guillaume Cazoulat, and Renaud de Crevoisier.  Deformable image registration for radiation therapy: principle, methods, applications and evaluation.  Acta Oncologica, 58(9):1225–1237, 2019.
  * [45] Daniel Rueckert, Paul Aljabar, Rolf A Heckemann, Joseph V Hajnal, and Alexander Hammers.  Diffeomorphic registration using b-splines.  In MICCAI, pages 702–709. Springer, 2006.
  * [46] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  NeurIPS, 33:7462–7473, 2020.
  * [47] Aristeidis Sotiras, Christos Davatzikos, and Nikos Paragios.  Deformable medical image registration: A survey.  TIP, 32(7):1153–1190, 2013.
  * [48] Maria Thor, Jørgen BB Petersen, Lise Bentzen, Morten Høyer, and Ludvig Paul Muren.  Deformable image registration for contour propagation from ct to cone-beam ct scans in radiotherapy of prostate cancer.  Acta Oncologica, 50(6):918–925, 2011.
  * [49] Vijay R Tripathi, Manish N Tibdewal, and Ravi Mishra.  A survey on motion artifact correction in magnetic resonance imaging for improved diagnostics.  SN Computer Science, 5(3):281, 2024.
  * [50] Joost Van Amersfoort, Wenzhe Shi, Alejandro Acosta, Francisco Massa, Johannes Totz, Zehan Wang, and Jose Caballero.  Frame interpolation with multi-scale deep loss functions and generative adversarial networks.  arXiv preprint arXiv:1711.06045, 2017.
  * [51] Louis D Van Harten, Jaap Stoker, and Ivana Išgum.  Robust deformable image registration using cycle-consistent implicit representations.  TIP, 2023.
  * [52] Tom Vercauteren, Xavier Pennec, Aymeric Perchant, and Nicholas Ayache.  Non-parametric diffeomorphic image registration with the demons algorithm.  In MICCAI, pages 319–326. Springer, 2007.
  * [53] Tom Vercauteren, Xavier Pennec, Aymeric Perchant, and Nicholas Ayache.  Diffeomorphic demons: Efficient non-parametric image registration.  NeuroImage, 45(1):S61–S72, 2009.
  * [54] Chunhao Wang and Fang-Fang Yin.  4d-mri in radiotherapy.  In Magnetic Resonance Imaging. IntechOpen, 2019.
  * [55] Chunhao Wang and Fang-Fang Yin.  4d-mri in radiotherapy.  In Lachezar Manchev, editor, Magnetic Resonance Imaging, chapter 6. IntechOpen, Rijeka, 2019.
  * [56] Tzu-Ti Wei, Chin Kuo, Yu-Chee Tseng, and Jen-Jee Chen.  Mpvf: 4d medical image inpainting by multi-pyramid voxel flows.  IEEE J. Biomed. Health Inform., 2023.
  * [57] Manuel Werlberger, Thomas Pock, Markus Unger, and Horst Bischof.  Optical flow guided tv-l 1 video interpolation and restoration.  In Energy Minimization Methods in Computer Vision and Pattern Recognition: 8th International Conference, EMMCVPR 2011, St. Petersburg, Russia, July 25-27, 2011. Proceedings 8, pages 273–286. Springer, 2011.
  * [58] Nicole M Wink, Christoph Panknin, and Timothy D Solberg.  Phase versus amplitude sorting of 4d-ct data.  J. Appl. Clin. Med. Phys., 7(1):77–85, 2006.
  * [59] Jelmer M Wolterink, Jesse C Zwienenberg, and Christoph Brune.  Implicit neural representations for deformable image registration.  In MIDL, pages 1349–1359. PMLR, 2022.
  * [60] Yifan Wu, Tom Z Jiahao, Jiancong Wang, Paul A Yushkevich, M Ani Hsieh, and James C Gee.  Nodeo: A neural ordinary differential equation based optimization framework for deformable image registration.  In CVPR, pages 20804–20813, 2022.
  * [61] Xiaoyu Xiang, Yapeng Tian, Yulun Zhang, Yun Fu, Jan P Allebach, and Chenliang Xu.  Zooming slow-mo: Fast and accurate one-stage space-time video super-resolution.  In CVPR, pages 3370–3379, 2020.
  * [62] Jiangjian Xiao, Hui Cheng, Harpreet Sawhney, Cen Rao, and Michael Isnardi.  Bilateral filtering-based optical flow estimation with occlusion detection.  In ECCV, pages 211–224. Springer, 2006.
  * [63] Junshen Xu, Eric Z Chen, Xiao Chen, Terrence Chen, and Shanhui Sun.  Multi-scale neural odes for 3d medical image registration.  In MICCAI, pages 213–223. Springer, 2021.
  * [64] Deshan Yang, Hua Li, Daniel A Low, Joseph O Deasy, and Issam El Naqa.  A fast inverse consistent deformable image registration method based on symmetric optical flow computation.  Phys. Med. Biol., 53(21):6143, 2008.
  * [65] Maxim Zaitsev, Julian Maclaren, and Michael Herbst.  Motion artifacts in mri: A complex problem with many partial solutions.  Journal of Magnetic Resonance Imaging, 42(4):887–901, 2015.
  * [66] Hubert Dariusz Zając, Natalia Rozalia Avlona, Finn Kensing, Tariq Osman Andersen, and Irina Shklovski.  Ground truth or dare: Factors affecting the creation of medical datasets for training ai.  In AAAI, pages 351–362, 2023.
  * [67] You Zhang, Xiaokun Huang, and Jing Wang.  Advanced 4-dimensional cone-beam computed tomography reconstruction by combining motion estimation, motion-compensated reconstruction, biomechanical modeling and deep learning.  Visual computing for industry, biomedicine, and art, 2(1):23, 2019.
  * [68] Zhengyou Zhang.  Iterative point matching for registration of free-form curves and surfaces.  IJCV, 13(2):119–152, 1994.
  * [69] Kun Zhou, Wenbo Li, Xiaoguang Han, and Jiangbo Lu.  Exploring motion ambiguity and alignment for high-quality video frame interpolation.  In CVPR, pages 22169–22179, 2023.