# TMP: Temporal Motion Propagation for
Online Video Super-Resolution

Zhengqiang Zhang, Ruihuang Li, Shi Guo, Yang Cao, and Lei Zhang Z. Zhang, R.
Li, S. Guo, and L. Zhang are with the Department of Computing, The Hong Kong
Polytechnic University, Hong Kong (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>cslzhang@comp.polyu.edu.hk).
This work is supported by the Hong Kong RGC RIF grant (R5001-18) and the
PolyU-OPPO Joint Innovation Lab.Y. Cao is with the Department of Computer
Science and Engineering, The Hong Kong University of Science and Technology
(e-mail: yangcao.cs@gmail.com).

###### Abstract

Online video super-resolution (online-VSR) highly relies on an effective
alignment module to aggregate temporal information, while the strict latency
requirement makes accurate and efficient alignment very challenging. Though
much progress has been achieved, most of the existing online-VSR methods
estimate the motion fields of each frame separately to perform alignment,
which is computationally redundant and ignores the fact that the motion fields
of adjacent frames are correlated. In this work, we propose an efficient
Temporal Motion Propagation (TMP) method, which leverages the continuity of
motion field to achieve fast pixel-level alignment among consecutive frames.
Specifically, we first propagate the offsets from previous frames to the
current frame, and then refine them in the neighborhood, which significantly
reduces the matching space and speeds up the offset estimation process.
Furthermore, to enhance the robustness of alignment, we perform spatial-wise
weighting on the warped features, where the positions with more precise
offsets are assigned higher importance. Experiments on benchmark datasets
demonstrate that the proposed TMP method achieves leading online-VSR accuracy
as well as inference speed. The source code of TMP can be found at
https://github.com/xtudbxk/TMP.

##  I Introduction

Video super-resolution (VSR) aims to reconstruct a high-resolution (HR) video
sequence from its low-resolution (LR) counterpart. In recent years, due to the
increasing popularity of online video conference, live broadcasting, etc.,
online-VSR has attracted growing attention from researchers [1, 2, 3]. In
online-VSR, the enhancement of the current frame is achieved by using only the
information from past frames in an online manner. Compared with other VSR
problems, the stricter constraint on latency of online-VSR brings more
challenges to frame alignment, which is critical for the efficiency of a VSR
model.

\begin{overpic}[width=433.62pt]{figures/motivation.v9.pdf}
\put(1.5,58.5){\footnotesize{Frame $t$-1}} \put(26.5,58.5){\footnotesize{Frame
$t$}} \put(51.5,58.5){\footnotesize{Frame $t$-1}}
\put(75.5,58.5){\footnotesize{Frame $t$}}
\par\put(-1.5,-1.0){{\begin{tabular}[]{c}Aligned\\\ Frame $t$-1\end{tabular}}}
\put(24.0,-1.0){{\begin{tabular}[]{c}Aligned\\\ Frame $t$\end{tabular}}}
\put(49.0,-1.0){{\begin{tabular}[]{c}Aligned\\\ Frame $t$-1\end{tabular}}}
\par\put(17.0,-6.0){\footnotesize{(a)}} \put(66.0,-6.0){\footnotesize{(b)}}
\end{overpic} Figure 1: (a) Existing online-VSR methods compute the motion
field of each frame separately via a weight-sharing offsets estimation module,
while (b) our method propagates the motion field of previous frame to the
current frame, reducing much the computational cost.

Compared to single image super-resolution (SISR) [4, 5, 6, 7, 8, 9, 10], VSR
employs an additional alignment module to aggregate the complementary
information from adjacent frames for achieving higher reconstruction ability
and better visual quality on the current frame. As a fundamental part of VSR
models, a few frame alignment methods [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
have been developed to improve the performance of VSR methods. Some methods
[14, 21, 15, 22] adopt the deformable strategy to predict the offsets for
every pixel, while some methods [16, 17, 23] enlarge the searching range by
using a non-local attention module to handle large motions. Though having
achieved considerable improvement in VSR performance, these methods can hardly
be applied to online-VSR tasks due to their long latency and high complexity.
To solve this issue, DAP [1] employs a deformable attention pyramid module,
which dynamically predicts a limited number of offsets for each pixel and
merges the corresponding features in a weighted average manner. On the other
hand, FRVSR [13] and CKBG [2] resort to the optical flow for frame alignment.
They adopt a lightweight optical flow network to estimate the displacement
between two frames and align them by warping. Nonetheless, those methods still
waste much computation on the repeated estimation of offsets.

As shown in Figure 1(a), most of the existing VSR methods apply a shared
alignment module to each frame and compute their motion fields separately,
ignoring the fact that the motion fields of neigboring frames are highly
correlated. In this work, we propose an efficient Temporal Motion Propagation
(TMP) method to speed up the alignment process by replacing the complex
offsets estimation module with an efficient motion field temporal propagation
scheme. As illustrated in Figure 1(b), instead of computing the motion field
of each frame from scratch, we inherit the motion field from the previous
frame, and apply a fast finetuning process to refine it. Considering the
different behaviors of different types of motions, we design two propagation
paths. One is the object motion propagation path, which locates the potential
destinations of moving objects in the current frame, while the other is the
camera motion propagation path, which exploits the position information of
static regions in the previous frame. The two paths work together to produce a
few offset candidates for each pixel in the current frame. In the finetuning
step, we pass the offsets of each pixel to its neighbors and evaluate all the
candidates to pick the best one that has the least matching distance. Finally,
we use the refined motion field to perform frame alignment by a warp
operation.

The alignment process will inevitably introduce some noise (_i.e._ , errors)
into the motion fields, which should be alleviated in the feature fusion
process. Some approaches [24, 15, 25] employ the reweighting strategy to
assign different importance to the aligned features in a spatial-wise manner
according to their similarities to the reference features. However, in this
way some well-matched features which contain complementary textures will be
discarded since they may have a low similarity to the reference features. To
address this issue, we propose to weigh the aligned features based on the
confidence of estimated offsets instead of the similarities between texture
features. Specifically, we employ two separate branches to extract motion and
texture features, respectively, and use only the motion features to evaluate
the precision of motion fields. In this way, we can more effectively assess
the confidence of motion fields since it is untangled from textures.

Extensive ablation studies and experiments on benchmark datasets demonstrate
the effectiveness and efficiency of our TMP method for online-VSR. Compared
with the existing state-of-the-arts, our model achieves not only higher
accuracy but also faster inference speed. In particular, compared with DAP
[1], our model achieves a PSNR gain of $0.08$ dB with nearly 65% latency on
the REDS4 dataset [26].

The rest of this paper is organized as follows. Section II summarizes the
relevant works on offline/online-VSR and feature alignment. Section III
presents the proposed TMP algorithm in detail. Section IV reports the
experimental results and analysis. Finally, Section V concludes this paper.

\begin{overpic}[width=433.62pt]{figures/framework.v8.pdf}
\put(6.5,11.3){\rotatebox{90.0}{Feature Extraction}}
\put(13.0,5.9){\rotatebox{90.0}{Motion Branch}}
\put(13.0,17.6){\rotatebox{90.0}{Texture Branch}}
\put(43.0,10.0){\rotatebox{90.0}{Reconstruction}}
\put(49.0,18.5){\rotatebox{90.0}{Bilinear Upsampling}}
\put(95.8,12.3){\rotatebox{90.0}{\begin{tabular}[]{c}Motion\\\
Confidence\end{tabular}}}
\put(95.8,25.6){\rotatebox{90.0}{\begin{tabular}[]{c}Motion\\\
Field\end{tabular}}} \par\put(16.5,1.5){$H^{0}_{t-1}$}
\put(23.0,1.5){$M_{t-1}$} \put(35.2,1.5){$H^{1}_{t-1}$}
\put(14.2,31.5){$H^{0}_{t}$} \put(27.0,31.5){$M_{t}$}
\put(41.0,31.5){$H^{1}_{t}$} \par\put(62.2,5.3){$M_{t-1}$}
\put(66.55,5.3){$H^{0}_{t-1}$} \put(71.6,5.3){$H^{0}_{t}$} \par\end{overpic}
Figure 2: Overview of our proposed online-VSR method. Left: The flowchart of
the proposed method. There are two major differences between our method and
the existing methods. One is the temporal motion propagation (TMP) module
(highlighted in green color box), which propagates the motion field from the
previous frame to the current frame. The other is the motion confidence
weighted fusion (highlighted in orange color box), which weighs the warped
features by the accuracy of estimated offsets. Right: The detailed
architecture of the TMP module. Best viewed in color.

##  II Related Work

###  II-A Video Super-resolution

Video super-resolution (VSR) aims at reconstructing visually pleasing HR
videos from the LR inputs via exploiting the temporal dependency between
consecutive frames. Depending on the usage of supporting frames, existing VSR
methods can be divided into four groups. Most of the earlier works [14, 15,
16, 17, 27, 28, 18, 29, 30, 31, 19, 32] belong to the sliding-window based
methods, which employ the frames within a short temporal window to enhance the
center frame. Xue et al. [28] warped the neighboring frames to the reference
frame by using task-oriented motion cues, acheiving obvious improvement. Song
et al. [31] proposed to aggregate the sequential information at different
stages of the neural network. TDAN [14] introduces deformable convolutions to
align the adjacent frames at the feature level, which is further extended by
EDVR [15] using a pyramidal cascade structure to fuse the temporal information
at various scales. The second group is the bidirectional propagation methods
[33, 22, 34], which leverage the information of past, current and future
frames in the sequence to enhance each frame. For example, RRCN [34] utilizes
two single-direction recurrent convolutional networks to collect the
complementary information from the whole video, while BasicVSR [33] and
BasicVSR++ [22] adopt a typical bidirectional loop structure as backbone. Due
to a large amount of supporting frames, these methods typically achieve better
accuracy by compromising the latency. The third group is unidirectional
propagation methods [35, 36], which aggregate the information from the past
and current frames, as well as several cached future frames. For example, RSLP
[35] transmits the information from the supporting frames using high
dimensional latent states. ETDM [36] utilizes the buffer mechanisms to store
differences between consecutive frames. When only the past and current frames
can be used, unidirectional propagation methods become online-VSR methods [13,
37, 38, 1], which have recently attracted much research attention due to their
short latency and hence potential applications in online video conferences and
live broadcasting.

###  II-B Online Video Super-resolution

The key issue of online-VSR is its strict latency requirement. One way to
handle this problem is to directly apply SISR methods [39, 40, 41, 42, 43, 44,
36]. For example, the interpolation-based SR approaches are widely applied in
the online video processing tasks, providing marginal accuracy improvement but
high speed. Zhang et al. [40] developed a bivariate rational fractal
interpolation model to preserve the textual and structural information of
images. Choi et al. [41] unified the interpolation step and quality-
enhancement step to reduce complexity. In addition, some lightweight learning-
based methods have been introduced to solve the online-VSR problem. Dong et
al. [43] decreased the resolution of feature extraction modules, greatly
reducing the inference latency. Kong et al. [44] first classified the image
regions to groups of different restoration difficulties, then processed those
patches by networks of different capacities to save computations.

Due to the better visual quality, designing a lightweight online-VSR network
draws increasingly attention. DAP [1] employs a deformable attention pyramid
module to dynamically attend to limited salient locations of supporting
frames, which significantly reduces the computational burden associated with
the classical attention mechanism. FRVSR [13] and CKBG [2] adopt a lightweight
network to estimate the optical flow and perform motion compensation. To break
the bottleneck brought by the small network capacity, FRVSR employs the
recurrent architecture that passes the previously estimated HR frame to the
following reconstruction module, while CKBG utilizes a kernel knowledge
transfer method to obtain additional information from the teacher network.
Despite the many efforts, most existing online-VSR methods still suffer from
low speed in inference, which hinders the widespread of these methods in many
applications.

###  II-C Frame Alignment

Frame alignment is critical to the performance of VSR for its role in
aggregating temporal information. Early works [28, 13, 11] mostly adopt the
optical flow technique to perform alignment. As pointed out in TOF [28],
however, the traditional optical flow methods are not the best choice for
estimating the motion fields in VSR. Later VSR methods focus more on implicit
alignment. TDAN [14] performs motion compensation via deformable convolutional
layers [45], which inspires a lot of following works such as EDVR [15] and
BasciVSR++ [22]. EDVR employs a pyramid, cascading and deformable alignment
module to handle complex motions in a coarse-to-fine manner, while BasicVSR++
leverages optical flow to improve the training stability of deformable
alignment. PFNL [16] utilizes the classical non-local mechanism to exploit the
sequential information from support video frames. MuCAN [17] and MANA [23]
adopt a similar strategy to avoid explicit motion estimation. Besides, DUF
[18] prefers to predict the upsampling filters to reconstruct the HR videos
directly.

Unfortunately, most of the existing VSR methods consider only the correlations
between two frames when estimating their relative displacement, ignoring the
fact that the motion fields of neighboring frames are similar. As a result,
DAP [1] has to limit the search space to a small range when performing
alignment. FRVSR [13] and CKBG [2] resort to unsound lightweight optical flow
networks for estimating the motion fields. In this paper, we propose to
temporally propagate the estimated motion fields across frames, which
significantly reduces the latency by removing the repeated computations of
estimating offsets for each frame.

##  III Methodology

###  III-A Overview

Online-VSR aims to reconstruct the HR reference frame
$I^{HR}_{t}\in\mathbb{R}^{sH\times sW\times C}$ from its corresponding LR
frame $I^{LR}_{t}\in\mathbb{R}^{H\times W\times C}$ and the $N-1$ support
frames $I^{LR}_{[t-N:t-1]}$, where $s$ denotes the upsampling factor, $H$ and
$W$ denote the height and width of LR frames, $C$ refers to the channel number
of each frame, and $t$ represents the timestamp of online video. Following
previous works [13, 37, 38], we employ the recurrent architecture for its
simplicity and efficiency. The overall framework of our proposed method is
depicted in the left part of Figure 2. Our network takes the current LR frame
$I^{LR}_{t}$, the estimated motion field of the last frame $M_{t-1}$ and the
hidden states $H^{0}_{t-1}$, $H^{1}_{t-1}$ as inputs, and outputs the
corresponding HR frame $I^{HR}_{t}$ and the updated hidden states $H^{0}_{t}$
and $H^{1}_{t}$. The hidden state $H^{0}_{t}$ is used to refine the inherited
motion field, while the hidden state $H^{1}_{t}$ stores the texture
information from previous video frames.

Compared with existing online-VSR methods, which separately estimate the
motion field of each frame for alignment, our proposed network architecture
has two major differences. First, our network forward propagates the estimated
motion field across frames to reduce repetitive computation by introducing the
Temporal Motion Propagation (TMP) module, whose detailed structures are
described in Section III-B. Second, we use two different branches to process
the motion features and texture features, respectively, bringing additional
advantages in fusing the aligned features. The details are presented in
Section III-C.

###  III-B Temporal Motion Propagation

Considering the temporal smoothness of motions, we propose a novel TMP module
to inherit the estimated motion field from the last frame and finetune it
based on sequential contexts. TMP circumvents the problem of motion field
estimation from scratch for each frame, and hence reduces greatly the
inference latency.

Given the hidden states of current frame $H^{0}_{t}$ and past frame
$H^{0}_{t-1}$ as inputs, TMP estimates the current motion field
$\boldsymbol{M}_{t}\in\mathbb{R}^{2\times H\times W}$ by taking
$\boldsymbol{M}_{t-1}$ of previous frame as a priori. The detailed process of
TMP is illustrated in the right part of Figure 2. We leverage the continuity
of motions to propagate the motion information from previous frames to the
current frame, and predict a set of offset candidates for $M_{t}$ dynamically
based on the propagated motions. To handle distinct motion behaviors of moving
objects and static regions, we employ two different motion propagation paths,
_e.g._ , object (OBJ) motion path and camera (CAM) motion path. During the
subsequent finetuning step, we shuffle the candidates of each pixel to its
neighbors and find the optimal offset with the least matching distance.

\begin{overpic}[width=459.63612pt]{figures/motions.v3.p1.pdf}
\put(11.0,0.5){\small{Frame $t$-2}} \put(38.0,0.5){\small{Frame $t$-1}}
\put(71.0,0.5){\small{Frame $t$}} \end{overpic} (a) OBJ motion propagation
\begin{overpic}[width=459.63612pt]{figures/motions.v3.p2.pdf}
\put(11.0,0.5){\small{Frame $t$-2}} \put(38.0,0.5){\small{Frame $t$-1}}
\put(71.0,0.5){\small{Frame $t$}} \end{overpic} (b) CAM motion propagation

Figure 3: Illustration of (a) object (OBJ) and (b) camera (CAM) motion
propagation paths. The OBJ path aims to locate moving objects in the current
frame, while the CAM path matches the static regions. The orange arrow
represents the estimated motion from $I^{LR}_{t-2}$ to $I^{LR}_{t-1}$, which
starts from the blue point and ends at the orange point. The red arrow
indicates the temporally propagated motion. In the CAM path, the green point
in $I^{LR}_{t}$ has the same position as the orange point in $I^{LR}_{t-1}$.
The red points indicates the potential positions of the object at the
corresponding frames, and the brighter colors represent higher likelihood.

OBJ Motion Propagation. There are two main types of motions between video
frames. The first type is for moving objects, termed as OBJ motion, which
generally results from a combination of the movement of objects and the
movement of camera. Based on the assumption that objects keep a relatively
smooth movement between adjacent frames, we propagate the OBJ motion from the
last frame to the current one. As shown in Figure 3(a), the object at the blue
point $P^{i,j}_{t-2}$ in frame $I^{LR}_{t-2}$ moves to the orange point
$P^{m,n}_{t-1}$ during the time interval $t-2$ to $t-1$. By extrapolating this
motion, the potential destination of this object should fall into the red
regions in frame $I^{LR}_{t}$, where the brighter colors indicate candidates
with higher likelihood. Without losing generality, we assume that the
potential destinations follow a two-dimensional Gaussian distribution centered
at pixel $P^{2m-i,2n-j}_{t}$. Mathematically, the OBJ propagation path can be
described as follows:

$\begin{split}&\begin{cases}\boldsymbol{M}^{m,n}_{t-1}=(\Delta p_{w},\Delta
p_{h}),\\\ \boldsymbol{M}_{t}^{m-\Delta p^{\prime}_{w},n-\Delta
p^{\prime}_{h}}=(\Delta p^{\prime}_{w},\Delta p^{\prime}_{h}),\\\
{\mathbb{P}}(\Delta o_{w},\Delta
o_{h})\sim\mathscr{N}(\boldsymbol{0},\sigma\boldsymbol{I}),\\\ \end{cases}\\\
&\text{with}~{}\Delta o_{w}=\Delta p^{\prime}_{w}-\Delta p_{w},\Delta
o_{h}=\Delta p^{\prime}_{h}-\Delta p_{h}\\\ \end{split}$ (1)

where $(\Delta p_{w},\Delta p_{h})$ denotes the motion inherited from the
previous frame, and $(\Delta p^{\prime}_{w},\Delta p^{\prime}_{h})$ is the
potential OBJ motion on current frame. $\boldsymbol{M}^{m,n}_{t-1}$ describes
the movement from pixel $P^{m+\Delta p_{w},n+\Delta p_{h}}_{t-2}$ in frame
$I^{LR}_{t-2}$ to pixel $P^{m,n}_{t-1}$ in frame $I^{LR}_{t-1}$. $\mathbb{P}$
represents the distribution of $\Delta o_{w},\Delta o_{h}$, and $\sigma$
denotes the standard deviation of two-dimensional Gaussian distribution. For
each OBJ motion, we select $k$ additional pixels as the candidates for offset
estimation.

CAM Motion Propagation. The other type of motion refers to the motions of
static regions, which is actually caused by the movement of camera. Thus we
call it as CAM motion. Unlike OBJ motion, CAM motion has the characteristic
that all static regions share the same camera motion, making it possible to
spread the camera motion inherited from one region to other regions,
especially for those unseen regions in the front frames. Considering that the
CAM motion is smooth among neighboring frames, we can transfer the CAM motion
from the previous frame to the current one. Figure 3(b) shows an example of
the CAM propagation path. The region at blue point $P^{i,j}_{t-2}$ in frame
$I^{LR}_{t-2}$ moves to the orange point $P^{m,n}_{t-1}$ in frame
$I^{LR}_{t-1}$. If both the green point $P^{m,n}_{t}$ and the orange point
$P^{m,n}_{t-1}$ belong to static regions, we can easily infer that the motion
of $P^{m,n}_{t}$ (red arrow) is similar to the one of $P^{m,n}_{t-1}$ (orange
arrow). Similar to the OBJ path, the CAM motion propagation path can be
formulated as follows:

$\begin{split}&\begin{cases}\boldsymbol{M}^{m,n}_{t-1}=(\Delta p_{w},\Delta
p_{h}),\\\ \boldsymbol{M}_{t}^{m,n}=(\Delta p^{\prime}_{w},\Delta
p^{\prime}_{h}),\\\ {\mathbb{P}}(\Delta o_{w},\Delta
o_{h})\sim\mathscr{N}(\boldsymbol{0},\sigma\boldsymbol{I}),\\\ \end{cases}\\\
&\text{with}~{}\Delta o_{w}=\Delta p^{\prime}_{w}-\Delta p_{w},\Delta
o_{h}=\Delta p^{\prime}_{h}-\Delta p_{h}\\\ \end{split}$ (2)

We also select $k$ candidates for each pixel in $I^{LR}_{t}$.

To simplify the implementation, we concurrently apply the two propagation
paths to each pixel. This will not lead to conflicts since each path only
generates a few candidates for the subsequent finetuning process.

Finetuning. Since the movement of objects and cameras may slightly change in
different frames, the motion fields propagated from past frames will
inevitably contain some noise. To ensure that the propagated motion field fits
the content of the current frame, we further refine it by a lightweight
process with the contextual features $H^{0}_{t-1}$ and $H^{0}_{t}$. The whole
process is illustrated in the right part of Figure 2. During this process, the
offset candidates of each pixel are first diffused to its neighbors to enhance
the diversity and then evaluated iteratively to obtain the optimal candidate.
We introduce the following matching distance function to evaluate the offsets:

$\begin{array}[]{l}\text{d}=||H^{0}_{t-1}(m+\Delta p^{\prime}_{w},n+\Delta
p^{\prime}_{h})-H^{0}_{t}(m,n)||^{2}_{2}.\end{array}$ (3)

###  III-C Motion Confidence Weighted Fusion

The alignment process will inevitably introduce errors in the motion field,
which should be processed in the feature fusion process. Most existing methods
[24, 15, 25] adopt a similarity-based re-weighting strategy to assign
different importance to aligned features according to their similarities with
reference features. However, in this way some well-matched features may be
deemed as insignificant elements due to the low similarities caused by
different yet complementary textures. Unlike those methods, we propose to
weigh the features based on the confidence of estimated offsets.

1

2

Inp: Features of frame $t$-1 $\boldsymbol{H}^{0}_{t-1}$; features of frame $t$
$\boldsymbol{H}^{0}_{t}$; motion fields of frame $t$-1 $\boldsymbol{M}_{t-1}$;
max iteration $k$; search range $\sigma$.

3

Out: Motion fields of frame $t$ $\boldsymbol{M}_{t}$; motion confidences of
frame $t$ $\boldsymbol{C}_{t}$

4

5// parallel for-loop in gpus

6For  _each $P^{m,n}_{t-1}$ in frame $t$-1 _ Do

7 Initialize candidates set $S_{m,n}=\\{\\}$;

8 For  _$i=0,1,\cdots,k-1$_ Do

9 Generate $(\Delta p^{\prime(i)}_{w},\Delta p^{\prime(i)}_{h})$ using Eq. 1;

10 Add this candidate into $S_{m-\Delta p^{\prime(i)}_{w},n-\Delta
p^{\prime(i)}_{h}}$;

11 Generate $(\Delta p^{\prime(i)}_{w},\Delta p^{\prime(i)}_{h})$ using Eq. 2;

12 Add this candidate into $S_{m,n}$.

13 End

14

15 Collect candidates from neighbors into $S_{m,n}$;

16 Compute distances for all candidates using Eq. 3;

17 Update $\boldsymbol{M}^{m,n}_{t}$ based on the minimal distance;

18 Update $\boldsymbol{C}^{m,n}_{t}$ using Eq. 4.

19 End

20

Algorithm 1 Temporal Motion Propagation Algorithm

Specifically, as shown in the orange color box in Figure 2, after extracting
the visual features of the current frame $I^{LR}_{t}$, we introduce two
independent branches for motion estimation and texture extraction,
respectively. The former generates the motion features $H^{0}_{t}$, which are
used to refine the motion field and evaluate the confidence of estimated
offsets. The latter extracts the texture features $H^{1}_{t}$ that are
utilized to reconstruct the HR frame. With such an architecture, we can assign
larger weights to warped texture features containing complementary textures,
even if they have low similarities to the reference texture features. To
further reduce the computations, we directly use the minimum distance map
obtained in the aforementioned finetuning process to measure the precision of
the estimated offsets. The whole process of motion confidence weighted fusion
(MCWF) can be described as follows:

$\begin{cases}\boldsymbol{C}_{t}=e^{-a\cdot D_{min}},\\\
\boldsymbol{\tilde{H}}^{1}_{t-1}=\boldsymbol{C}_{t}\cdot\boldsymbol{\hat{H}}^{1}_{t-1},\end{cases}$
(4)

where $\boldsymbol{\hat{H}}^{1}_{t-1}$ is the warped texture features from
previous frame $I^{LR}_{t-1}$, $\boldsymbol{\tilde{H}}^{1}_{t-1}$ is the
weighted texture features, $\boldsymbol{C}_{t}$ represents the confidence of
refined motion filed $M_{t}$, and $D_{min}$ denotes the minimum distance map
in the finetuning step of TMP. Finally, the updated texture features
$\boldsymbol{\tilde{H}}^{1}_{t-1}$ are concatenated with texture features of
the current frame $H^{1}_{t}$, and then fed to the following reconstruction
module.

The pseudo-code of the proposed TMP algorithm is shown in Algorithm 1.

TABLE I:  Comparison with state-of-the-art online-VSR and non-online VSR
methods. The runtime, frames per second (fps), params, and PSNR/SSIM metrics
on three benchmarks with two BI and BD degradations are reported. The category
(Cate.) and support frames (Supp.F.) of the competing methods are also
indicated. “SW.”, “Bi.” and “Uni.” denote the sliding-window based methods,
bidirectional propagation methods and unidirectional propagation methods,
respectively. We use “P” to denote all previous frames, and “P(n)” to
represent the $n$ preceding frames. Similarly, “F” denotes all future frames
while “F(n)” represents the $n$ forward frames. Runtime is computed on an LR
frame of size $180\times 320$. Following DAP [1], those methods which can
process 720p videos in at least 24fps are recognized as real-time (R-T.)
methods. We evaluate the complexity and speed (runtime/fps/MACs/Params) of all
open-source online-VSR methods on the same setting. The best and second best
results among online-VSR methods on each test set are highlighted in red and
blue. The symbol “-” means that the result is not available.

Cate. | Method | Supp.F. |  R-T. |  Run$\downarrow$ (ms) |  fps$\uparrow$ (1/s) |  MACs (G) |  Params (M) | BI degradation | BD degradation
---|---|---|---|---|---|---|---|---|---
REDS4(RGB)$\uparrow$ (PSNR/SSIM) |  Vid4(Y)$\uparrow$ (PSNR/SSIM) |  Vimeo-90K-T(Y)$\uparrow$ (PSNR/SSIM) |  Vid4(Y)$\uparrow$ (PSNR/SSIM)
TOFlow [28] | P(3)+F(3) | ✗ | - | - | - | 1.4 | 27.98/0.7990 | 25.89/0.7651 | 34.62/0.9212 | -
| DUF [18] | P(3)+F(3) | ✗ | 974 | 1.0 | - | 5.8 | 28.63/0.8251 | - | 36.87/0.9447 | 27.38/0.8329
| PFNL [16] | P(3)+F(3) | ✗ | 295 | 3.4 | - | 3.0 | 29.63/0.8502 | 26.73/0.8029 | - | 27.16/0.8355
SW. | RBPN [46] | P(3)+F(3) | ✗ | 1507 | 0.7 | - | 12.2 | 30.09/0.8590 | - | 37.20/0.9458 | -
MuCAN [17] | P(2)+F(2) | ✗ | - | - | $7^{\prime}923$ | - | 30.88/0.8750 | - | - | -
STAN [19] | P(10)+F(10) | ✗ | 108 | 9.30 | 980 | 16.2 | 28.85/0.8207 | 25.58/0.7430 | - | -
| TGA [27] | P(3)+F(3) | ✗ | - | - | - | 5.8 | - | - | 37.59/0.9516 | 27.63/0.8423
| EDVR-M [15] | P(2)+F(2) | ✗ | 118 | 8.5 | 462 | 3.3 | 30.53/0.8699 | 27.10/0.8186 | 37.33/0.9484 | 27.45/0.8406
| EDVR [15] | P(2)+F(2) | ✗ | 378 | 2.6 | 2’017 | 20.6 | 31.09/0.8800 | 27.35/0.8264 | 37.81/0.9523 | 27.85/0.8503
| IAM [25] | P(3)+F(3) | ✗ | - | - | - | 17.0 | 31.30/0.8850 | 27.90/0.8380 | 37.84/0.9498 | -
| MCRNet [47] | P(3)+F(3) | ✗ | - | - | 910 | 14.0 | 31.86/0.9013 | 27.53/0.8397 | - | -
Bi. | BasicVSR [33] | P+F | ✗ | 63 | 15.9 | 397 | 6.3 | 31.42/0.8909 | 27.24/0.8251 | 37.53/0.9498 | 27.96/0.8553
IconVSR [33] | P+F | ✗ | 70 | 14.3 | 452 | 8.7 | 31.67/0.8948 | 27.39/0.8279 | 37.84/0.9524 | 28.04/0.8570
BasicVSR++ [22] | P+F | ✗ | 77 | 13.0 | 418 | 7.3 | 32.39/0.9069 | 27.79/0.8400 | 38.21/0.9550 | 29.04/0.8753
| SSL [48] | P+F | ✗ | 24 | 41.7 | 92 | 1.0 | 31.06/0.8933 | 27.15/0.8208 | 37.06/0.9458 | 27.56/0.8431
Uni. | RLSP [35] | P+F(1) | ✗ | 49 | 20.4 | 252 | 4.2 | - | - | 36.49/0.9403 | 27.48/0.8388
ETDM [36] | P+F(3) | ✗ | 70 | 14.3 | - | 8.4 | 32.15/0.9024 | - | - | 28.81/0.8725
| Bicubic | N | ✓ | - | - | - | - | 26.14/0.7292 | 23.78/0.6347 | 31.30/0.8687 | 21.80/0.5246
| FRVSR [13] | P(1) | ✗ | 137 | 7.3 | - | 5.1 | - | - | 35.64/0.9319 | 26.69/0.8103
Online-VSR | RSDN [37] | P | ✗ | 49 | 20.4 | 356 | 6.2 | - | - | 37.23/0.9471 | 27.92/0.8505
RRN [38] | P | ✓ | 34 | 29.4 | 193 | 3.4 | - | - | 36.69/0.9432 | 27.69/0.8488
BasicVSR++* | P | ✓ | 40 | 25.0 | 146 | 3.0 | 30.44/0.8686 | 27.06/0.8173 | 37.11/0.9464 | 27.49/0.8426
DAP [1] | P | ✓ | 38 | 26.3 | 165 | - | 30.59/0.8703 | - | 37.29/0.9476 | -
| CKBG [2] | P | ✓ | 9 | 111.1 | 18 | 1.8 | 29.73/0.8514 | 26.34/0.7857 | - | -
| Ours | P | ✓ | 25 | 40.1 | 176 | 3.1 | 30.67/0.8710 | 27.10/0.8167 | 37.33/0.9481 | 27.61/0.8428

##  IV EXPERIMENT

###  IV-A Experimental Settings

Datasets. We conduct experiments on three popular benchmarks, REDS [26], Vid4
[49] and Vimeo-90K [28]. REDS is a high-quality video dataset which is widely
used for evaluating the performance of VSR methods due to its various motions
among frames. Following EDVR [15], we divide REDS into two parts. The training
set includes 266 clips (each has 100 consecutive frames), and the test set
(REDS4) contains only four typical videos. Vimeo-90K consists of about 90K
7-frame video clips. The diverse scenarios in this dataset make it a popular
benchmark for verifying the generalization performance of VSR models. We train
the models on the training set of Vimeo-90K and evaluate them on the
corresponding test set (denoted by Vimeo-90K-T). It should be noted that since
Vimeo-90K-T is proposed for evaluating the performance of sliding-window VSR
methods, we follow previous works [15, 22, 1] and only compute the PSNR and
SSIM for the center frame of the 7-frames clip. In addition, we directly
utilize the models trained on Vimeo-90K to evaluate their performances on
Vid4.

We adopt two different degradation strategies to perform $4\times$
downsampling, namely BI and BD. The BI degradation refers to bicubic
downsampling, which is implemented by using MATLAB function
$\mathrm{imresize}$ in our experiments. The BD refers to blur degradation. We
first blur the origin HR frame by a Gaussian filter with $\sigma=1.6$, and
then conduct subsampling for every four pixels.

Implementation Details. We adopt 3 residual blocks (RB) to extract the
features of each frame and another 30 RBs to reconstruct the HR frames and
generate the hidden states. Two RBs and 2 conv layers are used for the motion
branch, whose channel is reduced to 8 gradually. For the texture branch, a
single conv layer is employed to extract features. If not specified, the
channel number is set to 64.

In the TMP module, $k$ is set to 2 while $\sigma$ is set to 30. The $a$ in
MCWF is set to $1$ for simplicity. The downsampling factor is set to $s$ = 4
in our experiments. During training, we feed the network with randomly cropped
RGB sequences of size $64\times 64$ (low resolution) and adopt random flips,
rotations, and temporal inversion to augment the data. Similar to [15, 33,
22], the classical Charbonier loss [50], the Adam optimizer [51], and the
Cosine Annealing scheme [52] are employed to optimize the network. The initial
learning rates for the motion branch and the rest of the network are set to
$1e^{-3}$ and $1e^{-4}$, respectively. The total number of iterations is 600K
and the batch size is 8. The model is trained under the PyTorch framework with
two NVIDIA V100 GPUs.

When training on REDS [26], we set the batch size as 8, each of which contains
15 consecutive frames. On Vimeo-90K [28], since there are only 7 frames per
clip, we temporally flip the original sequence to obtain a 14-frame sequence
as input to the network. During testing, we evaluate the performance by taking
the whole sequence as input.

(a)  \begin{overpic}[width=433.62pt]{figures/experiments_fig.v2.pdf}
\put(27.0,53.0){\small{Bicubic}}
\put(40.0,53.0){\small{RRN~{}\cite[cite]{[\@@bibref{}{rrn}{}{}]}}}
\put(50.5,53.0){\small{BasicVSR++*}}
\put(64.5,53.0){\small{RSDN~{}\cite[cite]{[\@@bibref{}{rsdn}{}{}]}}}
\put(79.0,53.0){\small{{Ours}}} \put(92.5,53.0){\small{GT}}
\put(-0.2,2.5){\rotatebox{90.0}{{Clip} 81/954}}
\put(-0.2,16.3){\rotatebox{90.0}{{Clip} 3/670}}
\put(-0.2,28.8){\rotatebox{90.0}{{Clip} 82/748}}
\put(-0.2,41.8){\rotatebox{90.0}{{Clip} 24/315}} \end{overpic}

Figure 4: Qualitative comparisons on details of static regions and moving
objects. The proposed method restores more details on both static regions (top
two rows) and moving objects (bottom two rows) with CAM and OBJ motions,
respectively. The results are from Vimeo-90K-T [28]. Zoom in for the best
view.

\begin{overpic}[width=433.62pt,height=260.17464pt]{figures/REDS4/gt.015.png}
\put(-5.0,20.0){\rotatebox{90.0}{Clip 015/095}}
\put(112.0,60.5){\small{Bicubic}}
\put(140.0,60.5){\small{CKBG~{}\cite[cite]{[\@@bibref{}{ckbg}{}{}]}}}
\put(168.0,60.5){\footnotesize{BasicVSR++*}} \put(204.0,60.5){\small{{Ours}}}
\put(235.5,60.5){\small{GT}} \end{overpic}

\begin{overpic}[width=433.62pt,height=260.17464pt]{figures/REDS4/gt.020.png}
\put(-5.0,20.0){\rotatebox{90.0}{Clip 020/008}} \end{overpic}

Figure 5: Qualitative results on REDS4 [26]. Our TMP method recovers more
details on the frames that contain various motions. Zoom-in for best view.
\begin{overpic}[width=433.62pt]{figures/experiments_fig_sup.v3.pdf}
\put(24.0,99.5){\small{Bicubic}}
\put(34.5,99.5){\small{RRN~{}\cite[cite]{[\@@bibref{}{rrn}{}{}]}}}
\put(44.6,99.5){\small{BasicVSR++*}}
\put(57.4,99.5){\small{RSDN~{}\cite[cite]{[\@@bibref{}{rsdn}{}{}]}}}
\put(70.0,99.5){\small{{Ours}}} \put(82.0,99.5){\small{GT}}
\par\put(-0.2,1.3){\rotatebox{90.0}{{Clip} 83/963}}
\put(-0.2,14.1){\rotatebox{90.0}{{Clip} 80/837}}
\put(-0.2,26.7){\rotatebox{90.0}{{Clip} 48/660}}
\put(-0.2,39.7){\rotatebox{90.0}{{Clip} 34/986}}
\par\put(-0.2,52.2){\rotatebox{90.0}{{Clip} 25/898}}
\put(-0.2,64.8){\rotatebox{90.0}{{Clip} 12/040}}
\put(-0.2,77.5){\rotatebox{90.0}{{Clip} 10/935}}
\put(-0.2,90.6){\rotatebox{90.0}{{Clip} 1/837}} \end{overpic} Figure 6: More
qualitative results on Vimeo-90K-T [28]. The proposed TMP method generates
better textures on various scenes. Zoom-in for best view.

\begin{overpic}[width=212.47617pt,height=130.08731pt]{figures/vid4/gt.city.png}
\put(30.0,-5.0){The $29$-th frame of City clip} \put(130.0,-5.0){The $7$-th
frame of Walk clip} \put(9.0,-58.0){Bicubic}
\put(33.0,-58.0){FRVSR~{}\cite[cite]{[\@@bibref{}{frvsr}{}{}]}}
\put(64.0,-58.0){RRN~{}\cite[cite]{[\@@bibref{}{rrn}{}{}]}}
\put(90.0,-58.0){BasicVSR++*}
\put(120.5,-58.0){RSDN~{}\cite[cite]{[\@@bibref{}{rsdn}{}{}]}}
\put(153.5,-58.0){{Ours}} \put(183.5,-58.0){GT} \end{overpic}

Figure 7: Qualitative results on Vid4 [49]. The proposed TMP method obtains
better visual results . Zoom-in for best view.

(a)

\begin{overpic}[width=433.62pt]{figures/experiments_fig2.v2.rh.pdf}
\put(3.8,78.0){\Large{$y$}} \put(12.3,49.3){\Large{$t$}}
\put(7.0,-2.0){\small{(a)}} \put(26.0,-2.0){\small{(b)}}
\put(44.5,-2.0){\small{(c)}} \put(63.0,-2.0){\small{(d)}}
\put(82.0,-2.0){\small{(e)}} \end{overpic}

Figure 8:  Visualization of Temporal Consistency. From left to right: the
temporal consistency profile generated by (a) RRN [38], (b) BasicVSR++*, (c)
RSDN [37], (d) ours and (e) groundtruth. We observe the changes of pixels on
the green line across time. The camera is moving horizontally. The results of
other methods all contain irregular patterns, as indicated by the yellow
dashed box, while our method keeps the pattern well.

Compared Methods. We compare our method with state-of-the-art online-VSR
methods (FRVSR [13], RSDN [37], RRN [38], DAP [1] and CKBG [2]), as well as
sliding-window based methods (TOFlow [28], DUF [18], PFNL [16], RBPN [46],
MuCAN [17], TGA [27], EDVR [15], IAM [25] and STAN [19] and MCRNet [47]),
bidirectional propagation methods (BasicVSR [33], IconVSR [33], BasicVSR++
[22] and SSL [48]), and unidirectional propagation methods (RLSP [35] and ETDM
[36]). Additionally, we introduce a method “BasicVSR++*” by removing backward
propagation branches of BasicVSR++ and reducing its model size to meet the
access and latency constraints of online-VSR applications.

###  IV-B Results

Comparison with Online-VSR Methods. The quantitative results are listed in the
bottom of Table I. One can see that our method significantly surpasses other
online models in terms of balanced accuracy and speed. Though CKBG has the
fastest speed, it has poor PSNR/SSIM performance. Our method outperforms CKBG
by 0.94dB and 0.76dB on REDS4 and Vid4, respectively. Compared to DAP, our
method achieves a performance gain of 0.08dB on the RESD4 dataset and it is
1.52 times faster (25ms vs. 38ms). Our method also exhibits superior
performance (37.33dB in PSNR and 0.9481 in SSIM) on the large-scale
Vimeo-90K-T dataset, which demonstrates its capacity to effectively handle
various scenarios. The PSNR/SSIM indices of our method are slightly lower than
that of RSDN on BD-degrated Vid4 dataset. This is because the complex textures
of Vid4 are heavily corrupted by the BD blur degradation, while RSDN employs
much more parameters to solve this issue, making it much slower.

Figure 4 present qualitative comparisons on the recovered details of static
regions and moving objects, respectively. One can see that our method shows
better visual quality than other online-VSR methods on both scenes. This can
be attributed to the inherited motion fileds through CAM and OBJ motion
propagation during inference. For example, the image in the first row comes
from the “00081/0954” clip, where the camera is moving slowly. Our method
recovers more iron bars of the barrier and shows clearer details. The image
displayed in the third row is from the “00082/0748” clip, where the gear
exhibits rotational movement. Other online-VSR methods fail to accurately
align the frames and result in wrong structures of the gear, while our method
precisely tracks the rotation and successfully recover the ridges.

Figure 5 , Figure 6 and Figure 7 illustrate more visual results on REDS4 [26],
Vimeo-90K [28] and Vid4 [49], respectively. Although videos from REDS4 contain
relatively larger motions, one can see from Figure 5 that our method still
recovers better details than competing methods. Vimeo-90K contains various
scenes with different objects. As can be seen from Figure 6, the enriched
textures in the VSR results further affirm the effectiveness of our TMP
approach. The images from Vid4 dataset are heavily corrupted by BD
downsampling, making it harder for restoration. However, our TMP method still
reconstructs compelling details, as demonstrated in Figure 7.

Comparison with Non-Online VSR methods. We further compare the results of our
method with non-online VSR methods. As shown in Table I, there exist two key
factors that affect the accuracy of online-VSR and non-online VSR methods.
First, the set of support frames heavily affects the PSNR/SSIM results because
it determines the amount of information that a VSR model can exploit.
BasicVSR++, which belongs to the bidirectional propagation methods, aggregates
information from all video frames and achieves the best PSNR/SSIM indices on
all the three benchmarks. ETDM accesses more previous frames and obtains an
improvement of about 0.85dB on REDS4 over IAM. Second, the network capacity
plays an important role in the performance of VSR methods. We see that most
online-VSR methods have half the parameters of the state-of-the-art non-online
VSR methods (_e.g._ , 3.1M for our model while 8.4M for ETDM, 7.3M for
BasicVSR++, and 17.0M for IAM).

However, the longer latency of non-online VSR methods hinders them from the
online-VSR applications. For a fair comparison, we further train a variant of
BasicVSR++, namely BasicVSR++*, which only accesses the previous video frames
and has a comparable number of parameters with our model. As can be seen from
Table I, BasicVSR++* performs much worse than our model in terms of either
PSNR/SSIM or fps indices, validating the effectiveness and efficiency of the
proposed TMP and MCWF strategies.

Complexity and Speed. From Table I, one can see that our method achieves the
second fastest inference speed (25ms/40.1fps) among all VSR methods with
relatively good PSNR/SSIM performance. For online-VSR methods, DAP,
BasicVSR++* and RRN consume nearly the same amount of MACs as our method, but
they have obviously longer latency (38ms, 40ms and 34ms, respectively). On the
other hand, EDVR-M has a similar PSNR/SSIM performance to our method, but its
inference speed is about five times slower (fps: 8 vs. 40). PFNL contains a
similar number of parameters to ours, yet it takes nearly 300ms to process a
frame of $180\times 320$ resolution. In summary, compared with those methods
having similar complexity, parameters or performance, our method runs much
faster.

TABLE II: Temporal consistency on Vimeo-90K.

Metrics | Bicubic | RRN [38] | RSDN [37] | BasicVSR++* | TMP
---|---|---|---|---|---
tOF$\downarrow$ $\times 10$ | 6.43 | 2.23 | 2.21 | 1.98 | 2.02
TCC$\uparrow$ $\times 10$ | 2.36 | 3.40 | 3.64 | 3.44 | 3.53
FWE$\downarrow$ $\times 10^{4}$ | – | 3.32 | 3.45 | 3.07 | 3.14

Temporal Consistency. We use three metrics tOF [53], TCC [54] and Flow Warping
Error (FWE) [55] to measure the temporal consistency. The comparison results
are presented in Table II. It can be seen that our method achieves better
temporal consistency than most methods. Though BasicVSR++* slightly
outperforms our method, it adopts a sophisticated and cumbersome alignment
module which lags behind us in efficiency (please refer to Table I). We also
provide the visualization results in Figure 8. Taking the clip “00023/0216” of
Vimeo-90K-T as example, where the camera moves horizontally, our method
produces less distortions and generates a more consistent profile than the
competing methods across frames. This further demonstrates the effectiveness
of our TMP module in frame alignment.

TABLE III:  Ablation Studies on TMP and MCWF. Evaluations are conducted on the REDS4 dataset and the fps is measured with the HR frame of size $1280\times 720$.  Model # |  Rand |  OBJ |  TMP |  SRF |  MCWF | fps(1/s) | PSNR/SSIM
---|---|---|---|---|---|---|---
1 |  |  |  |  |  | 40.52 | 29.74/0.8507
2 |  ✓ |  |  |  |  | 40.45 | 29.92/0.8541
3 |  |  ✓ |  |  |  | 40.31 | 30.33/0.8649
4 |  |  |  ✓ |  |  | 40.30 | 30.41/0.8651
5 |  |  |  ✓ |  ✓ |  | 40.09 | 30.46/0.8666
6 |  |  |  ✓ |  |  ✓ | 40.13 | 30.60/0.8694

###  IV-C Ablation Studies

In this section, we perform several ablation studies on the proposed TMP
module and MCWF strategy to better illustrate their roles. We use REDS for
training and REDS4 for evaluation, and set the number of iterations to 300K.

Effectiveness of TMP and MCWF. To validate the effectiveness of our proposed
TMP module and MCWF strategy, we train five variant models of our method. As
shown in Table III, the baseline model (Model 1) directly fuses the hidden
states of the last frame without alignment, which shows poor performance.
Model 2 performs alignment with a warp operation. Different from Model 4 that
adopts the proposed TMP module, Model 2 randomly initializes the motion field
and then refines it using the same fintuning process as the TMP module. Model
2 achieves a $0.18$dB improvement over Model 1, while lagging $0.49$dB behind
Model 4. The performance gap demonstrates the effectiveness of TMP in
inheriting motion field from the previous frame. Besides, TMP has minimal
impact on the inference speed, only slightly dropping from 40.52fps to
40.30fps. Model 3 utilizes only the OBJ path to inherit the offsets, while
Model 4 applies both the OBJ and CAM paths. One can see that CAM can bring
additional improvement to OBJ since the CAM path can be applied to a broader
range of regions, such as unseen regions in previous frames. Model 5 employs
the similarity reweighting fusion (SRF) strategy, which assigns importance to
the warped features according to their similarities with the reference
features. Model 6 is the proposed network which applies the TMP module and
MCWF strategy simultaneously. One can see that our MCWF strategy achieves
0.27dB and 0.14dB performance gains over Model 4 and Model 5, respectively.

TABLE IV:  Hyper-parameter (HP) Selection of TMP. Evaluations (PSNR/ SSIM) are conducted on REDS4 in two different color spaces. “Y” represents YCrCb color space, while “RGB” denotes the normal RGB color space.  HP |  Values |  REDS4(Y) |  REDS4(RGB)
---|---|---|---
k |  0 |  31.72/0.8786 |  30.35/0.8635
|  1 |  31.95/0.8838 |  30.60/0.8693
|  2 |  31.96/0.8839 |  30.60/0.8694
|  4 |  31.96/0.8839 |  30.60/0.8694
$\sigma$ |  0.1 |  31.72/0.8788 |  30.36/0.8637
|  1 |  31.92/0.8828 |  30.55/0.8682
|  10 |  31.96/0.8838 |  30.59/0.8693
|  30 |  31.96/0.8839 |  30.60/0.8694
|  60 |  31.76/0.8796 |  30.40/0.8648

Hyper-parameter Selection of TMP. There are two major hyper-parameters in our
TMP module. The first one is $k$, which counts the offset candidates generated
by each propagation path. The second one is the standard deviation $\sigma$ of
the Gaussian distribution, which implicitly impacts the searching range of
estimated motions.

Table IV shows the ablation study results on the two hyper-parameters. One can
see that there is a significant performance decrease (about 0.24dB and 0.25dB
on Y and RGB color spaces, respectively) when $k$ is set to 0, which indicates
that directly using the inherited offsets is not the best choice for the
alignment of current frame. It is important to generate a few new offset
candidates for further evaluation. The best results are achieved when $k$ is
set to 2. That is, introducing only a few more candidates will be enough for
TMP to estimate the offsets. The performance saturates when $k$ is further
increased.

Regarding $\sigma$, the best results ($31.96$dB and $30.60$dB on Y and RGB
color spaces, respectively) are achieved when it is set between $10$ and $30$.
A smaller $\sigma$ limits the search space, which decreases the accuracy of
offset estimation for objects that change their motion. A larger $\sigma$
represents a wider range of searching space, which however lowers the
likelihood of the optimal candidates.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 9: Visualizations of OBJ & CAM paths and motion confidence. (a) The
previous frame (T-1). (b) The reference frame T. (c) Offset field with only
OBJ path. (d) Offset field with both paths. (e) Motion confidence with only
OBJ path. (f) Motion confidence with both paths.

Visulization of Motion Field and Confidence. The visualizations are shown in
Figure 9, where the cars move to the right while the buildings/roads shift to
the left relative to the camera. The OBJ path can only handle the motion of
objects that appear in consecutive frames but cannot track the motion of newly
appeared objects. As can be seen in Figure 9(c), the offsets of car and
building can be accurately estimated with only the OBJ path since they appear
in the two consecutive frames in Figure 9(a) and Figure 9(b). However, the
motion of the crosswalk that does not appear in the previous frame in Figure
9(a) is predicted incorrectly. By coupling with the CAM path, as shown in
Figure 9(d), the offsets of the crosswalk can be accurately estimated. In
Figure 9(e), the crosswalk region has a low (dark) motion confidence using
only the OBJ path, while the confidence becomes much higher (brighter) in
Figure 9(f) with both paths.

##  V Conclusion

We presented an efficient yet effective frame alignment method for the online-
VSR problem. Unlike existing online-VSR methods which compute the motion
fields of each frame separately with a weight-sharing offset estimation
module, we developed a novel temporal motion propagation (TMP) module, which
propagates the motion field of the previous frame to the current frame,
reducing largely the computational cost of offset estimation. We further
presented a motion confidence weighted fusion (MCRF) strategy to more
effectively exploit the well-matched and complementary features for video
enhancement. The proposed method achieved not only higher PSNR/SSIM measures
but also faster inference speed than the current state-of-the-art online-VSR
methods. It can also be extended to other non-online VSR tasks.

## References

  * [1] D. Fuoli, M. Danelljan, R. Timofte, and L. Van Gool, “Fast online video super-resolution with deformable attention pyramid,” in _Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV)_ , 2023, pp. 1735–1744.
  * [2] J. Xiao, X. Jiang, N. Zheng, H. Yang, Y. Yang, Y. Yang, D. Li, and K.-M. Lam, “Online video super-resolution with convolutional kernel bypass grafts,” _IEEE Trans. Multimedia_ , pp. 1–16, 2023.
  * [3] A. A. Baniya, G. Lee, P. Eklund, S. Aryal, and A. Robles-Kelly, “Online Video Super-Resolution using Unidirectional Recurrent Model,” _TechRxiv_ , 11 2022\.
  * [4] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 38, no. 2, pp. 295–307, 2015.
  * [5] C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in _Proc. Eur. Conf. Comput. Vis._ Springer, 2016, pp. 391–407.
  * [6] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in _IEEE Conf. Comput. Vis. Pattern Recog. Worksh._ , 2017, pp. 136–144.
  * [7] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang _et al._ , “Photo-realistic single image super-resolution using a generative adversarial network,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2017, pp. 4681–4690.
  * [8] T. Dai, J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 11 065–11 074.
  * [9] V. Papyan and M. Elad, “Multi-scale patch-based image restoration,” _IEEE Trans. Image Process._ , vol. 25, no. 1, pp. 249–261, 2016.
  * [10] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising,” _IEEE Trans. Image Process._ , vol. 26, no. 7, pp. 3142–3155, 2017.
  * [11] X. Tao, H. Gao, R. Liao, J. Wang, and J. Jia, “Detail-revealing deep video super-resolution,” in _Proc. Int. Conf. Comput. Vis._ , 2017, pp. 4472–4480.
  * [12] D. Liu, Z. Wang, Y. Fan, X. Liu, Z. Wang, S. Chang, and T. Huang, “Robust video super-resolution with learned temporal dynamics,” in _Proc. Int. Conf. Comput. Vis._ , 2017, pp. 2507–2515.
  * [13] M. S. Sajjadi, R. Vemulapalli, and M. Brown, “Frame-recurrent video super-resolution,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2018, pp. 6626–6634.
  * [14] Y. Tian, Y. Zhang, Y. Fu, and C. Xu, “Tdan: Temporally-deformable alignment network for video super-resolution,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 3360–3369.
  * [15] X. Wang, K. C. Chan, K. Yu, C. Dong, and C. Change Loy, “Edvr: Video restoration with enhanced deformable convolutional networks,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog. Worksh._ , 2019, pp. 0–0.
  * [16] P. Yi, Z. Wang, K. Jiang, J. Jiang, and J. Ma, “Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations,” in _Proc. Int. Conf. Comput. Vis._ , 2019, pp. 3106–3115.
  * [17] W. Li, X. Tao, T. Guo, L. Qi, J. Lu, and J. Jia, “Mucan: Multi-correspondence aggregation network for video super-resolution,” in _Proc. Eur. Conf. Comput. Vis._ Springer, 2020, pp. 335–351.
  * [18] Y. Jo, S. W. Oh, J. Kang, and S. J. Kim, “Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2018, pp. 3224–3232.
  * [19] W. Wen, W. Ren, Y. Shi, Y. Nie, J. Zhang, and X. Cao, “Video super-resolution via a spatio-temporal alignment network,” _IEEE Trans. Image Process._ , vol. 31, pp. 1761–1773, 2022.
  * [20] D. Liu, Z. Wang, Y. Fan, X. Liu, Z. Wang, S. Chang, X. Wang, and T. S. Huang, “Learning temporal dynamics for video super-resolution: A deep learning approach,” _IEEE Trans. Image Process._ , vol. 27, no. 7, pp. 3432–3445, 2018.
  * [21] K. C. Chan, X. Wang, K. Yu, C. Dong, and C. C. Loy, “Understanding deformable alignment in video super-resolution,” in _Proc. AAAI_ , vol. 35, no. 2, 2021, pp. 973–981.
  * [22] K. C. Chan, S. Zhou, X. Xu, and C. C. Loy, “Basicvsr++: Improving video super-resolution with enhanced propagation and alignment,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 5972–5981.
  * [23] J. Yu, J. Liu, L. Bo, and T. Mei, “Memory-augmented non-local attention for video super-resolution,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 17 834–17 843.
  * [24] A. Kappeler, S. Yoo, Q. Dai, and A. K. Katsaggelos, “Video super-resolution with convolutional neural networks,” _IEEE Trans. Comput. Imag._ , vol. 2, no. 2, pp. 109–122, 2016.
  * [25] K. Zhou, W. Li, L. Lu, X. Han, and J. Lu, “Revisiting temporal alignment for video restoration,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 6053–6062.
  * [26] S. Nah, S. Baik, S. Hong, G. Moon, S. Son, R. Timofte, and K. Mu Lee, “Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog. Worksh._ , 2019, pp. 0–0.
  * [27] T. Isobe, S. Li, X. Jia, S. Yuan, G. Slabaugh, C. Xu, Y.-L. Li, S. Wang, and Q. Tian, “Video super-resolution with temporal group attention,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 8008–8017.
  * [28] T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman, “Video enhancement with task-oriented flow,” _Int. J. Comput. Vis._ , vol. 127, pp. 1106–1125, 2019\.
  * [29] Z. Wang, P. Yi, K. Jiang, J. Jiang, Z. Han, T. Lu, and J. Ma, “Multi-memory convolutional neural network for video super-resolution,” _IEEE Trans. Image Process._ , vol. 28, no. 5, pp. 2530–2544, 2019.
  * [30] L. Wang, Y. Guo, L. Liu, Z. Lin, X. Deng, and W. An, “Deep video super-resolution using hr optical flow estimation,” _IEEE Trans. Image Process._ , vol. 29, pp. 4323–4336, 2020.
  * [31] H. Song, W. Xu, D. Liu, B. Liu, Q. Liu, and D. N. Metaxas, “Multi-stage feature fusion network for video super-resolution,” _IEEE Trans. Image Process._ , vol. 30, pp. 2923–2934, 2021.
  * [32] F. Li, H. Bai, and Y. Zhao, “Learning a deep dual attention network for video super-resolution,” _IEEE Trans. Image Process._ , vol. 29, pp. 4474–4488, 2020.
  * [33] K. C. Chan, X. Wang, K. Yu, C. Dong, and C. C. Loy, “Basicvsr: The search for essential components in video super-resolution and beyond,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2021, pp. 4947–4956.
  * [34] D. Li, Y. Liu, and Z. Wang, “Video super-resolution using non-simultaneous fully recurrent convolutional network,” _IEEE Trans. Image Process._ , vol. 28, no. 3, pp. 1342–1355, 2019.
  * [35] D. Fuoli, S. Gu, and R. Timofte, “Efficient video super-resolution through recurrent latent space propagation,” in _Proc. Int. Conf. Comput. Vis. Worksh._ IEEE, 2019, pp. 3476–3485.
  * [36] T. Isobe, X. Jia, X. Tao, C. Li, R. Li, Y. Shi, J. Mu, H. Lu, and Y.-W. Tai, “Look back and forth: video super-resolution with explicit temporal difference modeling,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 17 411–17 420.
  * [37] T. Isobe, X. Jia, S. Gu, S. Li, S. Wang, and Q. Tian, “Video super-resolution with recurrent structure-detail network,” in _Proc. Eur. Conf. Comput. Vis._ Springer, 2020, pp. 645–660.
  * [38] T. Isobe, F. Zhu, and S. Wang, “Revisiting temporal modeling for video super-resolution,” in _Proc. Brit. Mach. Vis. Conf._ BMVA Press, 2020. [Online]. Available: https://www.bmvc2020-conference.com/assets/papers/0033.pdf
  * [39] R. Keys, “Cubic convolution interpolation for digital image processing,” _IEEE Trans. Acoust. Speech Signal Process._ , vol. 29, no. 6, pp. 1153–1160, 1981.
  * [40] Y. Zhang, Q. Fan, F. Bao, Y. Liu, and C. Zhang, “Single-image super-resolution based on rational fractal interpolation,” _IEEE Trans. Image Process._ , vol. 27, no. 8, pp. 3782–3797, 2018.
  * [41] J.-S. Choi and M. Kim, “Super-interpolation with edge-orientation-based mapping kernels for low complex $2\times$ upscaling,” _IEEE Trans. Image Process._ , vol. 25, no. 1, pp. 469–483, 2016.
  * [42] A. Brifman, Y. Romano, and M. Elad, “Unified single-image and video super-resolution via denoising algorithms,” _IEEE Trans. Image Process._ , vol. 28, no. 12, pp. 6063–6076, 2019.
  * [43] C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in _Proc. Eur. Conf. Comput. Vis._ Springer, 2016, pp. 391–407.
  * [44] X. Kong, H. Zhao, Y. Qiao, and C. Dong, “Classsr: A general framework to accelerate super-resolution networks by data characteristic,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2021, pp. 12 016–12 025.
  * [45] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformable convolutional networks,” in _Proc. Int. Conf. Comput. Vis._ , 2017, pp. 764–773.
  * [46] M. Haris, G. Shakhnarovich, and N. Ukita, “Recurrent back-projection network for video super-resolution,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 3897–3906.
  * [47] S. Jin, M. Liu, Y. Guo, C. Yao, and M. S. Obaidat, “Multi-frame correlated representation network for video super-resolution,” in _2023 International Conference on Computer, Information and Telecommunication Systems (CITS)_ , 2023, pp. 01–07.
  * [48] B. Xia, J. He, Y. Zhang, Y. Wang, Y. Tian, W. Yang, and L. Van Gool, “Structured sparsity learning for efficient video super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2023, pp. 22 638–22 647.
  * [49] C. Liu and D. Sun, “On bayesian adaptive video super resolution,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 36, no. 2, pp. 346–360, 2013.
  * [50] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep laplacian pyramid networks for fast and accurate super-resolution,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2017, pp. 624–632.
  * [51] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv:1412.6980_ , 2014.
  * [52] I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” _arXiv:1608.03983_ , 2016.
  * [53] M. Chu, Y. Xie, J. Mayer, L. Leal-Taixé, and N. Thuerey, “Learning temporal coherence via self-supervision for gan-based video generation,” _ACM Trans. Graph._ , vol. 39, no. 4, pp. 75–1, 2020.
  * [54] Z. Chi, R. Mohammadi Nasiri, Z. Liu, J. Lu, J. Tang, and K. N. Plataniotis, “All at once: Temporally adaptive multi-frame interpolation with advanced motion modeling,” in _Proc. Eur. Conf. Comput. Vis._ Springer, 2020, pp. 107–123.
  * [55] W.-S. Lai, J.-B. Huang, O. Wang, E. Shechtman, E. Yumer, and M.-H. Yang, “Learning blind video temporal consistency,” in _Proc. Eur. Conf. Comput. Vis._ , 2018, pp. 170–185.