# Zooming SlowMo: An Efficient One-Stage Framework for Space-Time Video Super-
Resolution

Xiaoyu Xiang∗, Yapeng Tian∗, Yulun Zhang∗,
Yun Fu, Jan P. Allebach†, and Chenliang Xu† ∗Equal contribution; †Equal
advising. X. Xiang and J. Allebach are with Department of ECE, Purdue
University, West Lafayette, IN 47907. E-mail: {xiang43<EMAIL_ADDRESS>Y. Tian and C. Xu are with Department of Computer Science, University of
Rochester, Rochester, NY 14627. E-Mail: {yapengtian,
<EMAIL_ADDRESS>Y. Zhang is with Department of ECE, Northeastern
University, Boston, MA 02115. E-mail<EMAIL_ADDRESS>Y. Fu is with
Department of ECE and Khoury College of Computer Science, Northeastern
University, Boston, MA 02115. E-Mail<EMAIL_ADDRESS>
###### Abstract

In this paper,††∗Equal contribution; †Equal advising. we address the space-
time video super-resolution, which aims at generating a high-resolution (HR)
slow-motion video from a low-resolution (LR) and low frame rate (LFR) video
sequence. A naive method is to decompose it into two sub-tasks: video frame
interpolation (VFI) and video super-resolution (VSR). Nevertheless, temporal
interpolation and spatial upscaling are intra-related in this problem. Two-
stage approaches cannot fully make use of this natural property. Besides,
state-of-the-art VFI or VSR deep networks usually have a large frame
reconstruction module in order to obtain high-quality photo-realistic video
frames, which makes the two-stage approaches have large models and thus be
relatively time-consuming. To overcome the issues, we present a one-stage
space-time video super-resolution framework, which can directly reconstruct an
HR slow-motion video sequence from an input LR and LFR video. Instead of
reconstructing missing LR intermediate frames as VFI models do, we temporally
interpolate LR frame features of the missing LR frames capturing local
temporal contexts by a feature temporal interpolation module. Extensive
experiments on widely used benchmarks demonstrate that the proposed framework
not only achieves better qualitative and quantitative performance on both
clean and noisy LR frames but also is several times faster than recent state-
of-the-art two-stage networks. The source code is released in
https://github.com/Mukosame/Zooming-Slow-Mo-CVPR-2020.

###### Index Terms:

One-stage, space-time video super-resolution, feature temporal interpolation,
deformable ConvLSTM.

##  1 Introduction

Figure 1: Example of space-time video super-resolution. We propose a one-stage
STVSR network to directly predict high frame rate (HFR) and high-resolution
(HR) frames from the corresponding low-resolution (LR) and low frame rate
(LFR) frames without explicitly interpolating intermediate LR frames. A HR
intermediate frame $t$ and its neighboring LR frames: $t-1$ and $t+1$ as an
overlayed image are shown. Comparing with a SOTA two-stage method: DAIN
[1]$+$EDVR [60] on the HR intermediate frame $t$, our model is more effective
in handling video motions and therefore restores more accurate visual
structures and sharper edges. Furthermore, our network is more than $3$ times
faster on inference speed with a $4$ times smaller model size than the
DAIN$+$EDVR.

Space-Time Video Super-Resolution (STVSR) [51] aims to automatically generate
a photo-realistic video sequence with high-resolution (HR) and high frame rate
(HFR) from a low space-time resolution input video. Containing clear motion
dynamics and fine image details, HR slow-motion videos are more visually
appealing. They are desired in various applications, such as character
animation, film making, and high-definition television. However, the
reconstruction of such kind of videos is an ill-posed problem.

To address this problem, most existing research works [51, 37, 56, 49, 10, 25]
often introduce some prior knowledge, make strong assumptions, and adopt hand-
crafted regularization. For instance, Shechtman et al. [51] adopted space-time
directional smoothness prior. An assumption made in [37] shows that the
illumination change for the static pixels is not significant. However, these
strong constraints hinder the capacity of the methods to model diverse and
various space-time visual patterns. Moreover, it is computationally expensive
(_e.g_., $\sim$ $60$ frames per hour in [37]) for these methods in terms of
optimization.

Recently, deep convolutional neural networks (CNNs) have shown excellent
performance in various video restoration applications, such as video
deblurring [54], video frame interpolation (VFI) [40], video super-resolution
(VSR) [4], and even space-time video super-resolution (STVSR). To design a
network for such an STVSR task, one straightforward way is to sequentially
combine a VFI method (_e.g_., SepConv [41], ToFlow [67], DAIN [1] _etc_.) and
a VSR method (_e.g_., DUF [19], RBPN [13], EDVR [60] _etc_.) in a two-stage
manner. Specifically, it firstly interpolates missing intermediate low-
resolution (LR) video frames with VFI and then super-resolves all LR frames to
the desired resolution with VSR. However, it is not independent but intra-
related for temporal interpolation and spatial super-resolution in STVSR. By
splitting them into two individual procedures, the two-stage methods fail to
make better usage of this natural property. Moreover, to predict high-quality
video frames, both state-of-the-art (SOTA) VFI and VSR networks require a
large frame reconstruction module. Consequently, the composed two-stage STVSR
network will inevitably contain a huge number of parameters, resulting in high
computational complexity.

To alleviate the above issues, we propose Zooming SlowMo (ZSM), a unified one-
stage STVSR framework, to simultaneously conduct temporal interpolation and
spatial super-resolution. Specifically, instead of synthesizing pixel-wise LR
frames as in two-stage methods, we propose to adaptively learn a deformable
feature interpolation function to temporally interpolate intermediate LR frame
features. The interpolation function contains learnable offsets, which can not
only aggregate useful local temporal contexts but also help the temporal
interpolation handle complex visual motions. Furthermore, we introduce a new
deformable ConvLSTM model to effectively leverage global contexts with
simultaneous temporal alignment and aggregation. We then reconstruct the HR
video frames from the aggregated LR features with a deep SR network. To this
end, the one-stage network can learn end-to-end to map an LR, LFR video
sequence to its HR, HFR counterpart in a sequence-to-sequence manner.
Moreover, we propose a cyclic interpolation loss to guide the frame feature
interpolation and further improve STVSR performance. Extensive experiments
show that our proposed one-stage STVSR framework outperforms SOTA two-stage
methods even with much fewer parameters and is capable of handling multiple
degradations. We illustrate an STVSR example in Figure 1.

The main contributions of this work are three-fold:

  * •

We propose Zooming SlowMo (ZSM), a one-stage STVSR network, to conduct
temporal interpolation and spatial SR simultaneously. Our ZSM is more
effective than two-stage methods by taking advantage of the intra-relatedness
between the two sub-problems. It is also more computationally efficient
because it only requires one frame reconstruction module rather than two large
networks as in two-stage methods.

  * •

We propose a frame feature temporal interpolation network, which leverages
local temporal contexts based on deformable sampling for intermediate LR
frames. We further devise a novel deformable ConvLSTM to explicitly enhance
temporal alignment capacity and exploit global temporal contexts to handle
large motions in videos.

  * •

Our one-stage ZSM method achieves SOTA performance on commonly used benchmarks
(i.e., Vid4 [30] and Vimeo [67]). It is 3$\times$ faster than the two-stage
network: DAIN [1] \+ EDVR [60] and has a nearly 4$\times$ reduction in model
size.

A preliminary version of this manuscript has been presented in [64]. In the
current work, we incorporate additional contents to improve our method and
address the space-time video super-resolution in more challenging conditions:

  * •

We improve model performance via integrating the guided feature interpolation
learning into our one-stage framework.

  * •

We investigate space-time video super-resolution under even noisy conditions,
in which random noises or JPEG compression artifacts corrupt input LR video
frames. Such applications allow us to explore the flexibility and potential
breath of our Zooming SlowMo (ZSM) method.

  * •

Additional and extensive experimental results can demonstrate the
effectiveness of the proposed guided interpolation learning, and further show
the superiority of our one-stage network on tackling more challenging noisy
STVSR tasks.

Figure 2: Overview of our proposed one-stage STVSR framework ZSM. Without
synthesizing intermediate LR frames $I_{t}^{L}$, our ZSM directly predicts
temporally consecutive HR frames. We utilized frame feature temporal
interpolation and deformable ConvLSTM to leverage local and global temporal
contexts, which contribute to exploit temporal information and handle large
motions. It should be noted that we only show two LR input frames from a long
video sequence in the figure for better understanding.

##  2 Related Work

In this section, we discuss works on related topics: video frame interpolation
(VFI), video super-resolution (VSR), and space-time video super-resolution
(STVSR), and image degradation.

###  2.1 Video Frame Interpolation

Video frame interpolation (VFI) aims to synthesize non-existent intermediate
frames between the original frames. Meyer  _et al_. [36] introduced a phase-
based frame interpolation method, which utilizes per-pixel phase modification
to generate intermediate frames. Long _et al_. [33] applied an encoder-decoder
CNN to predict intermediate frames directly. Regarding the frame interpolation
as a local convolution over the two input frames, Niklaus  _et al_. [40, 41]
utilized a CNN to learn a spatially-adaptive convolution kernel for each
pixel. Choi  _et al_. [6] introduced a feature reshaping operation with
Pixelshuffle [53] and a channel attention module for motion estimation. Lee
_et al_. [24] proposed adaptive collaboration of flows as a new warping module
to deal with complex motions. To handle motions more explicitly, many flow-
based video interpolation approaches [18, 32, 38, 2, 1, 39, 43] have also been
proposed. However, these methods usually have inherent issues with
inaccuracies and missing information from optical flow results. Cycle
consistency constraints were utilized in the time domain to supervise the
intermediate frame generation [46, 31, 52]. Recently, [42, 45, 52, 29]
addressed the motion blur and motion aliasing of complex scenes in temporal
frame interpolation. Choi  _et al_. [5] proposed a novel scene adaptation
framework to further improve the frame interpolation models via meta-learning.
Instead of synthesizing the intermediate LR frames as current VFI methods do,
our one-stage framework interpolates features from two neighboring LR frames
to directly synthesize LR feature maps for missing frames without explicit
supervision.

###  2.2 Video Super-Resolution

Video super-resolution (VSR) aims to reconstruct an HR video frame from the
corresponding LR frame (reference frame) and its neighboring LR frames
(supporting frames). How to temporally align the LR supporting frames with the
reference frame is a key problem for VSR. Various VSR approaches [4, 57, 47,
59, 67] used optical flow for explicit temporal alignment. Such a temporal
alignment first estimates motions between the reference frame and each
supporting frame with optical flow and then warps the supporting frame using
the predicted motion map. In recent years, RBPN proposed to incorporate the
single image and multi-frame SR methods for VSR, where LR video frames are
directly concatenated with flow maps. However, there exist some problems. For
example, it is difficult to obtain accurate flow. Flow warping also introduces
artifacts into the aligned frames.

To alleviate this problem, DUF [19] with dynamic filters and TDAN [58] with
deformable alignment was proposed for implicit temporal alignment without
motion estimation. Wang  _et al_. proposed EDVR [60] by extending the
deformable alignment in TDAN with multiscale information. MuCAN [26] applies a
temporal multi-correspondence aggregation method to utilize similar patches in
both intra- and inter-frames. However, most of those methods are many-to-one
architectures, where a batch of LR frames are needed to predict only one HR
frame. Such a process makes the methods computationally inefficient. On the
other hand, recurrent neural networks (RNNs), such as convolutional LSTMs [66]
(ConvLSTM), can ease sequence-to-sequence (S2S) learning. VSR methods [27, 15,
16] widely adopted them to leverage temporal information. However, when
processing videos with large and complex motions, the RNN-based VSR networks
would suffer from obvious performance drop for the lack of explicit temporal
alignment.

To achieve efficient yet effective modeling, unlike existing methods, we
propose a novel ConvLSTM structure ZSM embedded with an explicit state
updating cell for STVSR. Instead of simply combining VFI and VSR networks to
solve the STVSR problem, we propose a more efficient and effective one-stage
framework ZSM. Our ZSM simultaneously learns the temporal feature
interpolation and the spatial super-resolution without synthesizing LR
intermediate frames during inference.

###  2.3 Space-Time Video Super-Resolution

Space-time video super-resolution (STVSR), as a more difficult video
restoration task, aims to super-resolve the space-time resolution according to
the LR and LFR video. Shechtman _et al_. [50] firstly proposed to extend SR to
the space-time domain. STVSR is a highly ill-posed inverse problem, because
pixels are missing in LR frames, and even several entire LR frames are
unavailable. To increase video resolution both in time and space, Shechtman
_et al_. [50] combined information from multiple video sequences of dynamic
scenes, which can be obtained at sub-pixel and sub-frame misalignments with a
directional space-time smoothness regularization to constrain the ill-posed
problem. Mudenagudi [37] posed STVSR as a reconstruction problem, which
utilized the Maximum a posteriori-Markov Random Field [11] and solved the
optimization problem via graph-cuts [3]. Local orientation and local motion
were exploited in [56] to steer spatio-temporal regression kernels. Shahar
_et al_. [49] proposed to exploit a space-time patch recurrence prior within
natural videos for STVSR. However, these methods have limited capacity to
model rich and complex space-time visual patterns. Plus, the optimization in
these methods is usually computationally expensive.

Very recently, [21] proposed a multi-scale temporal loss. [14] concatenated LR
images and pre-computed optical flow for intermediate feature estimation and
refinement. [20] used optical flow to explicitly warp the features. [65]
introduced a news perspective to exploit the STVSR problem from the temporal
profile. Unlike previous methods, we propose a one-stage framework ZSM to
directly learn the mapping between partial LR observations and HR video
frames, achieving accurate and fast STVSR.

###  2.4 Image Degradation

Besides downsampling, captured images might also be degraded by other
corruptions, such as noise, blur, and compression artifacts. To restore high-
quality images from the corresponding degraded inputs, many image restoration
methods have been developed. A pioneer CNN-based image restoration method is
from [17], in which Jain and Seung firstly propose a CNN architecture with
stacked convolutional layers for natural image denoising. Dong  _et al_.
introduce shallow plain CNN networks to address image super-resolution in [9]
and handle JPEG compression artifacts [8]. Furthermore, more advanced and
deeper models are proposed in [62, 12, 68, 35, 55, 69, 70, 72].

To further demonstrate the robustness of the proposed approach to different
image corruptions, we explore STVSR under noisy conditions. We introduce
random noise and JPEG compression artifacts into LR inputs, respectively, such
that our STVSR needs to hallucinate missing HR frames from noisy LR frames.

##  3 Space-Time Video Super-Resolution

In this section, we first give an overview of the proposed space-time video
super-resolution framework in Sec. 3.1. Built upon this framework, we then
propose a novel frame feature temporal interpolation network in Sec. 3.2;
deformable ConvLSTM in Sec. 3.3; frame reconstruction module in Sec. 3.4;
guided feature interpolation learning in Sec. 3.5. Finally, we provide details
about our implementation in Sec. 3.6.

###  3.1 Overview

Given an LFR and LR video sequence:
$\mathcal{I}^{L}=\\{I_{2t-1}^{L}\\}_{t=1}^{n+1}$, we aim to reconstruct the
corresponding slow-motion high-resolution video sequence:
$\mathcal{I}^{H}=\\{I_{t}^{H}\\}_{t=1}^{2n+1}$. For intermediate HR video
frames $\\{I_{2t}^{H}\\}_{t=1}^{n}$, there are no LR counterparts in the input
video sequence. To efficiently and effectively increase video resolution in
both time and space domains, we present a one-stage space-time video super-
resolution framework: Zooming SlowMo (ZSM) as shown in Figure 2. The framework
mainly includes four parts: feature extractor, frame feature temporal
interpolation module, deformable ConvLSTM network, HR frame reconstruction
module, and guided feature interpolation learning module.

We first utilize a feature extractor with one convolutional layer and $k_{1}$
residual units to extract visual feature maps:
$\\{F_{2t-1}^{L}\\}_{t=1}^{n+1}$ from each input video frame. Taking the
extracted feature maps as input, we then synthesize feature maps:
$\\{F_{2t}^{L}\\}_{t=1}^{n}$ of intermediate LR frames with a feature temporal
interpolation module. In addition, to better exploit temporal information, we
propose a deformable ConvLSTM to process the temporally consecutive feature
maps: $\\{F_{t}^{L}\\}_{t=1}^{2n+1}$. Different from vanilla ConvLSTM, our
deformable ConvLSTM can simultaneously employ temporal alignment and
aggregation. Finally, we restore the HR slow-mo video frames from the
aggregated feature maps. Since the features for reconstructing intermediate
frames are synthesized, there will be feature synthesis errors that will be
propagated into restored HR frames, making the predicted video sequence suffer
from jitters. To alleviate this problem, we further propose a guided feature
interpolation learning mechanism.

Figure 3: Frame feature temporal interpolation based on deformable sampling.
Since approximated $F_{2}^{L}$ is utilized to reconstruct the corresponding HR
video frame, it will implicitly enforce learnable convolution offsets to
capture temporally dynamic local contexts and be motion-aware.

###  3.2 Frame Feature Temporal Interpolation

Given extracted feature maps: $F_{1}^{L}$ and $F_{3}^{L}$ from input LR
frames: $I_{1}^{L}$ and $I_{3}^{L}$, we aim to synthesize the feature map
$F_{2}^{L}$ corresponding to the unavailable intermediate LR video frame
$I_{2}^{L}$. Previous video frame interpolation models usually employ temporal
interpolation on pixel-wise frames, which leads to a two-stage space-time
super-resolution design. Different from previous approaches, we propose to
learn a feature-wise temporal interpolation function $f(\cdot)$ to directly
synthesize the intermediate LR feature map $F_{2}^{L}$ (see Figure 3). A
general form of the proposed temporal interpolation function can be formulated
as:

$F_{2}^{L}=f(F_{1}^{L},F_{3}^{L})=H(T_{1}(F_{1}^{L},\Phi_{1}),T_{3}(F_{3}^{L},\Phi_{3}))\enspace,$
(1)

where $T_{1}(\cdot)$ and $T_{3}(\cdot)$ are two feature sampling functions and
$\Phi_{1}$ and $\Phi_{3}$ are sampling parameters; $H(\cdot)$ is a weighted
blending function to temporally aggregate sampled features.

For obtaining accurate $F_{2}^{L}$, $T_{1}(\cdot)$ should be able to capture
forward motion information between $F_{1}^{L}$ and $F_{2}^{L}$, and
$T_{3}(\cdot)$ should be able to capture backward motion information between
$F_{3}^{L}$ and $F_{2}^{L}$. However, $F_{2}^{L}$ is unavailable for directly
computing forward and backward temporally dynamic information in STVSR.

To mitigate this issue, we utilize temporal information between $F_{1}^{L}$
and $F_{3}^{L}$ to approximate the forward and backward motion. Motivated by
deformable alignment in [58] for video super-resolution, we propose to use
deformable sampling to implicitly capture dynamic motion information for
feature temporal interpolation. Exploiting rich locally temporal contexts by
deformable convolutions in the sampling functions, the proposed frame feature
temporal interpolation is capable of handling large video motions.

The two temporal sampling functions share the same architecture but have
different parameters. For simplicity, we take the $T_{1}(\cdot)$ as an
example. It uses LR feature maps $F_{1}^{L}$ and $F_{3}^{L}$ as the input to
predict a temporally dynamic offset for sampling the $F_{1}^{L}$:

$\Delta p_{1}=g_{1}([F_{1}^{L},F_{3}^{L}])\enspace,$ (2)

where $\Delta p_{1}$ is a learnable convolution offset and also denotes the
sampling parameter: $\Phi_{1}$; $g_{1}$ refers to a general function of
multiple convolution layers; $[,]$ is a channel-wise concatenation operator.
Using the learned offset, the sampling function can be implemented with a
deformable convolution [7, 73]:

$T_{1}(F_{1}^{L},\Phi_{1})=DConv(F_{1}^{L},\Delta p_{1})\enspace.$ (3)

Similarly, we can predict an offset: $\Delta
p_{3}=g_{3}([F_{3}^{L},F_{1}^{L}])$ as the sampling parameter: $\Phi_{3}$ and
then produce sampled features $T_{3}(F_{3}^{L},\Phi_{3})$ via a deformable
convolution.

To blend the sampled frame features, we adopt a simple linear weighted
blending function $H(\cdot)$:

$F_{2}^{L}=\alpha*T_{1}(F_{1}^{L},\Phi_{1})+\beta*T_{3}(F_{3}^{L},\Phi_{3})\enspace,$
(4)

where $\alpha$ and $\beta$ are two $1\times 1$ convolution kernels and $*$
denotes a convolution operator. Since the synthesized $F_{2}^{L}$ will be used
to predict the intermediate HR video frame $I_{2}^{H}$, it will enforce the
$F_{2}^{L}$ to be close to the real intermediate LR frame feature map.
Therefore, the offsets: $\Delta p_{1}$ and $\Delta p_{3}$ will implicitly
learn to capture the forward and backward dynamic information, respectively.

Applying the devised deformable temporal interpolation function to LR feature
maps: $\\{F_{2t-1}^{L}\\}_{t=1}^{n+1}$, we can predict a sequence of
intermediate LR frame feature maps: $\\{F_{2t}^{L}\\}_{t=1}^{n}$.

###  3.3 Deformable ConvLSTM

Given the consecutive LR frame feature maps: $\\{F_{t}^{L}\\}_{t=1}^{2n+1}$,
we will use a sequence-to-sequence mapping to reconstruct the corresponding HR
frames. It has been widely proved in video restoration tasks [67, 57, 60] that
temporal information is essential. Therefore, instead of generating HR frames
from individual feature maps, we adaptively aggregate temporal contexts from
neighboring video frames. ConvLSTM [66] is a popular 2D sequential data
modeling approach, and we can utilize it for temporal aggregation. At the time
step $t$, the ConvLSTM will update hidden state $h_{t}$ and cell state $c_{t}$
with:

$h_{t},c_{t}=ConvLSTM(h_{t-1},c_{t-1},{F}_{t}^{L})\enspace.$ (5)

From its internal state updating mechanism [66], we can observe that the
ConvLSTM can only implicitly capture temporal information between previous
states: $h_{t-1}$ and $c_{t-1}$ and the current input frame feature map with
small 2D convolution receptive fields. So, ConvLSTM has a relatively limited
ability to deal with large motions in natural videos. If there are large
motions in a video, a severe temporal mismatch between previous states and
$F_{t}^{L}$ will occur. With the state updating, $h_{t-1}$ and $c_{t-1}$ will
propagate temporally mismatched “noisy” content but not useful global temporal
contexts into $h_{t}$. Consequently, the generated HR video frame $I_{t}^{H}$
from $h_{t}$ will suffer from annoying visual artifacts.

To address the large motion issue and effectively leverage global temporal
contexts, we embed state-updating cells with deformable alignment into vanilla
ConvLSTM (see Figure 4):

$\begin{split}&\Delta p^{h}_{t}=g^{h}([h_{t-1},F_{t}^{L}])\enspace,\\\ &\Delta
p^{c}_{t}=g^{c}([c_{t-1},F_{t}^{L}])\enspace,\\\
&h^{a}_{t-1}=DConv(h_{t-1},\Delta p^{h}_{t})\enspace,\\\
&c^{a}_{t-1}=DConv(c_{t-1},\Delta p^{c}_{t})\enspace,\\\
&h_{t},c_{t}=ConvLSTM(h^{a}_{t-1},c^{a}_{t-1},F_{t}^{L})\enspace,\end{split}$
(6)

where $g^{h}$ and $g^{c}$ refer to general functions of convolution layers,
$\Delta p^{h}_{t}$ and $\Delta p^{c}_{t}$ are learned offsets, and
$h^{a}_{t-1}$ and $c^{a}_{t-1}$ are temporally aligned hidden and cell states,
respectively. Different from the vanilla ConvLSTM, we explicitly enforce
$h_{t-1}$ and $c_{t-1}$ to align with the current input feature map:
$F_{t}^{L}$ in our deformable ConvLSTM, which makes it more effective in
handling video motions. Besides, to make full use of temporal information, we
perform Deformable ConvLSTM in a bidirectional fashion [48]. We take
temporally reversed feature maps as inputs for the same Deformable ConvLSTM
and concatenate hidden states from forward and backward passes as the final
output hidden state $h_{t}$222We use $h_{t}$ to refer to the final hidden
state, but it will denote a concatenated hidden state in the Bidirectional
Deformable ConvLSTM. for HR video frame reconstruction.

Figure 4: Deformable ConvLSTM for better leveraging global temporal contexts
and dealing with fast motion videos. At time step $t$, we learn deformable
sampling to adaptively align hidden state $h_{t-1}$ and cell state $c_{t-1}$
with current input feature map: $F_{t}^{L}$.

###  3.4 Frame Reconstruction

To reconstruct HR frames, we use a shared frame synthesis network, which takes
individual hidden state $h_{t}$ as input and predicts the corresponding HR
frame. It has $k_{2}$ stacked residual units [28] for learning deep frame
features and adopts a sub-pixel upscaling network with PixelShuffle as in [53]
to obtain HR frames $\\{I_{t}^{H}\\}_{t=1}^{2n+1}$. To optimize the network,
we utilize a reconstruction loss function:

$l_{rec}=\sqrt{||I^{GT}_{t}-I_{t}^{H}||^{2}+\epsilon^{2}}\enspace,$ (7)

where $I^{GT}_{t}$ denotes the $t$-th ground-truth HR frame, Charbonnier
penalty function [23] is used as the objective function, and $\epsilon$ is
empirically set to $1\times 10^{-3}$. Since spatial and temporal SR problems
are intra-related in our STVSR task, the proposed model can simultaneously
learn this spatio-temporal interpolation using only supervision from HR frames
in an end-to-end manner.

TABLE I: Quantitative comparison of two-stage VFI and VSR methods and our
results on Vid4 [30] dataset. The best two results are highlighted in red and
blue colors, respectively. We measure the total runtime on the entire Vid4
dataset [30]. Note that we omit the baseline methods with Bicubic when
comparing in terms of runtime.

VFI Method |  SR Method | Vid4 |  Parameters (Million) |  Runtime-VFI (s) |  Runtime-SR (s) |  Total Runtime (s) |  Average Runtime (s/frame)
---|---|---|---|---|---|---|---
PSNR | SSIM
SuperSloMo [18] | Bicubic | 22.84 | 0.5772 | 19.8 | 0.28 | - | - | -
SuperSloMo [18] | RCAN [71] | 23.80 | 0.6397 | 19.8+16.0 | 0.28 | 68.15 | 68.43 | 0.4002
SuperSloMo [18] | RBPN [13] | 23.76 | 0.6362 | 19.8+12.7 | 0.28 | 82.62 | 82.90 | 0.4848
SuperSloMo [18] | EDVR [60] | 24.40 | 0.6706 | 19.8+20.7 | 0.28 | 24.65 | 24.93 | 0.1458
SepConv [41] | Bicubic | 23.51 | 0.6273 | 21.7 | 2.24 | - | - | -
SepConv [41] | RCAN [71] | 24.92 | 0.7236 | 21.7+16.0 | 2.24 | 68.15 | 70.39 | 0.4116
SepConv [41] | RBPN [13] | 26.08 | 0.7751 | 21.7+12.7 | 2.24 | 82.62 | 84.86 | 0.4963
SepConv [41] | EDVR [60] | 25.93 | 0.7792 | 21.7+20.7 | 2.24 | 24.65 | 26.89 | 0.1572
DAIN [1] | Bicubic | 23.55 | 0.6268 | 24.0 | 8.23 | - | - | -
DAIN [1] | RCAN [71] | 25.03 | 0.7261 | 24.0+16.0 | 8.23 | 68.15 | 76.38 | 0.4467
DAIN [1] | RBPN [13] | 25.96 | 0.7784 | 24.0+12.7 | 8.23 | 82.62 | 90.85 | 0.5313
DAIN [1] | EDVR [60] | 26.12 | 0.7836 | 24.0+20.7 | 8.23 | 24.65 | 32.88 | 0.1923
ZSM (Ours) | 26.49 | 0.8028 | 11.10 | - | - | 10.36 | 0.0606

###  3.5 Guided Feature Interpolation Learning

In addition, we employ a cyclic interpolation loss to guide the learning of
frame feature interpolation with LR frames. It utilizes the inherent temporal
coherence in natural video (see Figure 5).

Figure 5: Feature interpolation learning guided by LR frames. The cyclic
interpolation loss is computed between the ground truth LR frames and the 1st-
order and 2nd-order interpolated LR frames. By minimizing the difference of LR
frames and their corresponding interpolation acquired at each order, our
temporal interpolation module can be self-supervised with the natural temporal
coherence.

Given a sequence of LR, LFR inputs $\\{I^{L}_{2t-1}\\}_{t=1}^{n+1}$, we can
obtain the extracted frame feature maps $\\{F^{L}_{2t-1}\\}^{n+1}_{t=1}$, and
the interpolated intermediate frame feature maps $\\{F_{2t}^{L}\\}_{t=1}^{n}$.
During the training phase, we have a a set of LR ground truth
$\\{I^{LGT}_{t}\\}_{t=1}^{2n+1}$. The first-order interpolation loss is
defined as:

$l^{1}_{i}=||I^{LGT}_{2t}-\rho(F^{L}_{2t})||_{c},$ (8)

where $||\cdot||_{c}$ stands for the Charbonnier penalty function as defined
in Equation (7), and $\rho$ represents the LR synthesis module that turns
feature maps into the corresponding LR frames. We apply $k_{3}$ stacked
residual blocks [28] and a convolution layer, which is similar to the the
design in Section 3.4, to predict the LR frames.

If we conduct feature interpolation on the acquired intermediate frame feature
maps $\\{F_{2t}^{L}\\}_{t=1}^{n}$, we can get a sequence of re-interpolated
feature maps $\\{F_{2t+1}^{{}^{\prime}L}\\}_{t=1}^{n-1}$. Similar to Equation
(8), the second-order cyclic interpolation loss is defined as:

$l^{2}_{i}=||I^{LGT}_{2t+1}-\rho(F^{{}^{\prime}L}_{2t+1})||_{c}.$ (9)

The overall training loss is the weighted summation of the reconstruction
loss, and the 1st- and 2nd-order cyclic interpolation losses:

$L=\lambda_{1}l_{rec}+\lambda_{2}l_{i}^{1}+\lambda_{3}l_{i}^{2}.$ (10)

Overlayed LR frames HR Overlayed LR SepConv+RCAN SepConv+RBPN SepConv+EDVR
DAIN+Bicubic DAIN+RCAN DAIN+RPBN DAIN+EDVR ZSM (Ours)
---
Overlayed LR frames HR Overlayed LR SepConv+RCAN SepConv+RBPN SepConv+EDVR
DAIN+Bicubic DAIN+RCAN DAIN+RPBN DAIN+EDVR ZSM (Ours)
Overlayed LR frames HR Overlayed LR SepConv+RCAN SepConv+RBPN SepConv+EDVR
DAIN+Bicubic DAIN+RCAN DAIN+RPBN DAIN+EDVR ZSM (Ours)
Overlayed LR frames HR Overlayed LR SepConv+RCAN SepConv+RBPN SepConv+EDVR
DAIN+Bicubic DAIN+RCAN DAIN+RPBN DAIN+EDVR ZSM (Ours)
Figure 6: Visual comparisons of different methods on Vid4 and Vimeo datasets.
Our one-stage Zooming SlowMo model (ZSM) can generate more visually appealing
HR video frames with fewer blurring artifacts and more accurate image
structures.

(a) Overlayed LR

(b) HR

(c) Toflow+Sepconv+EDVR

(d) Toflow+DAIN+EDVR

(e) ZSM (Ours)

Figure 7: Visual comparisons of different methods on noisy input video frames.
Our one-stage Zooming SlowMo model (ZSM) can effectively restore clean missing
HR frames from noisy LR frames.

###  3.6 Implementation Details

In our implementation, $k_{1}=5$, $k_{2}=40$, and $k_{3}$ residual blocks are
used in feature extraction, HR frame reconstruction, and LR frame
reconstruction modules respectively. We randomly crop a sequence of down-
sampled image patches with the size of $32\times 32$ and take out the odd-
indexed 4 frames as LFR and LR inputs, and the corresponding consecutive
7-frame sequence of $4\times$333Considering recent state-of-the-art methods
(_e.g_., EDVR [60] and RBPN [13]) use only 4 as the upscaling factor, we adopt
the same practice. size as supervision. Besides, we perform data augmentation
by randomly rotating $90^{\circ}$, $180^{\circ}$, and $270^{\circ}$, and
horizontal-flipping. We adopt a Pyramid, Cascading and Deformable (PCD)
structure in [60] to employ deformable alignment and apply Adam [22]
optimizer, where we decay the learning rate with a cosine annealing for each
batch [34] from 4$\times 10^{-4}$ to 1$\times 10^{-7}$. We set batch size as
24 and train the network on $2$ Nvidia Titan XP GPUs.

(a) Overlayed LR

(b) HR

(c) DNCNN+DAIN+RBPN

(d) RNAN+DAIN+RBPN

(e) ZSM (Ours)

Figure 8: Visual comparisons of different methods on compressed LR frames. The
first, second, third, and fourth rows are results for QR = 10, 20, 30, and 40,
respectively.

##  4 Experiments and Analysis

###  4.1 Experimental Setup

Datasets

Vimeo-90K is used as the training set [67], including over 60,000 7-frame
training video sequences. Vimeo-90K is widely used in previous VFI and VSR
works [2, 1, 58, 13, 60]. Besides, Vid4 [30] and Vimeo testset [67] are used
as the evaluate datasets. To compare the performance of different methods
under different motion conditions, we split the Vimeo testset into fast
motion, medium motion, and slow motion sets as in [13], including $1225$,
$4977$ and $1613$ video clips, respectively. We remove $5$ videos from the
original medium motion set and $3$ videos from the slow motion set, which
include consecutively all-black frames that will lead to infinite values when
calculating PSNR. We generate LR frames by bicubic downsampling with
factor$=4$ and use odd-indexed LR frames as inputs for predicting the
corresponding consecutive HR and HFR video frames.

Evaluation Metrics

We adopt Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index
(SSIM) [61] to evaluate STVSR performance. To measure the efficiency of
different methods, we also compare the model parameters and inference time on
the entire Vid4 [30] dataset using an Nvidia Titan XP GPU.

TABLE II: Quantitative comparison of our one-stage ZSM and two-stage VFI and
VSR methods on Vimeo-90K [67] testset. The best two results are highlighted in
red and blue colors, respectively.

VFI Method |  SR Method | Vimeo-Fast | Vimeo-Medium | Vimeo-Slow
---|---|---|---|---
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
SuperSloMo [18] | Bicubic | 31.88 | 0.8793 | 29.94 | 0.8477 | 28.37 | 0.8102
SuperSloMo [18] | RCAN [71] | 34.52 | 0.9076 | 32.50 | 0.8884 | 30.69 | 0.8624
SuperSloMo [18] | RBPN [13] | 34.73 | 0.9108 | 32.79 | 0.8930 | 30.48 | 0.8584
SuperSloMo [18] | EDVR [60] | 35.05 | 0.9136 | 33.85 | 0.8967 | 30.99 | 0.8673
SepConv [41] | Bicubic | 32.27 | 0.8890 | 30.61 | 0.8633 | 29.04 | 0.8290
SepConv [41] | RCAN [71] | 34.97 | 0.9195 | 33.59 | 0.9125 | 32.13 | 0.8967
SepConv [41] | RBPN [13] | 35.07 | 0.9238 | 34.09 | 0.9229 | 32.77 | 0.9090
SepConv [41] | EDVR [60] | 35.23 | 0.9252 | 34.22 | 0.9240 | 32.96 | 0.9112
DAIN [1] | Bicubic | 32.41 | 0.8910 | 30.67 | 0.8636 | 29.06 | 0.8289
DAIN [1] | RCAN [71] | 35.27 | 0.9242 | 33.82 | 0.9146 | 32.26 | 0.8974
DAIN [1] | RBPN [13] | 35.55 | 0.9300 | 34.45 | 0.9262 | 32.92 | 0.9097
DAIN [1] | EDVR [60] | 35.81 | 0.9323 | 34.66 | 0.9281 | 33.11 | 0.9119
ZSM (Ours) | 36.96 | 0.9444 | 35.56 | 0.9385 | 33.50 | 0.9166

###  4.2 Space-Time Video Super-resolution

We compare the performance of our one-stage Zooming SlowMo (ZSM) network to
other two-stage methods that are composed of state-of-the-art (SOTA) VFI and
VSR networks. Three recent SOTA VFI approaches,
SepConv444https://github.com/sniklaus/sepconv-slomo [41], Super-SloMo555Since
there is no official code released, we used an unofficial PyTorch [44]
implementation from https://github.com/avinashpaliwal/Super-SloMo. [18], and
DAIN666https://github.com/baowenbo/DAIN [1], are compared. Besides, three SOTA
SR models, including a single-image SR model,
RCAN777https://github.com/yulunzhang/RCAN [71], and two recent VSR models,
RBPN888https://github.com/alterzero/RBPN-PyTorch [13] and
EDVR999https://github.com/xinntao/EDVR [60], are adopted to generate HR frames
from both original and interpolated LR frames.

Quantitative results on Vid4 and Vimeo testsets are shown in Tables I and II.
From these tables, we can observe the following facts: (1) DAIN+EDVR is the
best performing two-stage method among the compared 12 approaches; (2) VFI
model matters, especially for videos with fast motion. Although RBPN and EDVR
perform much better than RCAN for SR, however, when equipped with more recent
SOTA VFI network DAIN, DAIN+RCAN can achieve a comparable or even better
performance than SepConv+RBPN and SepConv+EDVR on the Vimeo-Fast testset; (3)
VSR model also matters. For example, equipped with the same VFI network DAIN,
EDVR keeps achieving better STVSR performance than other SR methods. Moreover,
we can observe that our ZSM outperforms the DAIN+EDVR by $0.19$dB on Vid4,
$0.25$dB on Vimeo-Slow, $0.75$dB on Vimeo-Medium, and $1.00$dB on Vimeo-Fast
in terms of PSNR. Such significant improvements obtained on fast-motion videos
demonstrate that our one-stage approach with simultaneously leveraging local
and global temporal contexts can better handle diverse space-time patterns,
including challenging large motions than two-stage methods.

TABLE III: Quantitative comparisons of our results and three-stage denoising,
VFI and VSR methods on video frames with noise. The best two results for each
quality factor(QF) are highlighted in red and blue colors, respectively.

DN Method | VFI Method | SR Method | Vid4 | Vimeo-Fast | Vimeo-Medium | Vimeo-Slow
---|---|---|---|---|---|---
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
ToFlow [67] | SepConv [41] | RBPN [13] | 22.78 | 0.5692 | 29.03 | 0.8376 | 28.46 | 0.8140 | 27.54 | 0.7826
SepConv [41] | EDVR [60] | 22.79 | 0.5692 | 29.02 | 0.8370 | 28.44 | 0.8131 | 27.52 | 0.7819
DAIN [1] | RBPN [13] | 22.79 | 0.5687 | 29.08 | 0.8388 | 28.50 | 0.8144 | 27.56 | 0.7830
DAIN [1] | EDVR [60] | 22.80 | 0.5693 | 29.08 | 0.8387 | 28.49 | 0.8144 | 27.55 | 0.7825
ZSM (Ours) | 23.91 | 0.6514 | 31.49 | 0.8446 | 30.49 | 0.8594 | 29.36 | 0.8321

Furthermore, we also compare the efficiency of different networks and show
their model sizes and runtime in Table I. To synthesize high-quality frames,
SOTA VFI and VSR networks usually come with very large frame reconstruction
modules. As a result, the composed two-stage SOTA STVSR networks will have a
large number of parameters. Our one-stage model is with only one frame
reconstruction module, thus has much fewer parameters than the SOTA two-stage
networks. Table I shows that our ZSM is more than $4\times$ and $3\times$
smaller than the DAIN+EDVR and DAIN+RBPN, respectively. In terms of runtime,
the smaller model size makes our network more than $8\times$ faster than
DAIN+RBPN and $3\times$ faster than the DAIN+EDVR. Our method is still more
than $2\times$ faster compared to two-stage methods with a fast VFI network:
SuperSlowMo. These results can validate the superiority of our one-stage ZSM
model in terms of efficiency.

Qualitative results of these different methods are illustrated in Figure 6.
Our method demonstrates noticeably perceptual improvements over other compared
two-stage methods. Clearly, our proposed network can synthesize visually
appealing HR video frames with more accurate structures, more fine details,
and fewer blurry artifacts, even for challenging video sequences with large
motions. We can also observe that the SOTA VFI methods: SepConv and DAIN fail
to handle fast motions. Consequently, the composed two-stage methods tend to
generate severe motion blurs in output frames. In the proposed one-stage
framework, we simultaneously learn temporal and spatial SR by exploring the
intra-relatedness within natural videos. Thus, our proposed framework: ZSM can
well address the large motion issue in temporal SR even with a much smaller
model.

TABLE IV: Quantitative comparison of our results and three-stage CAR, VFI and
VSR methods on compressed testsets (QF = 10, 20, 30, and 40). The best two
results are highlighted in red and blue colors, respectively.

QF | CAR Method | VFI Method | SR Method | Vid4 | Vimeo-Fast | Vimeo-Medium | Vimeo-Slow
---|---|---|---|---|---|---|---
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
10 | DNCNN [68] | SepConv [41] | RBPN [13] | 21.33 | 0.4659 | 27.84 | 0.7917 | 26.59 | 0.7475 | 25.46 | 0.7010
SepConv [41] | EDVR [60] | 21.31 | 0.4647 | 27.83 | 0.7909 | 26.58 | 0.7466 | 25.45 | 0.7001
DAIN [1] | RBPN [13] | 21.37 | 0.4688 | 27.92 | 0.7958 | 26.65 | 0.7510 | 25.49 | 0.7028
DAIN [1] | EDVR [60] | 21.35 | 0.4679 | 27.92 | 0.7955 | 26.64 | 0.7504 | 25.48 | 0.7020
RNAN [72] | SepConv [41] | RBPN [13] | 20.41 | 0.4864 | 23.80 | 0.7926 | 23.61 | 0.7607 | 22.83 | 0.7171
SepConv [41] | EDVR [60] | 20.40 | 0.4858 | 23.80 | 0.7925 | 23.60 | 0.7604 | 22.82 | 0.7165
DAIN [1] | RBPN [13] | 20.41 | 0.4859 | 23.81 | 0.7936 | 23.60 | 0.7610 | 22.82 | 0.7172
DAIN [1] | EDVR [60] | 20.41 | 0.4856 | 23.81 | 0.7936 | 23.59 | 0.7609 | 22.81 | 0.7167
ZSM (Ours) | 22.03 | 0.5216 | 29.52 | 0.8367 | 28.13 | 0.8009 | 26.64 | 0.7532
20 | DNCNN [68] | SepConv [41] | RBPN [13] | 22.10 | 0.5102 | 29.36 | 0.8232 | 28.03 | 0.7834 | 26.76 | 0.7396
SepConv [41] | EDVR [60] | 22.09 | 0.5089 | 29.36 | 0.8225 | 28.02 | 0.7825 | 26.75 | 0.7386
DAIN [1] | RBPN [13] | 22.14 | 0.5132 | 29.46 | 0.8266 | 28.10 | 0.7865 | 26.79 | 0.7411
DAIN [1] | EDVR [60] | 22.13 | 0.5121 | 29.46 | 0.8263 | 28.09 | 0.7859 | 26.78 | 0.7403
RNAN [72] | SepConv [41] | RBPN [13] | 20.91 | 0.5292 | 24.93 | 0.8197 | 24.76 | 0.7920 | 24.01 | 0.7527
SepConv [41] | EDVR [60] | 20.90 | 0.5285 | 24.92 | 0.8196 | 24.76 | 0.7917 | 24.01 | 0.7524
DAIN [1] | RBPN [13] | 20.93 | 0.5293 | 24.95 | 0.8213 | 24.77 | 0.7927 | 24.00 | 0.7530
DAIN [1] | EDVR [60] | 20.93 | 0.5290 | 24.95 | 0.8214 | 24.76 | 0.7927 | 24.00 | 0.7528
ZSM (Ours) | 22.78 | 0.5707 | 31.00 | 0.8607 | 29.54 | 0.8300 | 27.95 | 0.7864
30 | DNCNN [68] | SepConv [41] | RBPN [13] | 22.52 | 0.5361 | 30.08 | 0.8377 | 28.73 | 0.8009 | 27.43 | 0.7593
SepConv [41] | EDVR [60] | 22.51 | 0.5354 | 30.09 | 0.8372 | 28.72 | 0.8000 | 27.41 | 0.7583
DAIN [1] | RBPN [13] | 22.57 | 0.5393 | 30.19 | 0.8410 | 28.81 | 0.8039 | 27.46 | 0.7608
DAIN [1] | EDVR [60] | 22.56 | 0.5388 | 30.20 | 0.8401 | 28.80 | 0.8033 | 27.44 | 0.7598
RNAN [72] | SepConv [41] | RBPN [13] | 21.41 | 0.5569 | 25.54 | 0.8345 | 25.35 | 0.8084 | 24.64 | 0.7717
SepConv [41] | EDVR [60] | 21.41 | 0.5567 | 25.54 | 0.8344 | 25.34 | 0.8082 | 24.63 | 0.7715
DAIN [1] | RBPN [13] | 21.45 | 0.5576 | 25.58 | 0.8365 | 25.36 | 0.8095 | 24.63 | 0.7721
DAIN [1] | EDVR [60] | 21.44 | 0.5577 | 25.58 | 0.8367 | 25.36 | 0.8095 | 24.63 | 0.7720
ZSM (Ours) | 23.25 | 0.6013 | 31.81 | 0.8733 | 30.30 | 0.8082 | 28.65 | 0.8042
40 | DNCNN [68] | SepConv [41] | RBPN [13] | 22.83 | 0.5559 | 30.51 | 0.8464 | 29.17 | 0.8120 | 27.84 | 0.7720
SepConv [41] | EDVR [60] | 22.83 | 0.5560 | 30.51 | 0.8458 | 29.16 | 0.8110 | 27.83 | 0.7710
DAIN [1] | RBPN [13] | 22.88 | 0.5590 | 30.63 | 0.8497 | 29.26 | 0.8149 | 27.88 | 0.7734
DAIN [1] | EDVR [60] | 22.88 | 0.5592 | 30.64 | 0.8496 | 29.25 | 0.8144 | 27.86 | 0.7725
RNAN [72] | SepConv [41] | RBPN [13] | 21.63 | 0.5764 | 25.87 | 0.8426 | 25.71 | 0.8190 | 25.01 | 0.7837
SepConv [41] | EDVR [60] | 21.63 | 0.5764 | 25.87 | 0.8425 | 25.70 | 0.8187 | 25.01 | 0.7836
DAIN [1] | RBPN [13] | 21.67 | 0.5774 | 25.92 | 0.8449 | 25.73 | 0.8202 | 25.01 | 0.7842
DAIN [1] | EDVR [60] | 21.67 | 0.5779 | 25.92 | 0.8451 | 25.73 | 0.8203 | 25.01 | 0.7842
ZSM (Ours) | 23.59 | 0.6226 | 32.40 | 0.8818 | 30.85 | 0.8560 | 29.15 | 0.8164

###  4.3 Noisy Space-Time Video Super-resolution

Real-world videos are usually compressed and come with complicated noise. To
further validate the robustness of the proposed space-time video super-
resolution method on noisy data, we add random noise and JPEG compression
artifacts into LR video frames [63], respectively.

####  4.3.1 Random Noise

In our experiments, we train our model on the Vimeo-90K dataset with a mixture
of Gaussian noise and 10% salt-and-pepper noise is added to input LR frames as
in ToFlow [67]. We compare our method with four models: ToFlow+SepConv+RBPN,
Toflow+SepConv+EDVR, Toflow+DAIN+RBPN, and Toflow+DAIN+EDVR on Vid4, Vimeo-
fast, Vimeo-Medium, and Vimeo-Slow datasets. Here, Toflow is used for
denoising in the compared methods. The quantitative and qualitative results
are shown in Table III and Figure 7, respectively.

From Table III, we can see that our one-stage Zooming SlowMo (ZSM) achieves
the best performance among all compared approaches in terms of four evaluation
datasets. Our method outperforms the best three-stage method by $1.11$ dB on
Vid4, $2.41$ dB on Vimeo-Fast, $2.00$ dB on Vimeo-Medium, and $1.81$ dB on
Vimeo-Slow in terms of PSNR. This trend is more obvious in Visual results.
From Figure 7, we observe that all three methods can effectively remove severe
noises in input LR frames while the reconstructed HR frames by our model have
more visual details and fewer artifacts. The results demonstrate that our one-
stage network is capable of handling STVSR even for noisy inputs.

(a) Overlayed LR

(b) HR

(c) w/ DFI

(d) w/ DFI+ConvLSTM

(e) w/ DFI+DConvLSTM

Figure 9: Ablation study on Deformable ConvLSTM (DConvLSTM). Vanilla ConvLSTM
will fail on videos with fast motions. Embedded with state updating cells, the
proposed DConvLSTM is more capable of leveraging global temporal contexts for
reconstructing more accurate content, even for fast-motion videos.

(a) HR

(b) w/o bidirectional

(c) w/ bidirectional

Figure 10: Ablation study on the bidirectional mechanism in DConvLSTM. By
adding the bidirectional mechanism into DConvLSTM, our model can utilize both
previous and future contexts, and therefore can reconstruct more visually
appealing frames with finer details, especially for video frames at the first
step, which cannot access any temporal information from preceding frames.

(a) Overlayed LR

(b) HR

(c) w/o DFI@model (a)

(d) w/ DFI@model (b)

Figure 11: Ablation study on feature interpolation. The naive feature
interpolation model without deformable sampling will obtain overly smooth
results for videos with fast motions. With the proposed deformable feature
interpolation (DFI), our model can well exploit local contexts in adjacent
frames, thus is more effective in handling large motions.

####  4.3.2 JPEG Compression Artifact

For this setting, we train our model on the Vimeo-90K dataset with JPEG
compression of different quality factors (QF = 10, 20, 30, and 40). We compare
our model with two compression artifact reduction (CAR) methods: DNCNN [68]
and RNAN [72] +DAIN+RBPN on Vid4 [30] and three subsets of Vimeo90K’s testset.
The quantitative results and visual comparisons for each quality factor are
shown in Table IV and Figure 8, respectively.

It is shown in Table IV that our one-stage Zooming SlowMo (ZSM) outperforms
the existing three-stage methods for all QFs significantly, and yielding the
highest overall PNSR and SSIM across all testsets, especially on Vimeo-fast
that is with large motions: our method exceeds the second-best result by
$1.60$ dB for QF=10 test data, $1.54$ dB for QF=20, $1.61$ dB for QF=30 and
$1.76$ dB for QF=40. Figure 8 provides some qualitative examples. We can
assess the quality of the output images from the following perspectives:
compression artifact reductions (e.g. blocking and ringing artifacts, color
shift), and the reconstruction of details. According to Figure 8, even the
other methods can reduce the compression artifacts in most cases, they suffer
from over-smooth and lack of details. Compared with them, our output is with
fewer ringing and blocking artifacts and sharper edges under different QFs.
These results demonstrate that our one-stage framework can reach a better
balance between artifact reduction and high-fidelity reconstruction. It is
worth noting that our method is capable of restoring the color aberrations in
the images compressed by low quality factors (see the top 2 rows of Figure 8).

###  4.4 Ablation Study

In previous sections, we have already illustrated the superiority of our
proposed one-stage framework. In this section, we make a comprehensive
ablation study to further demonstrate the effectiveness of different modules
in our network.

TABLE V: Ablation study on the proposed modules. While ConvLSTM performs worse
for fast-motion videos, our deformable feature interpolation and deformable
ConvLSTM can effectively handle motions and improve overall STVSR performance.

Method | (a) | (b) | (c) | (d) | (e) | (f)
---|---|---|---|---|---|---
Naive feature interpolation | $\surd$ |  |  |  |  |
DFI |  | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$
ConvLSTM |  |  | $\surd$ |  |  |
DConvLSTM |  |  |  | $\surd$ |  |
Bidirectional DConvLSTM |  |  |  |  | $\surd$ | $\surd$
GFI |  |  |  |  |  | $\surd$
Vid4 (slow motion) | 25.18 | 25.34 | 25.68 | 26.18 | 26.31 | 26.49
Vimeo-Fast (fast motion) | 34.93 | 35.66 | 35.39 | 36.56 | 36.81 | 36.96

####  4.4.1 Effectiveness of Deformable ConvLSTM

To investigate the proposed Deformable ConvLSTM (DConvLSTM) module, we take
four different models for comparison: (b), (c), (d), and (e), where (c) adds a
vanilla ConvLSTM into (b), (d) adds the proposed DConvLSTM, and (e) utilizes
the DConvLSTM module in a bidirectional manner.

From Table V, we can see that (c) outperforms (b) on Vid4 with slow motion
while performs worse than (b) on Vimeo-Fast with fast motion videos. The
results indicate that vanilla ConvLSTM can utilize global temporal contexts
for slow motion videos, but cannot handle sequences with large motion.
Furthermore, we observe that model (d) is significantly better than both (b)
and (c), which demonstrates that our DConvLSTM can learn the temporal
alignment between previous states and the current feature map. Therefore, it
can better exploit global contexts for reconstructing visually appealing
frames with more vivid details. Our findings are supported by qualitative
results in Figure 9.

In addition, we verify the bidirectional mechanism in DConvLSTM by comparing
(e) and (d) in Table V and Figure 10. From Table V, we observe that (e) with
bidirectional DConvLSTM can further improve STVSR performance over (d) on both
slow motion and fast motion testing sets. The visual results in Figure 10 show
that our full model with a bidirectional mechanism can restore more details by
making better use of global temporal information for all input frames.

####  4.4.2 Effectiveness of Deformable Feature Interpolation

To validate our deformable feature interpolation (DFI) module, we introduce
two baselines for comparison: (a) and (b), where the model (a) only uses
convolutions to blend LR features instead of deformable sampling functions as
in model (b). Note that neither model (a) or (b) has ConvLSTM or DConvLSTM.

From Table V, we observe that (b) outperforms (a) by 0.16dB on Vid4 and 0.73
dB on Vimeo-Fast dataset with fast motions in terms of PSNR. Figure 11 shows a
qualitative comparison, where we can see that model (a) generates a face with
severe motion blur, while the proposed deformable feature interpolation module
can effectively address the large motion issue by exploiting local temporal
contexts and help the model (b) generate more clear face structures and
details. The superiority of the proposed DFI module demonstrates that the
learned offsets in the deformable sampling functions can better exploit local
temporal contexts and capture forward and backward motions in natural video
sequences, even without any explicit supervision.

####  4.4.3 Guided Feature Interpolation Critic

We first validate the strength of the guided feature interpolation (GFI)
learning on STVSR task by comparing the model trained with it (model f) to the
model without its supervision (model e) in Table V. Since the final
performance is evaluated on the overall video sequence, optimizing the
intermediate reward provided by LR frames can improve temporal consistency
across frames. Visual results illustrated in Figure 12 can further demonstrate
the superiority of the proposed GFI.

Furthermore, we demonstrate the influence of the guided feature interpolation
on noisy STVSR in Table VI. We compare PSNR and SSIM of the Y channel of
models trained with/without the guided loss on Vid4 dataset [30]. For fair
comparisons, both models are trained for the same number of iterations. We
observe that the intermediate features guide actually decreases the
performance in most cases when the input data is noisy.

(a) Overlayed LR

(b) w/o guidance

(c) w/ guidance

Figure 12: Ablation study on guided feature interpolation module. The
additional guidance can help to strengthen the ability of the temporal feature
interpolation network on handling motions.  TABLE VI: Ablation study of guided
feature learning on noisy STVSR. We compare the PSNR and SSIM of the Y channel
of models trained with/without the guided loss on Vid4 dataset [30].

Type | w/o GFI | w/GFI
---|---|---
PSNR | SSIM | PSNR | SSIM
Noise | 23.91 | 0.6514 | 23.89 | 0.6510
Compression | QF=10 | 22.03 | 0.5216 | 21.99 | 0.5204
QF=20 | 22.78 | 0.5696 | 22.78 | 0.5707
QF=30 | 23.25 | 0.6000 | 23.25 | 0.6013
QF=40 | 23.59 | 0.6226 | 23.55 | 0.6220

(a) DAIN+EDVR: $t-1$

(b) DAIN+EDVR: $t$

(c) DAIN+EDVR: $t+1$

(d) ZSM: $t-1$

(e) ZSM: $t$

(f) ZSM: $t+1$

Figure 13: Temporal inconsistent issue in STVSR. It is more difficult to
synthesis HR frame $t$ than HR frames: $t-1$ and $t+1$, since LR frames: $t-1$
and $t+1$ are available and LR frame $t$ is missing during testing.
Synthesized HR frame at the time step $t$ is more blurry with fewer visual
details than results at $t-1$ and $t+1$.

##  5 Discussion

For space-time video super-resolution, the goal is to reconstruct HR frames
for both missing intermediate and input LR frames. For example, given four
input LR frames: $\\{I_{1}^{L},I_{3}^{L},I_{5}^{L},I_{7}^{L}\\}$, we want to
obtain the corresponding seven consecutive HR frames:
$\\{I_{1}^{H},I_{2}^{H},...,I_{7}^{H}\\}$. Since $I_{2}^{L}$, $I_{4}^{L}$, and
$I_{6}^{L}$ are unavailable, and we need to hallucinate missing information
for the frames. Therefore, it is more challenging to reconstruct
$\\{I_{2}^{H},I_{4}^{H},I_{6}^{H}\\}$ than
$\\{I_{1}^{H},I_{3}^{H},I_{5}^{H},I_{7}^{H}\\}$. The imbalanced difficulty can
lead to temporal inconsistent video results, and the potential video jittering
becomes one of the most crucial issues that we need to consider when designing
a space-time video super-resolution method. In our current framework, the
proposed deformable ConvLSTM can implicitly enforce temporal coherence by
handling visual motions and aggregating temporal contexts. However, results
from our model still suffer from the temporal inconsistency issue due to the
essential imbalanced difficulty (see Figure 13). To further alleviate the
problem, we could consider devising new temporal consistency constraints to
explicitly guide HR video frame reconstruction.

Videos might contain dramatically changing scenes or objects due to the
existence of temporal motions and geometric deformations. Although our Zooming
SlowMo is capable of handling large-motion videos, it might fail for certain
dynamic objects with severe deformations. Multiple parts in a video object
cross temporal frames might have different motion patterns, which leads to
complex object deformations and introduces additional difficulties for space-
time video super-resolution. We illustrate one failure example in Figure 14.
Although the synthesized frame from our model contains fewer artifacts, the
reconstructed hands are pretty blurry due to large deformations in the
animated character. To mitigate the issue, we desire a more deformation-robust
temporal alignment approach. We believe it would be a promising direction.

(a) Overlayed LR

(b) DAIN+EDVR

(c) ZSM

Figure 14: Failure example. Our model might fail to handle dynamic video
objects with severe geometric deformations.

##  6 Conclusion

In this paper, we propose a one-stage framework that directly reconstructs
high-resolution and high frame rate videos without synthesizing intermediate
low-resolution frames for space-time video super-resolution. To achieve this,
we introduce a deformable feature interpolation network for feature-level
temporal interpolation. Furthermore, we propose a deformable ConvLSTM for
aggregating temporal information and handling motions. Our network can take
advantage of the intra-relatedness between temporal interpolation and spatial
super-resolution in the STVSR task with such a one-stage design. It enforces
our model to adaptively learn to leverage useful local and global temporal
contexts for alleviating large motion issues. Moreover, we introduce
intermediate guidance to strengthen the temporal feature interpolation.
Extensive experimental results show that our one-stage framework is more
effective yet efficient than existing two-stage methods; the proposed feature
temporal interpolation network and deformable ConvLSTM are capable of handling
very challenging fast motion videos; our network can successfully tackle more
challenging noisy STVSR tasks.

## References

  * [1] Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang.  Depth-aware video frame interpolation.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 3703–3712, 2019.
  * [2] Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang.  Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement.  IEEE Trans. Pattern Anal. Mach. Intell., 2019.
  * [3] Yuri Boykov, Olga Veksler, and Ramin Zabih.  Fast approximate energy minimization via graph cuts.  IEEE Trans. Pattern Anal. Mach. Intell., 23(11):1222–1239, 2001\.
  * [4] Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Real-time video super-resolution with spatio-temporal networks and motion compensation.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 4778–4787, 2017.
  * [5] Myungsub Choi, Janghoon Choi, Sungyong Baik, Tae Hyun Kim, and Kyoung Mu Lee.  Scene-adaptive video frame interpolation via meta-learning.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 9444–9453, 2020.
  * [6] Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee.  Channel attention is all you need for video frame interpolation.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10663–10671, 2020.
  * [7] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In IEEE Int. Conf. Comput. Vis., pages 764–773, 2017.
  * [8] Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang.  Compression artifacts reduction by a deep convolutional network.  In IEEE Int. Conf. Comput. Vis., pages 576–584, 2015.
  * [9] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Learning a deep convolutional network for image super-resolution.  In Proc. Eur. Conf. Comput. Vis., pages 184–199. Springer, 2014\.
  * [10] Esmaeil Faramarzi, Dinesh Rajan, and Marc P Christensen.  Space-time super-resolution from multiple-videos.  In International Conference on Information Science, Signal Processing and their Applications, pages 23–28. IEEE, 2012.
  * [11] Stuart Geman and Donald Geman.  Stochastic relaxation, gibbs distributions, and the bayesian restoration of images.  IEEE Trans. Pattern Anal. Mach. Intell., (6):721–741, 1984.
  * [12] Jun Guo and Hongyang Chao.  Building dual-domain representations for compression artifacts reduction.  In Proc. Eur. Conf. Comput. Vis., pages 628–644. Springer, 2016\.
  * [13] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita.  Recurrent back-projection network for video super-resolution.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 3897–3906, 2019.
  * [14] Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita.  Space-time-aware multi-resolution video enhancement.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recog., pages 2859–2868, 2020.
  * [15] Yan Huang, Wei Wang, and Liang Wang.  Video super-resolution via bidirectional recurrent convolutional networks.  IEEE Trans. Pattern Anal. Mach. Intell., 40(4):1015–1028, 2017\.
  * [16] Takashi Isobe, Xu Jia, Shuhang Gu, Songjiang Li, Shengjin Wang, and Qi Tian.  Video super-resolution with recurrent structure-detail network.  In Proc. Eur. Conf. Comput. Vis., pages 645–660. Springer, 2020\.
  * [17] Viren Jain and Sebastian Seung.  Natural image denoising with convolutional networks.  In Proc. Adv. Neural Inf. Process. Syst., pages 769–776, 2009.
  * [18] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz.  Super slomo: High quality estimation of multiple intermediate frames for video interpolation.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 9000–9008, 2018.
  * [19] Younghyun Jo, Seoung Wug Oh, Jaeyeon Kang, and Seon Joo Kim.  Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 3224–3232, 2018.
  * [20] Jaeyeon Kang, Younghyun Jo, Seoung Wug Oh, Peter Vajda, and Seon Joo Kim.  Deep space-time video upsampling networks.  In Proc. Eur. Conf. Comput. Vis. Springer, 2020.
  * [21] Soo Ye Kim, Jihyong Oh, and Munchurl Kim.  Fisr: Deep joint frame interpolation and super-resolution with a multi-scale temporal loss.  In AAAI, pages 11278–11286, 2020.
  * [22] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [23] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang.  Deep laplacian pyramid networks for fast and accurate super-resolution.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 624–632, 2017.
  * [24] Hyeongmin Lee, Taeoh Kim, Tae-young Chung, Daehyun Pak, Yuseok Ban, and Sangyoun Lee.  Adacof: Adaptive collaboration of flows for video frame interpolation.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 5316–5325, 2020.
  * [25] Tao Li, Xiaohai He, Qizhi Teng, Zhengyong Wang, and Chao Ren.  Space–time super-resolution with patch group cuts prior.  Signal Processing: Image Communication, 30:147–165, 2015.
  * [26] Wenbo Li, Xin Tao, Taian Guo, Lu Qi, Jiangbo Lu, and Jiaya Jia.  Mucan: Multi-correspondence aggregation network for video super-resolution.  arXiv preprint arXiv:2007.11803, 2020.
  * [27] Bee Lim and Kyoung Mu Lee.  Deep recurrent resnet for video super-resolution.  In Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1452–1455. IEEE, 2017.
  * [28] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog. Workshops, pages 136–144, 2017.
  * [29] Songnan Lin, Jiawei Zhang, Jinshan Pan, Zhe Jiang, Dongqing Zou, Yongtian Wang, Jing Chen, and Jimmy Ren.  Learning event-driven video deblurring and interpolation.  In Proc. Eur. Conf. Comput. Vis. Springer, 2020.
  * [30] Ce Liu and Deqing Sun.  A bayesian approach to adaptive video super resolution.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 209–216. IEEE, 2011.
  * [31] Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, and Yung-Yu Chuang.  Deep video frame interpolation using cyclic frame generation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8794–8802, 2019.
  * [32] Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala.  Video frame synthesis using deep voxel flow.  In IEEE Int. Conf. Comput. Vis., pages 4463–4471, 2017.
  * [33] Gucan Long, Laurent Kneip, Jose M Alvarez, Hongdong Li, Xiaohu Zhang, and Qifeng Yu.  Learning image matching by simply watching video.  In Proc. Eur. Conf. Comput. Vis., pages 434–450. Springer, 2016\.
  * [34] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [35] Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang.  Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections.  In Proc. Adv. Neural Inf. Process. Syst., pages 2802–2810, 2016\.
  * [36] Simone Meyer, Oliver Wang, Henning Zimmer, Max Grosse, and Alexander Sorkine-Hornung.  Phase-based frame interpolation for video.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 1410–1418, 2015.
  * [37] Uma Mudenagudi, Subhashis Banerjee, and Prem Kumar Kalra.  Space-time super-resolution using graph-cut optimization.  IEEE Trans. Pattern Anal. Mach. Intell., 33(5):995–1008, 2010.
  * [38] Simon Niklaus and Feng Liu.  Context-aware synthesis for video frame interpolation.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 1701–1710, 2018.
  * [39] Simon Niklaus and Feng Liu.  Softmax splatting for video frame interpolation.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 5437–5446, 2020.
  * [40] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive convolution.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 670–679, 2017.
  * [41] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive separable convolution.  In IEEE Int. Conf. Comput. Vis., pages 261–270, 2017.
  * [42] Liyuan Pan, Cedric Scheerlinck, Xin Yu, Richard Hartley, Miaomiao Liu, and Yuchao Dai.  Bringing a blurry frame alive at high frame-rate with an event camera.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 6820–6829, 2019.
  * [43] Junheum Park, Keunsoo Ko, Chul Lee, and Chang-Su Kim.  Bmbc: Bilateral motion estimation with bilateral cost volume for video interpolation.  arXiv preprint arXiv:2007.12622, 2020.
  * [44] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Proc. Adv. Neural Inf. Process. Syst. 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [45] Liad Pollak Zuckerman, Eyal Naor, George Pisha, Shai Bagon, and Michal Irani.  Across scales & across dimensions: Temporal super-resolution using deep internal learning.  In Proc. Eur. Conf. Comput. Vis. Springer, 2020.
  * [46] Fitsum A Reda, Deqing Sun, Aysegul Dundar, Mohammad Shoeybi, Guilin Liu, Kevin J Shih, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  Unsupervised video interpolation using cycle consistency.  In IEEE Int. Conf. Comput. Vis., pages 892–900, 2019.
  * [47] Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown.  Frame-recurrent video super-resolution.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 6626–6634, 2018.
  * [48] Mike Schuster and Kuldip K Paliwal.  Bidirectional recurrent neural networks.  IEEE Trans. Signal Process., 45(11):2673–2681, 1997.
  * [49] Oded Shahar, Alon Faktor, and Michal Irani.  Space-time super-resolution from a single video.  IEEE, 2011.
  * [50] Eli Shechtman, Yaron Caspi, and Michal Irani.  Increasing space-time resolution in video.  In Proc. Eur. Conf. Comput. Vis., pages 753–768. Springer, 2002\.
  * [51] Eli Shechtman, Yaron Caspi, and Michal Irani.  Space-time super-resolution.  IEEE Trans. Pattern Anal. Mach. Intell., (4):531–545, 2005.
  * [52] Wang Shen, Wenbo Bao, Guangtao Zhai, Li Chen, Xiongkuo Min, and Zhiyong Gao.  Blurry video frame interpolation.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 5114–5123, 2020.
  * [53] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 1874–1883, 2016.
  * [54] Shuochen Su, Mauricio Delbracio, Jue Wang, Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang.  Deep video deblurring for hand-held cameras.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 1279–1288, 2017.
  * [55] Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu.  Memnet: A persistent memory network for image restoration.  In IEEE Int. Conf. Comput. Vis., pages 4539–4547, 2017.
  * [56] Hiroyuki Takeda, Peter Van Beek, and Peyman Milanfar.  Spatiotemporal video upscaling using motion-assisted steering kernel (mask) regression.  In High-Quality Visual Experience, pages 245–274. Springer, 2010\.
  * [57] Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia.  Detail-revealing deep video super-resolution.  In IEEE Int. Conf. Comput. Vis., pages 4472–4480, 2017.
  * [58] Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu.  Tdan: Temporally-deformable alignment network for video super-resolution.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recog., pages 3360–3369, 2020.
  * [59] Longguang Wang, Yulan Guo, Zaiping Lin, Xinpu Deng, and Wei An.  Learning for video super-resolution through hr optical flow estimation.  In Asian Conference on Computer Vision, pages 514–529. Springer, 2018.
  * [60] Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy.  Edvr: Video restoration with enhanced deformable convolutional networks.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog. Workshops, pages 0–0, 2019.
  * [61] Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al.  Image quality assessment: from error visibility to structural similarity.  IEEE Trans. Image Process., 13(4):600–612, 2004.
  * [62] Zhangyang Wang, Ding Liu, Shiyu Chang, Qing Ling, Yingzhen Yang, and Thomas S Huang.  D3: Deep dual-domain based fast restoration of jpeg-compressed images.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 2764–2772, 2016.
  * [63] Xiaoyu Xiang, Qian Lin, and Jan P Allebach.  Boosting high-level vision with joint compression artifacts reduction and super-resolution.  In International Conference on Pattern Recognition. IEEE, 2020.
  * [64] Xiaoyu Xiang, Yapeng Tian, Yulun Zhang, Yun Fu, Jan P Allebach, and Chenliang Xu.  Zooming slow-mo: Fast and accurate one-stage space-time video super-resolution.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 3370–3379, 2020.
  * [65] Zeyu Xiao, Zhiwei Xiong, Xueyang Fu, Dong Liu, and Zheng-Jun Zha.  Space-time video super-resolution using temporal profiles.  In Proceedings of the ACM International Conference on Multimedia, pages 664–672, 2020.
  * [66] SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo.  Convolutional lstm network: A machine learning approach for precipitation nowcasting.  In Proc. Adv. Neural Inf. Process. Syst., pages 802–810, 2015.
  * [67] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman.  Video enhancement with task-oriented flow.  Int. J. Comput. Vis., 127(8):1106–1125, 2019.
  * [68] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.  Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.  IEEE Trans. Image Process., 26(7):3142–3155, 2017.
  * [69] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang.  Learning deep cnn denoiser prior for image restoration.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 3929–3938, 2017.
  * [70] Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Ffdnet: Toward a fast and flexible solution for cnn-based image denoising.  IEEE Trans. Image Process., 27(9):4608–4622, 2018.
  * [71] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In Proc. Eur. Conf. Comput. Vis., pages 286–301, 2018.
  * [72] Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu.  Residual non-local attention networks for image restoration.  arXiv preprint arXiv:1903.10082, 2019.
  * [73] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 9308–9316, 2019.

|  Xiaoyu Xiang received the B.E. degree in engineering physics from Tsinghua
University in 2015. She is currently pursuing a Ph.D. degree in the department
of electrical and computer engineering at Purdue University. Her primary area
of research has been image and video restoration.
---|---
|  Yapeng Tian received the B.E. degree in electronic engineering from Xidian
University, Xian, China, in 2013, M.E. degree in electronic engineering at
Tsinghua University, Beijing, China, in 2017, and is currently working toward
the Ph.D. degree in the department of computer science at the University of
Rochester, USA. His research interests include audio-visual scene
understanding and low-level vision.
---|---
|  Yulun Zhang received B.E. degree from School of Electronic Engineering,
Xidian University, China, in 2013 and M.E. degree from Department of
Automation, Tsinghua University, China, in 2017. He is currently pursuing a
Ph.D. degree with the Department of ECE, Northeastern University, USA. He was
the receipt of the Best Student Paper Award at the IEEE International
Conference on Visual Communication and Image Processing(VCIP) in 2015. He also
won the Best Paper Award at IEEE International Conference on Computer Vision
(ICCV) RLQ Workshop in 2019. His research interests include image restoration
and deep learning.
---|---
|  Yun Fu (S’07-M’08-SM’11-F’19) received the B.Eng. degree in information
engineering and the M.Eng. degree in pattern recognition and intelligence
systems from Xian Jiaotong University, China, respectively, and the M.S.
degree in statistics and the Ph.D. degree in electrical and computer
engineering from the University of Illinois at Urbana-Champaign, respectively.
He is an interdisciplinary faculty member affiliated with College of
Engineering and the Khoury College of Computer Sciences at Northeastern
University since 2012. His research interests are Machine Learning,
Computational Intelligence, Big Data Mining, Computer Vision, Pattern
Recognition, and Cyber-Physical Systems. He has extensive publications in
leading journals, books/book chapters and international conferences/workshops.
He serves as associate editor, chairs, PC member and reviewer of many top
journals and international conferences/workshops. He received seven
Prestigious Young Investigator Awards from NAE, ONR, ARO, IEEE, INNS, UIUC,
Grainger Foundation; eleven Best Paper Awards from IEEE, ACM, IAPR, SPIE,
SIAM; many major Industrial Research Awards from Google, Samsung, and Adobe,
etc. He is currently an Associate Editor of several journals, such as IEEE
Transactions on Neural Networks and Leaning Systems (TNNLS), IEEE Transactions
on Image Processing (TIP), and ACM Transactions on Knowledge Discovery from
Data (TKDD). He is fellow of IEEE, IAPR, OSA, and SPIE, a Lifetime
Distinguished Member of ACM, Lifetime Member of AAAI and Institute of
Mathematical Statistics, member of ACM Future of Computing Academy, Global
Young Academy, AAAS, INNS, and Beckman Graduate Fellow during 2007-2008.
---|---
|  Jan P. Allebach (LF’16) is currently a Hewlett-Packard Distinguished
Professor of electrical and computer engineering at Purdue University. He is a
fellow of the National Academy of Inventors, the Society for Imaging Science
and Technology (IS&T), and SPIE. He was named the Electronic Imaging Scientist
of the Year by IS&T and SPIE, and was named the Honorary Member of IS&T, the
highest award that IS&T bestows. He has received the IEEE Daniel E. Noble
Award, the IS&T/OSA Edwin Land Medal, and is a member of the National Academy
of Engineering. He has served as an IEEE Signal Processing Society
Distinguished Lecturer from 1994 to 1995, and again from 2016 to 2017.
---|---
|  Chenliang Xu is an Assistant Professor in the Department of Computer
Science at the University of Rochester. He received his Ph.D. degree from the
University of Michigan in 2016. His research interests range from solving core
computer vision problems to pushing the limits of their understanding in the
broader AI contexts, including multisensory perception, cognitive robotics,
and data science. He was the recipient of multiple NSF awards, including
BIGDATA and CDS&E, the University of Rochester’s AR/VR Pilot Award, Tencent
Rhino-Bird Award, a Best Paper Award at Sound and Music Computing Conference,
a Best Paper Award at ACM SIGGRAPH VRCAI, and an Open Source Code Award in
IEEE CVPR. He has authored more than 50 peer-reviewed papers in computer
vision and artificial intelligence venues.
---|---