# Learning Spatio-Temporal Downsampling for Effective Video Upscaling

Xiaoyu Xiang1, Yapeng Tian2, Vijay Rengarajan1, Lucas Young1, Bo Zhu1, Rakesh
Ranjan1
1Meta Reality Labs, 2University of Rochester
<EMAIL_ADDRESS><EMAIL_ADDRESS>This work is done during Yapeng’s internship at Meta.

###### Abstract

Downsampling is one of the most basic image processing operations. Improper
spatio-temporal downsampling applied on videos can cause aliasing issues such
as moiré patterns in space and the wagon-wheel effect in time. Consequently,
the inverse task of upscaling a low-resolution, low frame-rate video in space
and time becomes a challenging ill-posed problem due to information loss and
aliasing artifacts. In this paper, we aim to solve the space-time aliasing
problem by learning a spatio-temporal downsampler. Towards this goal, we
propose a neural network framework that jointly learns spatio-temporal
downsampling and upsampling. It enables the downsampler to retain the key
patterns of the original video and maximizes the reconstruction performance of
the upsampler. To make the downsamping results compatible with popular image
and video storage formats, the downsampling results are encoded to uint8 with
a differentiable quantization layer. To fully utilize the space-time
correspondences, we propose two novel modules for explicit temporal
propagation and space-time feature rearrangement. Experimental results show
that our proposed method significantly boosts the space-time reconstruction
quality by preserving spatial textures and motion patterns in both
downsampling and upscaling. Moreover, our framework enables a variety of
applications, including arbitrary video resampling, blurry frame
reconstruction, and efficient video storage.

##  1 Introduction

(a) $xyt$ volume

(b) GT image/temporal profile

(c) Bicubic+Nearest

(d) ZSM-Output [82]

(e) Ours-Downsample

(f) Ours-Upscale

Figure 1: Effect of learned downsampling and upscaling for space-time
reconstruction. Compared to previous methods, the outputs of our learned
downsampler maintain better spatial-temporal patterns, thus leading to more
visually-appealing reconstruction results with better space-time consistency.

Resizing is one of the most commonly used operations in digital image
processing. Due to the limit of available memory and transfer bandwidth in
compact devices, _e.g_. mobile phones and glasses, the high resolutions, high
frame rate videos captured by such devices trade off either spatial or
temporal resolution [2, 80]. While nearest-neighbor downsampling is the
standard operation to perform such reduction in space and time, it is not the
best option: it folds over high-frequency information in the downsampled
frequency domain, leading to aliasing as indicated by the Nyquist theorem
[59]. One way to avoid aliasing is to deliberately smudge high-frequency
information by allowing more space and time to capture a single sample, _i.e_.
by optical blur [26, 51] and motion blur [32, 63, 11], respectively. These
spatial and temporal anti-aliasing filters band-limit frequencies, making it
possible to reconstruct fine details during post-capture. Pre-designed anti-
aliasing filters can be employed with the downsampler during capture time; for
instance, using an optical low-pass filter [65] for spatial blur, and
computational cameras such as the flutter shutter [50] camera for temporal
blur.

To design the optimal filter for a specific task, it is natural to incorporate
the downstream performance in the loop, where the weights of the downsampling
filter are updated by the objective function of the task [67]. Kim  _et al_.
[28] inverts the super-resolution network as a downscaling encoder. Zou  _et
al_. [91] proposes to adaptively predict filter weights for each spatial
location.

For the video restoration task, a major benefit of pre-designing the
downsampler is to allow the co-design of an upsampler that recovers missing
high-frequency details. While traditional methods mainly focus on upsampling –
super-resolution (SR) in the case of space, and video frame interpolation
(VFI) in the case of time – they assume the downsampler to be a trivial
module. The LR images for SR tasks are usually acquired by bicubic
downsampling. For temporal reconstruction tasks like VFI, the low-fps (frames
per second) frames are acquired by nearest-neighbor sampling, which keeps one
frame per interval. Obviously, these operations are not optimal - the stride
in time will lead to temporal aliasing artifacts. This independent tackling of
the upsampling stage makes solving the inverse problem harder, and it is
typical to employ heavy priors on texture and motion, which results in
inaccurate hallucination of lost details. On the other hand, jointly handling
upsampling together with downsampling would enable better performance in
retaining and recovering spatio-temporal details. In this paper, we explore
the design of a joint framework through simultaneous learning of a downsampler
and an upsampler that effectively captures and reconstructs high-frequency
details in both space and time.

Based on the above observations, we propose a unified framework that jointly
learns spatio-temporal downsampling and upsampling, which works like an auto-
encoder for low-fps, low-resolution frames. To handle the ill-posed space-time
video super-resolution problem, we first make the downsampler to find the
optimal representation in the low-resolution, low-fps domain that maximizes
the restoration performance. Moreover, considering the downsampled
representation should be stored and transmitted in the common image and video
data formats, we quantize them to be uint8 with a differentiable quantization
layer that enables end-to-end training. Finally, the downsampled frames are
upscaled by our upsampler. To improve the reconstruction capability, we devise
space-time pixel-shuffle and deformable temporal propagation modules to better
exploit the space-time correspondences.

The main contributions of our paper are summarized as follows:

1\. We provide a new perspective for space-time video downsampling by learning
it jointly with upsampling, which preserves better space-time patterns and
boosts restoration performance.

2\. We observe that naive 3D convolution cannot achieve high reconstruction
performance, and hence, we propose the deformable temporal propagation and
space-time pixel-shuffle modules to realize a highly effective model design.

3\. Our proposed framework exhibits great potential to inspire the community.
We discuss the following applications: video resampling with arbitrary ratio,
blurry frame reconstruction, and efficient video storage.

##  2 Related Works

###  2.1 Video Downsampling

Spatial. Spatial downsampling is a long-standing research problem in image
processing. Classical approaches, such as the box, nearest, bicubic, and
Lasnczos [27, 52, 13], usually design image filters to generate low resolution
(LR) images by removing high frequencies and mitigating aliasing. Since most
visual details exist in the high frequencies, they are also removed during
downsampling. To address the problem, a series of structure and detail-
preserving image downscaling methods [31, 46, 78] are proposed. Although these
approaches can produce visually appealing LR images, they cannot guarantee
that upscaling methods can restore the original high resolution (HR) images
due to aliasing and non-uniform structure deformation in the downscaled LR
images. Pioneering research [72, 71] demonstrates that it is possible to
design filters that allow for reconstructing the high-resolution input image
with minimum error. The key is to add a small amount of optical blurring
before sampling. Inspired by this, we propose to automatically learn the best
blurring filters during downsampling for more effective visual upscaling.

Temporal. A simple way to downsample along the temporal dimension is to
increase the exposure time of a frame and capture the scene motion via blur.
The loss of texture due to averaging is traded off for the ability to embed
motion information in a single frame. Blur-to-video methods [23, 49, 22, 61,
88, 3] leverage motion blur and recover the image sequence, constraining the
optimization with spatial sharpness and temporal smoothness. To regularize the
loss of texture during capture, Yuan  _et al_. [86] use a long and short
exposure pair for deblurring, while Rengarajan  _et al_. [53] exploit the idea
to reconstruct high-speed videos. Coded exposure methods replace the box-
filter averaging over time with a broadband filter averaging by switching the
shutter on and off multiple times with varying on-off durations within a
single exposure period. This results in better reconstruction owing to the
preservation of high-frequency details over space. Raskar  _et al_. [50] use
such a coded exposure camera for deblurring, while Holloway  _et al_. [18]
recover a high-speed video from a coded low frame rate video. Our work
contributes an extension of this previous work by learning the optimal
temporal filters during downsampling for restoring sharp high-fps videos.

###  2.2 Video Upscaling

Video Super-Resolution. The goal of video super-resolution (VSR) is to restore
HR video frames from their LR counterparts. Due to the existence of visual
motion, the core problem to solve in VSR is how to temporally align
neighboring LR frames with the reference LR frame. Optical flow methods seek
to compute local pixel shifts and capture motions. Thus, a range of VSR
approaches [7, 68, 57, 74, 85, 16] use optical flow to estimate motion and
then perform motion compensation with warping. However, optical flow is
generally limited in handling large motions, and flow warping can introduce
artifacts into aligned frames. To avoid computing optical flow, implicit
temporal alignment approaches, such dynamic upsampling filters [24], recurrent
propagation [33, 19, 20], and deformable alignment [69, 75, 8, 9] are utilized
to handle complex motions.

Video Frame Interpolation. Video frame interpolation (VFI) aims to synthesize
intermediate video frames in between the original frames and upscale the
temporal resolution of videos. Meyer  _et al_. [38] utilizes phase information
to assist frame interpolation. [37, 25] proposed an encoder-decoder framework
to directly predict intermediate video frames. Niklaus  _et al_. [43, 44]
utilizes a spatially-adaptive convolution kernel for each pixel for
synthesizing missing frames. Similar to VSR, optical flow is also adopted in
VFI approaches [21, 36, 42, 5, 4, 64] to explicitly handle motions.

Space-Time Video Super-resolution The pioneering work to extend SR to both
space and time domains was proposed by Shechtman _et al_. [60]. Compared with
VSR and VFI tasks, STVSR is even more ill-posed since pixels are missing along
both spatial and temporal axes. To constrain the problem, early approaches
[60, 40, 66, 58] usually exploited space-time local smoothness priors. Very
recently, deep learning-based STVSR frameworks [29, 81, 17, 83, 14, 82, 84]
have been developed. These methods directly interpolate the missing frame
features and then upsample all frames in space. Like the video frame-
interpolation methods, the input frames are anchors of timestamps, which
limits the upscale ratio in the temporal dimension. Unlike these methods, we
aim to freely resize the space-time volume with arbitrary scale ratios in this
work.

##  3 Space-Time Anti-Aliasing (STAA)

We first explain the intuition behind STAA: treating video as spatio-temporal
$xyt$ volume and leveraging the characteristic spatio-temporal patterns for
video reconstruction. Towards this end, we propose efficiently utilizing the
spatio-temporal patterns in upscaling with space-time pixel-shuffle. However,
naively regarding time as an additional dimension beyond space has a
limitation: the pixel $(x,y)$ at the $i$-th frame is usually not related to
the same $(x,y)$ at the $i+k$-th frame. To tackle this problem, we model the
temporal correspondences with deformable sampling.

###  3.1 Intuition

To tackle the aliasing problem, previous methods insert low-pass filters
either in space or time. However, we noticed that the space and time
dimensions of a $xyt$ volume are not independent: as shown in Fig. 1, the
temporal profiles $xt$ and $yt$ display similar patterns as the spatial
patches. In an ideal case where the 2D objects move with a constant velocity,
the temporal profile will appear as a downsampled version of the object [58,
48], as illustrated in Fig. 5. This space-time patch recurrence makes it
possible to aid the reconstruction of the under-sampled dimension with
abundant information from other dimensions. Based upon this observation, we
adopt a 3D low-pass filter on both space and time to better utilize the
correspondences across dimensions. Accordingly, our upsampler network also
adopts the 3D convolutional layers as the basic building block due to its
capability of jointly handling the spatio-temporal features.

###  3.2 Module Design

Space-Time Pixel-Shuffle. Pixel-shuffle [62] is a widely-used operation in
single image super-resolution (SISR) for efficient subpixel convolution. It
has two advantages: the learned upscaling filter can achieve the optimal
performance; computational complexity is reduced by rearranging the elements
in LR feature maps. Inspired by its success in SISR, we extend it to space-
time. Fig. 2 illustrates the shuffling operation: for a input tensor with
shape $(r\cdot s^{2}\cdot C,N,H,W)$, the elements are shuffled periodically
into the shape $(C,rN,sH,sW)$.

Figure 2: Space-Time Pixel-Shuffle.

Naive Deconvolution Is Insufficient. If we simply regard the space-time
upscaling as the reverse process of the downsampling, then conceptually, a
deconvolution should be enough to handle this process. To investigate this
idea, we build a small network using 3D convolutions and the aforementioned
space-time pixel-shuffle layers with a style of ESPCN [62]. Although this
network does converge, it only improves the PSNR by $\sim 0.5$ dB compared
with trilinear upscaling the $xyt$ cube – such improvement is too trivial to
be considered effective, particularly when compared to the success of ESPCN in
SISR.

Enhance Temporal Modeling Capacity. As noted above, this suboptimal result was
expected due to the lost correspondences between the $i$-th and $i+k$-th
frames. Thus, 3D convolution alone cannot guarantee a good reconstruction
performance due to its relatively small field of view. Understanding “what
went where” [79] is the fundamental problem for video tasks. Such
correspondence is even more critical in our framework: our STAA downsampler
encodes the motion by dispersing the space feature along the temporal
dimension. Correspondingly, during the reconstruction stage, the supporting
information can come from neighboring frames. Motivated by this, we devise a
deformable module to build temporal correspondences and enhance the model’s
capability to handle dynamic scenes: for a frame at time $i$, it should look
at adjoining $i-k,\ldots i+k$ frames and refine the current feature by
aggregating the relevance. For efficient implementation, we split the
information propagation into forward and backward directions, where the
temporal correspondence is built and passed recurrently per direction, as
shown in Fig. 3. The refined features from both forward and backward passes
are aggregated to yield the output. Hence, the difficulty of perceiving long-
range information within the 3D convolutional receptive field is alleviated.

Figure 3: Temporal propagation module.

##  4 Joint Downscaling and Upscaling Framework

Our framework architecture is shown in Fig. 4: given a sequence of video
frames $V=\\{I_{i}\\}_{1}^{rN}$ where each $I_{i}$ is an RGB image of
dimensions $sW\times sH$ ($r$ and $s$ as scale factors), our goal is to design
(a) a downsampler which would produce $V_{\downarrow}=\\{D(I_{i})\\}_{1}^{N}$
where each $D(I_{i})$ has the dimensions $W\times H$, and, (b) an upsampler
which would produce $\widetilde{V}=\\{U(D(I_{i}))\\}_{1}^{rN}$ where, in the
perfect case, $V=\widetilde{V}$.

Figure 4: Our training framework functions as an auto-encoder in which we
train the downsampler $D$ (encoder), and the upsampler $U$ (decoder) jointly
in an end-to-end manner.

###  4.1 Downsampler

Our downsampler consists of a 3D low pass filter $h(\cdot)$ followed by a
downsampling operation by striding. Given a sequence of input frames
$V=\\{I_{i}\\}_{1}^{rN}$ that needs to be downsampled, we first convolve it
with the filter $h$ as follows:

$h(V)[t,x,y]=\sum_{i,j,k\in\Omega}h[i,j,k]\cdot V[t-i,x-j,y-k].$ (1)

To ensure that the learned filters are low-pass, we add a softmax layer to
regularize the weight values within the range $[0,1]$ and the sum to be 1. We
then use striding to produce our desired downsampled frames in both space and
time. An ideal anti-aliasing filter should restrict the bandwidth to satisfy
the Nyquist theorem without distorting the in-band frequencies.

Analysis of learned filters. We present a study based on the frequency domain
analysis of spatio-temporal images to compare different types of low-pass
filters. We analyze the canonical case of a single object moving with uniform
velocity. The basic setup is shown in the first column of Fig. 5, where the
top row shows the static object and the bottom row shows the $xyt$ volume
corresponding to the motion. Figs. 5(a) to (f) show the temporal profiles $xt$
corresponding to the 1D scanline (marked in red) for various scenarios/filters
in the top row and their corresponding Fourier domain plots in the bottom row.
Please check the supplementary material for more details about how the Fourier
plots are calculated and what the spectra components represent.

In Fig. 5, (a) is a space-time diagram for a static scene (zero velocity), so
there is no change along the time (vertical) dimension. In (b), we can see
that the motion causes a time-varying effect, which results in shear along the
spatial $x$ direction. This shows the coupling of spatial and temporal
dimensions. Applying just the nearest-neighbor downsampling in time leads to
severe aliasing, as shown by duplication of streaks in (c) bottom row. Thus,
the plain downsampling method leads to temporal pattern distortions.
Characteristic spatio-temporal patterns relate to events in the video [45, 87,
10]. Thus, good downsampling methods should also retain the “textures” in time
dimension.

Figs. 5(d), (e), and (f) show the images and frequency plots for the case of
applying Gaussian, box, and our STAA low pass filters, respectively, first,
followed by nearest-neighbor downsampling. The convolution with these filters
causes blurring across space and time dimensions, as shown in the images, and
since convolution corresponds to frequency domain multiplication, we can see
the benefits of these filters visually in the frequency domain plots. The
Gaussian filter reduces the subbands and high spatiotemporal frequencies in
(d). The Box filter along the temporal dimension, which is used to describe
the motion blur [15, 6, 55], destroys spatial details and attenuates certain
temporal frequencies, which causes post-aliasing [39] during reconstruction.
Our proposed STAA filter attenuates high frequencies and the subbands like the
Gaussian filter does, but at the same time, it preserves more energy in the
main spectra component, as shown in (f). This characteristic ensures the
prefiltered image maintains a good spatio-temporal texture, thus benefiting
the reconstruction process.

Figure 5: Space-time and Fourier domain plots for a moving object.

Connection to coded exposure. Traditional motion blur caused by long exposure
can be regarded as filtering with a temporal box filter. While long exposure
acts as a natural form of filtering during the capture time itself, the box
filter is not the best filter for alias-free reconstruction [39]. Hence, coded
exposure methods “flutter” the shutter in a designed sequence to record the
motion without completely smearing the object across the spatial dimension to
recover sharp high-frame-rate images and videos [50, 18]. Our learned
downsampler can be regarded as a learned form of the coded exposure:
considering the temporal kernel size as an exposure window, we aggregate the
pixels at each time step to preserve an optimal space-time pattern for better
reconstruction.

Differentiable quantization layer. The direct output of our downsampler is a
floating-point tensor, while in practical applications, images are usually
encoded as 8-bit RGB (uint8) format. Quantization is needed to make our
downsampled frames compatible with popular image storage and transmission
pipelines. However, this operation is not differentiable. The gap between
float and discrete integer causes training unstable and a drop in performance.
To bridge the gap, we adopt a differentiable quantization layer that enables
end-to-end training. More details can be found in supplementary material.

###  4.2 Upsampler

Given a sequence of downsampled frames,
$V_{\downarrow}=\\{D(I_{i})\\}^{N}_{i=1}$, the upsampler $U$ aims to increase
the resolution in both space and time. The estimated upscaled video
$\widetilde{V}=\\{U(D(I_{i})\\}_{i=1}^{rN}$ should be as close to the original
input as possible.

To achieve this purpose, we choose 3D convolution as our basic building block
for the upsampler. The input sequence is converted to the feature domain
$\mathcal{F}$ by a 3D convolution. We adopt an deformable temporal modeling
(DTM) subnetwork to aggregate the long-range dependencies recurrently. It
takes the last aggregated frame feature $DTM(f_{i-1})$ at time step $i-1$ and
the current feature $f_{i}$ as inputs, outputting the current aggregated
feature:

$DTM(f_{i})=T(f_{i},DTM(f_{i-1})),$ (2)

where $f_{i}$ is the frame feature at time step $i$, and $T$ denotes a general
function that finds and aligns the corresponding information to the current
feature. We adopt the deformable sampling function [12, 90] as $T$ to capture
such correspondences. To fully exploit the temporal information, we implement
the DTM in a bidirectional manner that aggregates the refined features from
both forward and backward passes.

The refined sequence is then passed to the reconstruction module that is
composed of 3D convolutions. To fully explore the hierarchical features from
these convolutional layers, we organize them into residual dense blocks [89].
It densely connects the 3D convolution layers into local groups and fuses the
features of different layers. Following the previous super-resolution networks
[34], no BatchNorm layer is used in our reconstruction module. Finally, a
space-time pixel-shuffle layer is adopted to rearrange the features with a
periodic shuffling across the $xyt$ volume [56].

We denote the output just after the space-time pixel-shuffle as
$F(V_{\downarrow})$, where $F(\cdot)$ is all the previous operations for
upscaling the input $V_{\downarrow}$. To help the main network focus on
generating high-frequency information, we bilinearly upscale the input
sequence and add it to the reconstructed features as the final output:

$U(V_{\downarrow})=V_{\downarrow}\uparrow_{M}+F(V_{\downarrow}).$ (3)

This long-range skip-connection allows the low-frequency information of the
input to bypass the major network and makes the major part of the network
predict the residue. It “lower-bounds” the reconstruction performance and
increases the convergence speed of the network.

##  5 Experiments

First, we compare the proposed STAA framework to the state-of-the-art methods
for upscaling videos in space (VSR) and time(VFI) to show the benefit of the
jointly learned downsampling and upscaling. We also conduct ablation studies
on the downsampler and upsampler designs. Finally, we discuss several
potential applications of our method.

Datasets. We adopt the Vimeo-90k dataset [85] to train the proposed framework.
It consists of more than 60,000 training video sequences, and each video
sequence has seven frames. These frames are used as both the inputs and
outputs of the auto-encoder. Consistent with previous works, we evaluate our
method on the Vid4[35] dataset and the test set of Vimeo-90k to compare with
the state-of-the-art VFI, VSR, and STVSR methods.

Implementation Details. In our proposed STAA framework, we adopt a filter of
shape $3\times 3\times 3$ for the downsampler. In the upsampler, one 3D
convolutional layer is used to convert the input frame sequence to the feature
domain. We adopt the deformable ConvLSTM [81, 82] for temporal propagation.
Five 3D residual-dense blocks are stacked to build the reconstruction module.
During the training phase, we augment the training frames by randomly flipping
horizontally and $90^{\circ}$ rotations. The training patch size is $128\times
128$, and the batch size is set to be 32. We train the network for 100 epochs
using the Adam [30] optimizer. The initial learning rate is set to $2\times
10^{-4}$ which scales down by a factor of $0.2$ each at epochs, 50 and 80. Our
network is implemented in PyTorch [47]. Our STAA framework is trained end-to-
end with the $L$1-loss: $L(V,\widetilde{V})=||V-\widetilde{V}||_{1}$.

Evaluation. We use the Peak Signal-to-Noise-Ratio (PSNR) and Structural
Similarity Index (SSIM) [77] metrics to evaluate video restoration. We also
compare the number of parameters (million, M) to evaluate model efficiency.

###  5.1 Comparison with State-of-the-Art Methods

We compare the performance of reconstructing a video in space and time with
the state-of-the-art methods in a cascaded manner: XVFI [64] and FLAVR [25]
for video frame-interpolation, and BasicVSR++ [9] for video super-resolution.
We also compare with the one-stage ZSM [81, 82] for space-time video super-
resolution. Note that the previous methods cannot upsample by $2\times$ in
time: for two input frames, they generate one interpolated frame along with
the two inputs, while our STAA generates four upsampled ones. For an apples-
to-apples comparison, we only calculate the PSNR/SSIM of the synthesized
frames. Quantitative results on Vimeo-90k [85] and Vid4 [35] are shown in
Table 1.

Our method outperforms the previous methods by a large margin on all datasets
and settings. For temporal upscaling, adopting the STAA downsampling and
upscaling exceeds the second-best method by 8.28 dB on Vimeo-90k and 9.95 dB
on Vid4, which validates the importance of anti-aliasing in the temporal
dimension. For the challenging case of $4\times$ space/$2\times$ time, our
method still demonstrates remarkable improvement by 2.4 dB on the Vid4 and
more than 1 dB on the Vimeo-90k datasets. Such significant improvement brought
by the co-design of downsampling filter and upscaling network provides a new
possibility for improving current video restoration methods.

Table 1: Comparisons with SOTA cascaded video frame interpolation (VFI) and
super-resolution (VSR), and space-time super-resolution (STVSR) methods.

Upscale rate | Downsampler | Reconstruction Method | Params/M | Vimeo-90k | Vid4
---|---|---|---|---|---
time/space | time/space | VFI | VSR | PSNR | SSIM | PSNR | SSIM
2$\times$/1$\times$ | Nearest/- | XVFI [64] | - | 5.7 | 34.76 | 0.9532 | 29.21 | 0.9496
FLAVR [25] | - | 42.1 | 36.73 | 0.9632 | 29.83 | 0.9585
STAA | Ours | 15.9 | 45.01 | 0.9912 | 39.78 | 0.9926
2$\times$/4$\times$ | Nearest/Bicubic | XVFI [64] | BasicVSR++ [9] | 5.7+7.3 | 32.41 | 0.9123 | 24.90 | 0.7726
FLAVR [25] | BasicVSR++ [9] | 42.1+7.3 | 32.74 | 0.9119 | 24.79 | 0.7678
ZSM [81] | 11.1 | 33.48 | 0.9178 | 24.82 | 0.7763
| STDAN [73] | 8.3 | 33.59 | 0.9192 | 24.91 | 0.7832
STAA | Ours | 16.0 | 34.53 | 0.9426 | 27.31 | 0.9173

###  5.2 Ablation Studies

Type of downsampling filter. To verify the effectiveness of our learned
downsampling filter, we compare the reconstruction performance by switching
downsampler types. The idea is: since the reconstruction capability of the
same upsampler architecture is unchanged, a better reconstruction result means
that the downsampler produces a better space-time representation. For an
apples-to-apples comparison, we use the same kernel size for Gaussian and our
STAA filters. We experiment different combinations of time/space sampling
rates: $1\times t,2\times s$, $2\times t,1\times s$, and $2\times t,2\times
s$.

As shown in Table 2, the nearest-bicubic downsampling, which is adopted by
previous video reconstruction tasks, provides the worst representation among
all. For the $2\times t/2\times s$ setting, the reconstruction network cannot
converge to global optimal. Although it is still the dominant setting, it
cannot handle the temporal-aliasing issue and might hinder the development of
video reconstruction methods. Pre-filtering with a 3D Gaussian blur kernel can
alleviate the aliasing problem, which exceeds the nearest-neighbor
downsampling in time and the bicubic downsampling in space. Still, the
Gaussian filter cannot produce the optimal spatio-temporal textures. Compared
with these classical methods, our STAA filters improve the reconstruction
performance by a large margin, as shown in the last four rows of each setting.
We believe that our proposed STAA downsampler has the potential to serve as a
new benchmark for video reconstruction method design and inspire the community
from multiple perspectives.

We visualize the downsampled frames and their corresponding reconstruction
results in Fig. 6. The nearest-bicubic downsampled result looks nice per
frame; however, the temporal profile has severe aliasing. In comparison, the
anti-aliasing filters make the downsampled frames “blurry” to embed the motion
information.

Table 2: Quantitative comparison of downsampling filters. We compare the
reconstruction results of the classical nearest/bicubic sampling and Gaussian
filter to our learned STAA filters with different constraints. The best two
results are highlighted in red and blue, respectively.

Downsampling rate | Time | Space | PSNR | SSIM
---|---|---|---|---
1$\times t$/2$\times s$ | - | Bicubic | 41.61 | 0.9816
| Gaussian | 41.69 | 0.9827
| STAAno | 43.47 | 0.9883
| STAAsoft | 44.19 | 0.9901
| STAAquant | 44.19 | 0.9899
| STAAada | 42.05 | 0.9840
2$\times t$/1$\times s$ | Nearest | - | 38.22 | 0.9715
| Gaussian | 39.26 | 0.9796
| STAAno | 46.50 | 0.9942
| STAAsoft | 46.59 | 0.9942
| STAAquant | 46.75 | 0.9944
| STAAada | 45.32 | 0.9929
2$\times t$/2$\times s$ | Nearest | Bicubic | 28.88 | 0.9073
| Gaussian | 37.44 | 0.9679
| STAAno | 39.44 | 0.9775
| STAAsoft | 40.40 | 0.9812
| STAAquant | 40.42 | 0.9811
| STAAada | 38.13 | 0.9720

Figure 6: Visual comparison at $2\times t/2\times s$ setting.

Constraints of the encoded frames. The classical downsampling filters, _e.g_.,
nearest and Gaussian, can naturally guarantee that the downsampled frames
resemble the input’s appearance. However, in our auto-encoder framework, there
is actually no guarantee that the encoded frames look like the original input.
A straightforward way is to use the downsampled results from these classical
methods as supervision, but it might impede the downsampler from learning the
optimal spatio-temporal representation. So we turn to control the encoded
frames implicitly by constraining the downsampling filter. We experiment three
types of learned filters: (1) no: no constraints added; (2) soft: use the
softmax to regularize the weight values; (3) quant: add the differentiable
quantization layer; (4)ada: dynamically generate filters for each spatial
location according to the input content (also with softmax).

From the last four rows of Table 2, all STAA filters outperform the classical
ones for reconstruction. The filter without any constraint is not necessary to
be low-pass. Besides, it may cause color shifts in the encoded frames.
Constraining the filter weights with softmax can alleviate color shifts and
improve the reconstruction results due to anti-aliasing. Still, the moving
regions are encoded as the color difference. Adding the quantization layer
does not cause performance degradation, which validates the effectiveness of
our differentiable implementation. Making the filter weights conditioned on
the input content creates visually pleasing LR frames. However, the
reconstruction performance degrades, probably because the changing weights of
the downsampling filter confuse the upsampler.

Comparing different downsampling settings, we observe that our STAA is more
robust to temporal downsampling than previous methods. Specifically, the
reconstruction quality is correlated to the logarithm of the percentage of
pixels in the downsampling representation. More discussions are in our
Appendix.

Effectiveness of proposed modules. In Table 3, we compare the video
reconstruction results and the computational cost with different modules of
the upsampler. We check the FLOPs per million pixels (MP) using the open-
source tool fvcore [54]. From the first row, we can observe that naive 3D
convolution performs bad. Changing it to a more complex 3D residual-dense
block (RDB) improves the performance by 2.68 dB, with a rapid increase of the
computational cost. Although this network still cannot explicitly find the
temporal correspondence, the deeper structure enlarges the perceiving area,
thus enabling capturing dependencies with large displacement. In the third
row, adopting deformable temporal modeling (DTM) shows a great performance
improvement with relatively low computational cost, which validates the
importance of aggregating the displaced information across space and time.
Such spatio-temporal aggregated features can be effectively utilized by the 3D
CNN, resulting in improved PSNR and SSIM results (see the last row).

Table 3: Ablation of modules in the upsampler.

Naive 3D Conv | 3D RDB | DTM | Params/M | GFlops/MP | PSNR | SSIM
---|---|---|---|---|---|---
✓ |  |  | 0.3 | 3.2 | 28.67 | 0.8536
| ✓ |  | 11.0 | 114.4 | 31.35 | 0.9016
|  | ✓ | 5.3 | 51.7 | 31.55 | 0.9032
| ✓ | ✓ | 16.0 | 164.7 | 32.00 | 0.9109

###  5.3 Applications

The STAA downsampler can be used to reduce the resolution and frame rate of a
video for efficient video transmission. The learned upsampler can also be
applied to process natural videos with a simple modification. Besides, as
discussed in Section 4.1, motion blur is a natural space-time low-pass filter.
So our upsampler network can also reconstruct crisp clean frames from a blurry
sequence. The training details of each application are in the supplementary
material.

Video Resampling. The space-time pixel-shuffle module makes it possible to
change the frame rate with arbitrary ratios while keeping the motion patterns.
Video frame interpolation methods can only synthesize new frames in-between
the input frames, so the scale ratio can only be integers. Tools like ffmpeg
[70] change frame rate by dropping or duplication at a certain interval, which
changes the motion pattern of the original timestamps and cannot generate
smooth results. Another option is to use frame blending to map the
intermediate motion between keyframes while creating fuzzy and ghosting
artifacts. Some softwares adopt optical flow warping, which can synthesize
better results than the above two methods. Still, it cannot handle large
motions or morph.

Our upsampler can maintain the space-time patterns when upscaling the temporal
dimension at any given ratio: we show an example of converting 20 fps to 24
fps (1.2$\times t$) in Fig. 7, which does synthesize the correct motion at the
non-existent time steps. The well-placed motions lead to smoother visual
results. Our temporal modeling module can map long-range dependencies among
the input frames, and together with the space-time convolutional layers, they
can reconstruct sharp and crisp frames. To the best of our knowledge, this is
the first deep-learning-based technique for arbitrary frame rate conversion.

Figure 7: Our methods enable smooth frame rate conversion with arbitrary
rates, _e.g_. 20 fps to 24 fps, which needs convert 5 frames into 6 frames
with the same time length. The top row shows the input frames at 20 fps, and
the bottom row shows the output sequence at 24 fps. We plot a timeline in the
middle and mark the timestamp of each frame. The generated frames appear
natural in motion transition, and have vivid textures, _e.g_. water flow,
reflection, and refraction.

(a) LR blurry input

(b) Output: 0

(c) Output: 1

(d) Output: 2

(e) Output: 3

Figure 8: Our upsampler can also be used for reconstructing sharp and crisp
details from videos with motion-blur. The left column shows the overlaid two
LR blurry inputs, and the right four columns are our reconstruction results
with $2\times$ in time and $4\times$ in space, which recover shapes and
textures from motion blur.

Blurry Frame Reconstruction. As discussed in Sec. 4.1, motion blur is a
temporal low-pass filter. It is a real-world case of our STAA filter: the
temporal kernel size is the exposure time window, and the weights at each time
step are equal. Hence, there is a good reason to believe that our designed
upsampler can be applied on blurry frame reconstruction, which turns the low-
resolution blurry sequence into a high frame-rate and high-resolution clean
sequence. We trained our upsampler with a $4\times s,2\times t$ upscale
setting using the REDS-blur [41] data. We show the restoration images in Fig.
8. Even when the motion is rather large and the object texture is badly
smeared, our upsampler does a good job in reconstructing the shape and
structures at the correct timestep.

Efficient Video Storage and Transmission. Since the downsampler output is
still in the same color space and data type (_e.g_. 8-bit RGB) as the input,
it can be processed by any existing encoding and compression algorithms for
storage and transmission without extra elaborations. Especially, the
downsampled frames still preserve the temporal connections implying their
compatibility with video codecs.

##  6 Conclusions

In this paper, we propose to learn a space-time downsampler and upsampler
jointly to optimize the coding of the intermediate downsampled representations
and ultimately boost video reconstruction performance. The downsampler
includes a learned 3D low-pass filter for spatio-temporal anti-aliasing and a
differentiable quantization layer to ensure the downsampled frames are encoded
as standard 8-bit RGB images. For the upsampler, we propose the space-time
pixel-shuffle to enable upscaling the $xyt$ volume at any given ratio. We
further exploit the temporal correspondences between consecutive frames by
explicit temporal modeling. Due to the advantages of these designs, our
framework outperforms state-of-the-art works in VSR and VFI by a large margin.
Moreover, we demonstrate that our proposed upsampler can be used for highly
accurate arbitrary frame-rate conversion, generating high-fidelity motion and
visual details at the new timestamps for the first time. Our network can also
be applied to blurry frame reconstruction and efficient video storage. We
believe that our approach provides a new perspective on space-time video
super-resolution tasks and has a broad potential to inspire novel methods for
future works _e.g_. quantization-aware image/video reconstruction,
restoration-oriented video compression, and hardware applications such as
coded exposure and optical anti-aliasing filter.

## Acknowledgements

We thank Hai Wang for the quantitative results of previous SOTA methods, Salma
Abdel Magid for feedback on the manuscript, Chakravarty Reddy Alla Chaitanya
and Siddharth Bhargav for discussions.

## References

  * [1] Adobe Inc.  Adobe premiere pro.
  * [2] Jan Allebach and Ping Wah Wong.  Edge-directed interpolation.  In IEEE International Conference on Image Processing, volume 3, pages 707–710. IEEE, 1996.
  * [3] Dawit Mureja Argaw, Junsik Kim, Francois Rameau, and In So Kweon.  Motion-blurred video interpolation and extrapolation.  In AAAI Conference on Artificial Intelligence, 2021.
  * [4] Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang.  Depth-aware video frame interpolation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3703–3712, 2019.
  * [5] Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang.  Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [6] Tim Brooks and Jonathan T Barron.  Learning to synthesize motion blur.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 6840–6848, 2019.
  * [7] Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Real-time video super-resolution with spatio-temporal networks and motion compensation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 4778–4787, 2017.
  * [8] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Understanding deformable alignment in video super-resolution.  arXiv preprint arXiv:2009.07265, 4(3):4, 2020.
  * [9] Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy.  Basicvsr++: Improving video super-resolution with enhanced propagation and alignment.  arXiv preprint arXiv:2104.13371, 2021.
  * [10] Matthew Cooper, Ting Liu, and Eleanor Rieffel.  Video segmentation via temporal pattern classification.  IEEE Transactions on Multimedia, 9(3):610–618, 2007.
  * [11] Frank Dachille and Arie Kaufman.  High-degree temporal antialiasing.  In Proceedings Computer Animation, pages 49–54. IEEE, 2000.
  * [12] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In IEEE International Conference on Computer Vision, pages 764–773, 2017.
  * [13] Claude E Duchon.  Lanczos filtering in one and two dimensions.  Journal of Applied Meteorology and Climatology, 18(8):1016–1022, 1979.
  * [14] Saikat Dutta, Nisarg A Shah, and Anurag Mittal.  Efficient space-time video super resolution using low-resolution flow and mask upsampling.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 314–323, 2021.
  * [15] Kevin Egan, Yu-Ting Tseng, Nicolas Holzschuch, Frédo Durand, and Ravi Ramamoorthi.  Frequency analysis and sheared reconstruction for rendering motion blur.  ACM Transactions on Graphics, 28(3):93–1, 2009.
  * [16] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita.  Recurrent back-projection network for video super-resolution.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3897–3906, 2019.
  * [17] Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita.  Space-time-aware multi-resolution video enhancement.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 2859–2868, 2020.
  * [18] Jason Holloway, Aswin C Sankaranarayanan, Ashok Veeraraghavan, and Salil Tambe.  Flutter shutter video camera for compressive sensing of videos.  In International Conference on Computational Photography, pages 1–9. IEEE, 2012.
  * [19] Yan Huang, Wei Wang, and Liang Wang.  Video super-resolution via bidirectional recurrent convolutional networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):1015–1028, 2017.
  * [20] Takashi Isobe, Xu Jia, Shuhang Gu, Songjiang Li, Shengjin Wang, and Qi Tian.  Video super-resolution with recurrent structure-detail network.  In European Conference on Computer Vision, pages 645–660. Springer, 2020.
  * [21] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz.  Super slomo: High quality estimation of multiple intermediate frames for video interpolation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 9000–9008, 2018.
  * [22] Meiguang Jin, Zhe Hu, and Paolo Favaro.  Learning to extract flawless slow motion from blurry videos.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 8112–8121, 2019.
  * [23] Meiguang Jin, Givi Meishvili, and Paolo Favaro.  Learning to extract a video sequence from a single motion-blurred image.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 6334–6342, 2018.
  * [24] Younghyun Jo, Seoung Wug Oh, Jaeyeon Kang, and Seon Joo Kim.  Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3224–3232, 2018.
  * [25] Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Du Tran.  Flavr: Flow-agnostic video representations for fast frame interpolation.  arXiv preprint arXiv:2012.08512, 2020.
  * [26] Brian Keelan.  Handbook of image quality: characterization and prediction.  CRC Press, 2002.
  * [27] Robert Keys.  Cubic convolution interpolation for digital image processing.  IEEE Transactions on Acoustics, Speech, and Signal Processing, 29(6):1153–1160, 1981.
  * [28] Heewon Kim, Myungsub Choi, Bee Lim, and Kyoung Mu Lee.  Task-aware image downscaling.  In European Conference on Computer Vision, pages 399–414, 2018\.
  * [29] Soo Ye Kim, Jihyong Oh, and Munchurl Kim.  Fisr: Deep joint frame interpolation and super-resolution with a multi-scale temporal loss.  In AAAI Conference on Artificial Intelligence, pages 11278–11286, 2020.
  * [30] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [31] Johannes Kopf, Ariel Shamir, and Pieter Peers.  Content-adaptive image downscaling.  ACM Transactions on Graphics, 32(6):1–8, 2013.
  * [32] Jonathan Korein and Norman Badler.  Temporal anti-aliasing in computer generated animation.  In Annual Conference on Computer Graphics and Interactive Techniques, pages 377–388, 1983.
  * [33] Bee Lim and Kyoung Mu Lee.  Deep recurrent resnet for video super-resolution.  In Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, pages 1452–1455. IEEE, 2017.
  * [34] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 136–144, 2017.
  * [35] Ce Liu and Deqing Sun.  A bayesian approach to adaptive video super resolution.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 209–216. IEEE, 2011.
  * [36] Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala.  Video frame synthesis using deep voxel flow.  In IEEE International Conference on Computer Vision, pages 4463–4471, 2017.
  * [37] Gucan Long, Laurent Kneip, Jose M Alvarez, Hongdong Li, Xiaohu Zhang, and Qifeng Yu.  Learning image matching by simply watching video.  In European Conference on Computer Vision, pages 434–450. Springer, 2016.
  * [38] Simone Meyer, Oliver Wang, Henning Zimmer, Max Grosse, and Alexander Sorkine-Hornung.  Phase-based frame interpolation for video.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 1410–1418, 2015.
  * [39] Don P Mitchell and Arun N Netravali.  Reconstruction filters in computer-graphics.  ACM Siggraph Computer Graphics, 22(4):221–228, 1988.
  * [40] Uma Mudenagudi, Subhashis Banerjee, and Prem Kumar Kalra.  Space-time super-resolution using graph-cut optimization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5):995–1008, 2010.
  * [41] Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee.  Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, June 2019.
  * [42] Simon Niklaus and Feng Liu.  Context-aware synthesis for video frame interpolation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 1701–1710, 2018.
  * [43] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive convolution.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 670–679, 2017.
  * [44] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive separable convolution.  In IEEE International Conference on Computer Vision, pages 261–270, 2017.
  * [45] Sourabh A Niyogi and Edward H Adelson.  Analyzing gait with spatiotemporal surfaces.  In IEEE Workshop on Motion of Non-rigid and Articulated Objects, pages 64–69. IEEE, 1994.
  * [46] A Cengiz Oeztireli and Markus Gross.  Perceptually based downscaling of images.  ACM Transactions on Graphics, 34(4):1–10, 2015.
  * [47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, pages 8024–8035. 2019.
  * [48] Liad Pollak Zuckerman, Eyal Naor, George Pisha, Shai Bagon, and Michal Irani.  Across scales & across dimensions: Temporal super-resolution using deep internal learning.  In European Conference on Computer Vision. Springer, 2020.
  * [49] Kuldeep Purohit, Anshul Shah, and AN Rajagopalan.  Bringing alive blurred moments.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 6830–6839, 2019.
  * [50] Ramesh Raskar, Amit Agrawal, and Jack Tumblin.  Coded exposure photography: Motion deblurring using fluttered shutter.  ACM Transactions on Graphics, 25(3):795–804, 2006.
  * [51] Sidney Ray.  Scientific photography and applied imaging.  Routledge, 1999.
  * [52] Erik Reinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward, and Karol Myszkowski.  High dynamic range imaging: acquisition, display, and image-based lighting.  Morgan Kaufmann, 2010.
  * [53] Vijay Rengarajan, Shuo Zhao, Ruiwen Zhen, John Glotzbach, Hamid Sheikh, and Aswin C Sankaranarayanan.  Photosequencing of motion blur using short and long exposures.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 510–511, 2020.
  * [54] Meta Research.  fvcore.  https://github.com/facebookresearch/fvcore, 2019.
  * [55] Jaesung Rim, Geonung Kim, Jungeon Kim, Junyong Lee, Seungyong Lee, and Sunghyun Cho.  Realistic blur synthesis for learning image deblurring.  arXiv preprint arXiv:2202.08771, 2022.
  * [56] Alex Rogozhnikov.  Einops: Clear and reliable tensor manipulations with einstein-like notation.  In International Conference on Learning Representations, 2021.
  * [57] Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown.  Frame-recurrent video super-resolution.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 6626–6634, 2018.
  * [58] Oded Shahar, Alon Faktor, and Michal Irani.  Space-time super-resolution from a single video.  IEEE, 2011.
  * [59] C.E. Shannon.  Communication in the presence of noise.  Proceedings of the IRE, 37(1):10–21, Jan 1949.
  * [60] Eli Shechtman, Yaron Caspi, and Michal Irani.  Increasing space-time resolution in video.  In European Conference on Computer Vision, pages 753–768. Springer, 2002.
  * [61] Wang Shen, Wenbo Bao, Guangtao Zhai, Li Chen, Xiongkuo Min, and Zhiyong Gao.  Blurry video frame interpolation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 5114–5123, 2020.
  * [62] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 1874–1883, 2016.
  * [63] Mikio Shinya.  Spatial anti-aliasing for animation sequences with spatio-temporal filtering.  In Annual Conference on Computer Graphics and Interactive Techniques, pages 289–296, 1993.
  * [64] Hyeonjun Sim, Jihyong Oh, and Munchurl Kim.  Xvfi: extreme video frame interpolation.  In IEEE International Conference on Computer Vision, 2021.
  * [65] Takashi Suzuki.  Optical low-pass filter, Jan 1987.
  * [66] Hiroyuki Takeda, Peter Van Beek, and Peyman Milanfar.  Spatiotemporal video upscaling using motion-assisted steering kernel (mask) regression.  In High-Quality Visual Experience, pages 245–274. Springer, 2010\.
  * [67] Hossein Talebi and Peyman Milanfar.  Learning to resize images for computer vision tasks.  In IEEE International Conference on Computer Vision, pages 497–506, October 2021.
  * [68] Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia.  Detail-revealing deep video super-resolution.  In IEEE International Conference on Computer Vision, pages 4472–4480, 2017.
  * [69] Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu.  Tdan: Temporally-deformable alignment network for video super-resolution.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3360–3369, 2020.
  * [70] Suramya Tomar.  Converting video formats with ffmpeg.  Linux Journal, 2006(146):10, 2006.
  * [71] Matthew Trentacoste, Rafal Mantiuk, and Wolfgang Heidrich.  Blur-aware image downsampling.  In Computer Graphics Forum, volume 30, pages 573–582. Wiley Online Library, 2011.
  * [72] Bill Triggs.  Empirical filter estimation for subpixel interpolation and matching.  In IEEE International Conference on Computer Vision, volume 2, pages 550–557. IEEE, 2001.
  * [73] Hai Wang, Xiaoyu Xiang, Yapeng Tian, Wenming Yang, and Qingmin Liao.  Stdan: Deformable attention network for space-time video super-resolution.  arXiv preprint arXiv:2203.06841, 2022.
  * [74] Longguang Wang, Yulan Guo, Zaiping Lin, Xinpu Deng, and Wei An.  Learning for video super-resolution through hr optical flow estimation.  In Asian Conference on Computer Vision, pages 514–529. Springer, 2018.
  * [75] Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy.  EDVR: Video restoration with enhanced deformable convolutional networks.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [76] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy.  Esrgan: Enhanced super-resolution generative adversarial networks.  In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0–0, 2018.
  * [77] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [78] Nicolas Weber, Michael Waechter, Sandra C Amend, Stefan Guthe, and Michael Goesele.  Rapid, detail-preserving image downscaling.  ACM Transactions on Graphics, 35(6):1–6, 2016.
  * [79] J. Wills, S. Agarwal, and S. Belongie.  What went where [motion segmentation].  In IEEE Conference on Computer Vision and Pattern Recognition, 2003\.
  * [80] Xiaoyu Xiang, Qian Lin, and Jan P Allebach.  Boosting high-level vision with joint compression artifacts reduction and super-resolution.  In International Conference on Pattern Recognition. IEEE, 2020.
  * [81] Xiaoyu Xiang, Yapeng Tian, Yulun Zhang, Yun Fu, Jan P Allebach, and Chenliang Xu.  Zooming slow-mo: Fast and accurate one-stage space-time video super-resolution.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3370–3379, 2020.
  * [82] Xiaoyu Xiang, Yapeng Tian, Yulun Zhang, Yun Fu, Jan P Allebach, and Chenliang Xu.  Zooming slowmo: An efficient one-stage framework for space-time video super-resolution.  arXiv preprint arXiv:2104.07473, 2021.
  * [83] Zeyu Xiao, Zhiwei Xiong, Xueyang Fu, Dong Liu, and Zheng-Jun Zha.  Space-time video super-resolution using temporal profiles.  In ACM International Conference on Multimedia, pages 664–672, 2020\.
  * [84] Gang Xu, Jun Xu, Zhen Li, Liang Wang, Xing Sun, and Ming-Ming Cheng.  Temporal modulation network for controllable space-time video super-resolution.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 6388–6397, 2021.
  * [85] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman.  Video enhancement with task-oriented flow.  International Journal of Computer Vision, 127(8):1106–1125, 2019\.
  * [86] Lu Yuan, Jian Sun, Long Quan, and Heung-Yeung Shum.  Image deblurring with blurred/noisy image pairs.  ACM Transactions on Graphics, 26(3), 2007.
  * [87] Lihi Zelnik-Manor and Michal Irani.  Event-based analysis of video.  In IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pages II–II. IEEE, 2001.
  * [88] Kaihao Zhang, Wenhan Luo, Björn Stenger, Wenqi Ren, Lin Ma, and Hongdong Li.  Every moment matters: Detail-aware networks to bring a blurry image alive.  In ACM International Conference on Multimedia, pages 384–392, 2020\.
  * [89] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu.  Residual dense network for image super-resolution.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 2472–2481, 2018.
  * [90] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 9308–9316, 2019.
  * [91] X Zou, F Xiao, Z Yu, and YJ Lee.  Delving deeper into anti-aliasing in convnets.  In British Machine Vision Conference, 2020.

In this appendix, we first discuss the difference between video frame
interpolation and temporal upscaling in Sec. A. Then, we provide more module
design details in Sec. B. Besides, we provide the background knowledge of
space-time Fourier analysis in Sec. C to help the readers understand the
relevant part in the main paper. Sec. D includes more comparisons with the
state-of-the-art (SOTA) methods and the results of integrating other network
structures in the Space-Time Anti-Aliasing (STAA) framework. Lastly, we show
the training setting and more results of the extensive applications in Sec. E.

##  Appendix A Difference between Video Frame Interpolation and Temporal
Upscaling

In this section, we explain the difference between the video frame
interpolation (VFI) task and the temporal upscaling proposed in this paper.

(a) (a) Video Frame Interpolation

(b) (b) Temporal Upscaling

Figure 9: Illustration of the difference between video frame interpolation and
temporal upscaling. In VFI settings, each downsampled frame corresponds to
only a single frame of the original input. While for temporal upscaling, we
allow the downsampled frame to assimilate information from the adjacent frames
of the original input. Thus, we could reconstruct a full sequence from it.

Fig. 9 shows the illustration of these two different settings: given an input
sequence of frames $I_{1},I_{2},I_{3},I_{4},\ldots$, VFI adopts nearest-
neighbor downsampling to acquire the downsampled sequence
$I_{1},I_{2},\ldots$, as shown in (a). The acquired sequence explicitly
corresponds to the timestep in the original input. In the restoration step, it
needs to synthesize the missing frames from neighbors in the downsampled
sequence: _e.g_.

$I_{1},I_{3}\rightarrow\widetilde{I}_{2}.$ (4)

Thus, in the final sequence with a total length of $2n-1$, $n-1$ frames are
synthesized and $n$ frames are directly copied from the original input. So VFI
methods cannot perfectly reconstruct a sequence with an even number of frames.

Temporal upscaling is more like spatial-upscaling: given an input sequence
$I_{1},I_{2},I_{3},I_{4},\ldots$, the anti-aliasing filter blends the
information from nearby frames, thus each frame of the downsampled sequence
should contain information from multiple frames of the original input. Thus,
it does not explicitly correspond to the timestep in the original input. So we
annotate the downsampled frames as $I_{123},I_{234},\ldots$. Since this
downsampled representation contains the motion information at each timestep of
the original sequence, we should be able to reconstruct them with proper
design. For the restoration output, all frames in the sequence are
synthesized:

$I_{123},I_{234}\rightarrow\widetilde{I}_{1},\widetilde{I}_{2},\widetilde{I}_{3},\widetilde{I}_{4}.$
(5)

The temporal upscaling method can reconstruct a sequence with either an even
or odd number of frames.

Due to the above differences, it is unfair to directly compare the output
between these two methods. So in this paper, we only compare the frames that
are synthesized by both methods (_e.g_. in the above example,
$\widetilde{I}_{2},\widetilde{I}_{4},\ldots$) to measure the reconstruction
capability of the network. For the full sequence, we compare the temporal
profile to evaluate the reconstructed motion patterns.

##  Appendix B Module Design Details

In this section, we provide more details about the two modules of the
downsampler: the space-time anti-aliasing filter and the differentiable
quantization layer, and the two modules of the upsampler: the deformable
temporal modeling (DTM) and the residual dense block with 3D convolutions.

###  B.1 Space-Time Anti-Aliasing Downsampler

Fig. 10 illustrates the two operations in the downsampler: filtering with the
learned 3D space-time filter and downsampling with stride. In our
implementation, the above two steps are simplified into a convolution: the
convolution kernel size is the window size in space/time; the convolution
weights are learnable with custom constraints; the downsampling in each
dimension is controlled by stride.

Figure 10: Illustration of our learned space-time anti-aliasing downsampler.

###  B.2 Differentiable Quantization Layer

The direct output of our downsampler is a floating-point tensor, while in
practical applications, images are usually encoded as 8-bit RGB (uint8). To
make our downsampled frames compatible with popular image data storage and
transmission pipelines, we propose a quantization layer and include it in our
end-to-end training process.

Directly casting the tensor type to uint8 does not work: such operations are
not differentiable and thus cannot be used to train our network in an end-to-
end manner. Consequently, if we directly send a uint8 input to the upsampler,
there is going to be a performance drop due to the precision gap between the
float and the uint8. So we split the quantization for tensors into two steps:

(1) Clipping: limit the values to a certain range (_e.g_. [0, 255] for uint8)
to avoid blown-out colors. There is a serious problem: the derivative out of
the $[min,max]$ value is masked to be 0, which makes the training at early
iterations unstable, as reported in [28]. To solve this problem, we modify the
clipping function $Q(\cdot)$ into:

$\begin{split}\hat{x}&=x+\text{ReLU}(min-x),\\\
Q(x)&=\hat{x}-\text{ReLU}(\hat{x}-max),\end{split}$ (6)

where $x$ is our input value, $[min,max]$ denotes the clipping range, and
$Q(x)$ is the clipped output. The derivative of the clipping function becomes:

$\dfrac{dQ(x)}{dx}=\begin{cases}2,&\text{if $x\leq min$}\\\ 1,&\text{if
$x\in(min,max)$}\\\ 2,&\text{if $x\geq max$}\end{cases}$ (7)

This modification does not influence clipping results but makes the derivative
non-zero everywhere.

(2) Round: round the values to the closest integer. The gradient of this
function is zero almost everywhere, which is bad for optimization. To avoid
this issue, we override the gradient map to 1 during backward propagation.

As a result, the quantization operation’s derivative is non-zero everywhere,
thus enabling end-to-end training. Fig. 11 shows the training and validation
loss of our framework. It shows our proposed method can be trained stably from
scratch.

(a) Training Loss

(b) Validation Loss

Figure 11: Plot of our training progress. With the aid of the differentiable
quantization layer, our network’s training is stable from scratch.

###  B.3 Deformable Temporal Modeling (DTM)

Here we describe the detailed structure of the deformable temporal module in
Fig. 12: as illustrated in the main paper, the temporal correspondences are
propagated in both forward and backward directions. At each step $i$, it takes
a current feature $f_{i}$ as input and refine it with the last step’s output
$f^{\prime}_{i-1}$ with deformable alignment function $T(\cdot)$:

$f^{\prime}_{i}=T(f_{i},f^{\prime}_{i-1}).$ (8)

The deformable alignment function $T(\cdot)$ is basically a deformable
convolution, which takes an input feature map $f^{\prime}_{i-1}$ and an offset
map $\Delta p_{i}$:

$T(f_{i},f^{\prime}_{i-1})=DConv(f^{\prime}_{i-1},\Delta p_{i}),$ (9)

where the offset $\Delta p_{i}$ is estimated from the refined feature map from
the last step and the current step with convolutional function $g(\cdot)$:

$\Delta p_{i}=g(f^{\prime}_{i-1},f_{i}).$ (10)

In this way, the previous feature maps are aligned with the current one, which
enhances our network’s capability of handling motions. Then we aggregate the
current feature $f_{i}$ and the aligned feature $f^{\prime}_{i}$ to get the
refined feature: $r(f_{i})=c(f_{i},f^{\prime}_{i})$ with the ConvLSTM $c$.
After bi-directional propagation, for each timestep $i$, there will be two
refined features from each direction. The aggregated output can be acquired by
a blending function:

$DTM(f_{i})=w_{f}*r_{f}(f_{i})+w_{b}r_{b}(f_{i}),$ (11)

where $w_{f},w_{b}$ are conv$1\times 1$ kernels and $*$ denotes the
convolutional operator. In this way, we get a well-aligned sequence as output.

Figure 12: Illustration of the deformable temporal modeling network. The
temporal propagation is implemented in a recurrent manner with forward and
backward directions. At each step, we model the temporal correspondence
between the last refined frame and the current frame with deformable alignment
and output the refined current frame.

###  B.4 Residual Dense Block with 3D Convolutions

As the major building block of the reconstruction trunk in our upsampler, the
3D convolutions are organized exactly the same way as the residual dense block
in [89, 76] while replacing the 2D convolutions with 3D, as shown in Fig. 13.
The skip-connection of the residual block is with a residual scaling parameter
$\beta$; in the dense block, features of different hierarchical levels are
concatenated.

Figure 13: Illustration of the residual dense block with 3D convolution.

##  Appendix C Background of Space-Time Fourier Analysis

Given a video signal $f(x,y,t)$, we can analyze its space-time frequency
characteristics through Fourier transform ($\mathcal{F}(\cdot)$):

$F(\Omega_{x},\Omega_{y},\Omega_{t})=\mathcal{F}_{x,y,t}(f(x,y,t)),$ (12)

where $\Omega$ denotes the frequency for each dimension, and $F$ is the
Fourier transform of $f$. For the uniform-velocity case as discussed in our
main paper, we can denote the constant velocities as $v_{x}$ and $v_{y}$ for
each direction. Thus, the object signal is translated through time by:

$f(x,y,t)=g(x-v_{x}t,y-v_{y}t),$ (13)

where $g(x,y)$ is the signal of the 2D object. We denote its Fourier transform
as $G$. Correspondingly, the Fourier transform in Eq. 12 becomes:

$\begin{split}F(\Omega_{x},\Omega_{y},\Omega_{t})&=\mathcal{F}_{x,y,t}[g(x-v_{x}t,y-v_{y}t)]\\\
&=G(\Omega_{x}\Omega_{y})\int e^{-2\pi
it(\Omega_{x}v_{x}+\Omega_{y}v_{y}+\Omega_{t})}dt\\\
&=G(\Omega_{x}\Omega_{y})\delta(\Omega_{x}v_{x}+\Omega_{y}v_{y}+\Omega_{t}).\end{split}$
(14)

This explains why moving the 2D object would result in a shearing in the
frequency domain: theoretically, all the non-zero frequency components should
lie on the plane $\Omega_{x}v_{x}+\Omega_{y}v_{y}+\Omega_{t}$ in the 3D
Fourier space, which reflects the coupling of space and time dimensions.
However, our digital image is discrete – our simulated temporal profile has
jigsaw-like boundaries between the moving object and the background due to the
motion being at the pixel level. As a result, the transformed figure has two
small sub-bands, as shown in the main paper. For visualization purposes, our
analysis in the main paper is on the $xt$ 2D signals, thus Eq. 14 can be
written as:

$F(\Omega_{x},\Omega_{t})=G(\Omega_{x})\delta(\Omega_{x}v_{x}+\Omega_{t}),$
(15)

which shows the spectrum is limited by the delta function to a single line
$\Omega_{x}v_{x}+\Omega_{t}=0$. The velocity $v_{x}$ decides the slope of this
line. For the $xt$ 2D signal, convolving with a filter $h(\cdot)$ equals to a
multiplication with $H(\cdot)$ in the Fourier domain:

$H_{F}(\Omega_{x},\Omega_{t})=G(\Omega_{x})\delta(\Omega_{x}v_{x}+\Omega_{t})H(\Omega_{x},\Omega_{t}).$
(16)

This explains the effect of different filters: Gaussian filter’s Fourier
transform is still a Gaussian function, which gradually attenuates the spatio-
temporal high frequency; box filter (motion blur)’s Fourier transform is a
sinc function, which attenuates certain frequency components.

##  Appendix D More Experiments

###  D.1 More Comparisons with State-of-the-Art Methods

In this section, we first show more visual results that compare the previous
reconstruction methods with our joint-learned STAA in Fig. 14 and Fig. 15.

Fig. 14 shows the temporal upscaling results ($1\times s,2\times t$). Compared
with previous VFI methods, the reconstruction results of our proposed STAA
framework have more vivid spatial textures while preserving the correct motion
patterns. Because our STAA downsampler can encode the motion of each frame in
the original input sequence, our reconstruction result maintains the fast
motions that are lost in the previous nearest-neighbor sampling (see the
pigeon in the second row).

Fig. 15 shows the space-time upscaling results ($4\times s,2\times t$). We
choose SepConv [44], DAIN [4], FLAVR [25] and XVFI [64] as VFI methods, and
EDVR [75], BasicVSR++ [9] as VSR methods, ZSM [81, 82] as STVSR methods for
comparison. Benefiting from the joint encoding of space-time dimensions, our
reconstruction result has much richer spatial details even without the aid of
perceptual or adversarial loss, which is almost impossible to be restored by
previous methods. These results demonstrate the great advantage of co-
designing the downsampler with the upsampler, which could be a potential
direction for the video reconstruction task.

(a) Input-Overlayed

(b) GT

(c) FLAVR [25]

(d) XVFI [64]

(e) Ours

Figure 14: Qualitative results of $1\times s,2\times t$ upscaling on Vid4
dataset. Compared with previous video frame interpolation methods, the
reconstruction results of our proposed STAA framework have better textures
while preserving the motion patterns due to the co-design of downsampler and
upsampler.

(a) Input-Overlayed

(b) GT

(c) SepConv [44]+EDVR [75]

(d) DAIN [4]+EDVR [75]

(e) FLAVR [25]+BasicVSR++ [9]

(f) XVFI [64]+BasicVSR++ [9]

(g) ZSM [81]

(h) Ours

(i) Input-Overlayed

(j) GT

(k) SepConv [44]+EDVR [75]

(l) DAIN [4]+EDVR [75]

(m) FLAVR [25]+BasicVSR++ [9]

(n) XVFI [64]+BasicVSR++ [9]

(o) ZSM [81]

(p) Ours

(q) Input-Overlayed

(r) GT

(s) SepConv [44]+EDVR [75]

(t) DAIN [4]+EDVR [75]

(u) FLAVR [25]+BasicVSR++ [9]

(v) XVFI [64]+BasicVSR++ [9]

(w) ZSM [81]

(x) Ours

Figure 15: Qualitative results of $4\times s,2\times t$ upscaling on Vid4
dataset. Compared with previous two-stage and one-stage space-time super-
resolution methods, the reconstruction results of our proposed STAA framework
have much better textures while preserving the motion patterns due to the co-
design of downsampler and upsampler.

We also compare the computational efficiency of different space-time video
upscaling methods by computing the FLOPs per million pixels (MP) using the
open-source tool fvcore [54], as shown in Table 4. Compared with the two-stage
methods with cascaded VFI and VSR networks, the one-stage space-time super-
resolution network is more efficient. Our proposed upsampler requires the
least computation cost among all compared methods.

Table 4: Comparison of computational costs among cascaded VFI and VSR methods, STVSR methods, and our method. Method | GFLOPs/MP
---|---
VFI | VSR
FLAVR [25] | BasicVSR++ [9] | 185.34+140.29
XVFI [64] | BasicVSR++ [9] | 676.65+140.29
ZSM [81] | 198.51
Ours | 163.98

###  D.2 Our Framework Can be Generalized to More Networks

To compare the influence of the joint-learning mechanism of our framework, we
switch the upsampler to an STVSR network ZSM 111Note that the reconstructed
result is not comparable with our video upscaling setting of $4\times
s,2\times t$, since our network decodes 8 frames from 4 LR inputs, which is
more challenging than decoding 7 out of 4., which converts 4 LR frames to 7 HR
frames. In the updated STAA-ZSM setup, the 4 LR frames are generated by our
STAA downsampler and then sent to the ZSM for reconstruction. Compared with
the original ZSM in Table 5, we can observe that the STAA downsampled
representation improves the PSNR by 2.14 dB and SSIM by 0.0348. These results
demonstrate that the improvement brought by the joint learning framework can
be generalized to other networks.

Table 5: Using STAA filter for space-time super-resolution. Method | Vimeo-90k
---|---
PSNR | SSIM
ZSM [81] | 33.48 | 0.9178
STAA-ZSM | 35.62 | 0.9526

To evaluate the effectiveness of our upsampler design, we train another
framework under the setting $1\times s,2\times t$ by substituting the
upsampler with FLAVR [25]. We modify the input and output numbers of the FLAVR
to make it compatible with the temporal upscaling ratio. From Table 6, we can
see that our proposed upsampler outperforms FLAVR by a large margin.
Considering that both networks adopt 3D convolution as the basic building
block, we believe that such improvement should attribute to the deformable
temporal modeling.

Table 6: Performance comparison for temporal upscaling: $2\times t$. Method | Vimeo-90k | Params(M)
---|---|---
PSNR | SSIM
STAA-FLAVR [25] | 44.72 | 0.9915 | 42.1
STAA-Ours | 46.59 | 0.9942 | 15.9

##  Appendix E Extensive Applications

In this section, we provide the training details for each application, and
show more results and analysis. Besides, we also discuss the trade-off between
the data storage and the reconstruction performance.

###  E.1 Frame Rate Conversion

Training Details. We adopt REDS [41] as our training set: it contains 270
videos of dynamic scenes at $720\times 1280$ resolution, in which 240 videos
are split as the training set and 30 videos as the validation set. In this
application, we train our upsampler network with the training set. During the
training, we take the 5-frame sequence as input and the 6-frame sequence as
the supervision for output.

To quantitatively evaluate the frame rate conversion on in-the-wild videos of
our network, we took 120-fps video clips using a high-speed camera. In this
way, we can get the input and corresponding ground truth for 20-fps and 24-fps
frames by sampling every 6 or 5 frames, respectively. We measure the PSNR and
SSIM of the output and show the results in Table 7. We also take the outputs
of several popular video editing tools and software for comparison: ffmpeg
[70] (skip/duplicate frames), and Adobe Premiere Pro [1]. Adobe Premiere Pro
has three options for frame rate conversion: frame sampling, frame blending,
and optical flow warping. Among these compared methods, only “blending” and
“warping” can synthesize new frames at the new timestamp. We show the
synthesized frames in Fig. 16.

From Table 7, we can conclude that the frame duplication and sampling perform
the worst since it cannot synthesize frames at the new timestamp. Frame
blending is a little bit better in terms of PSNR and SSIM; still, it suffers
from ghosting artifacts (transparent edges due to the overlayed objects) as
shown in the second figure in Fig. 16. Optical flow warping can synthesize the
correct motion for the new timestamp and thus improve the reconstruction PSNR
and SSIM. However, the reconstruction performance relies on the estimated
optical flow field: if the estimation accuracy is low, then the warped frames
would have holes, as shown in the third figure. Compared with the above
methods, our network does not have these artifacts and achieves the highest
reconstruction quality in terms of PSNR and SSIM.

Table 7: Performance comparison for frame rate conversion: 20 to 24 fps. Method | ffmpeg [70] | premiere-sampling | premiere-blend | premiere-warp | Ours
---|---|---|---|---|---
PSNR | 32.44 | 32.39 | 33.90 | 34.54 | 36.99
SSIM | 0.9514 | 0.9474 | 0.9536 | 0.9588 | 0.9784

(a) GT

(b) premiere-blend

(c) premiere-warp

(d) Ours

Figure 16: Comparison of the intermediate frame synthesized by different
methods. The blending result has obvious ghosting artifacts (transparent
edges), while the optical-flow warping result has holes in the leaf. Compared
with the other two methods, our result does not have these artifacts.

###  E.2 Blurry Frame Reconstruction

Training Details. For this application, we adopt REDS [41] as our training
set: it provides blurry videos at 24-fps and the corresponding clean videos at
both 24-fps or 120-fps. We conduct two experiments: in the main paper, we show
the results of $4\times s,2\times t$, which are trained with bicubic-
downsampled blurry videos at 12-fps as input, and clean HR videos at 24-fps as
output. Besides, we also experiment on another setting with temporal-upscaling
only: $1\times s,5\times t$, which means the model needs to decode 5 clean
frames per input frame. This network is trained with 24-fps blurry videos as
input and 120-fps clean videos as output. To handle the motion ambiguity, we
take 2 frames as input and make the network generate $2\times t$ frames as
output. We use the training subset of REDS to train the networks and show
results on the validation subset.

To further illustrate the effectiveness of our upsampler design, We compare
our upsampler’s structure with FLAVR [25] under the setting $1\times s,5\times
t$ in Table 8 trained with the same hyperparameters. Our method exceeds it by
a large margin in terms of PSNR and SSIM. We also provide the qualitative
results of the validation videos from REDS in Fig. 18. As we can see, our
method can reduce the motion blur and produce clean and crisp frames. Compared
with FLAVR, our network generates vivid textures for each frame while keeping
a good motion pattern at each timestep.

Table 8: Comparison for blurry frame upscaling: $1\times s,5\times t$. Method | REDS
---|---
PSNR | SSIM
FLAVR [25] | 28.50 | 0.8337
Ours | 32.29 | 0.9153

(a) (a) PSNR

(b) (b) SSIM

Figure 17: Plot of reconstruction PSNR/SSIM for different percentages of
pixels in the downsampled representation. With the increase of pixel
percentage, the reconstruction results improves.

(a) Input

(b) Input

(c) GT-0

(d) GT-1

(e) GT-2

(f) GT-3

(g) GT-4

(h) FLAVR-0

(i) FLAVR-1

(j) FLAVR-2

(k) FLAVR-3

(l) FLAVR-4

(m) Ours-0

(n) Ours-1

(o) Ours-2

(p) Ours-3

(q) Ours-4

Figure 18: Qualitative results of the blurry frame reconstruction on REDS. The
first row is the input blurry frame, and the other rows are corresponding
clean frames. Our method can generate clean frames with vivid textures and
correct motions at each timestep.

###  E.3 Efficient Video Storage and Transmission

For this application, we assume the video capturing and reconstruction steps
are asynchronous: the captured video can be downsampled and saved or
transmitted for later reconstruction. In this process, the downsampling filter
can help save the data to be stored or transmitted while keeping high-quality
reconstruction results. Our proposed downsampling method is a better way to
reduce the number of pixels in space and time dimensions, and it is compatible
with the previous image and video compression methods in the standard ISP
pipeline. In practical applications, a video can be first resized by our STAA
downsampler and then compressed with video codecs for transmission.
Correspondingly, it requires the upsampler to be trained with the compressed
data for better handling the compressed video reconstruction. We do not
conduct the above experiments since it is beyond the scope of this paper. It
can be a promising direction for future works.

For efficient video storage and transmission, we care about the downsampling
ratio since it directly influences the amount of data to be stored or
transmitted: intuitively, the restoration quality should increase with the
sampling ratio. In the extreme case when the ratio = 1.0, the
stored/transmitted video is exactly the original input. We discuss the trade-
off between the data storage and reconstruction performance as follows.

Trade-off between Storage and Reconstruction Performance. We evaluate the
percentage of pixels kept in the downsampled representation: for temporal
downsampling ratio of $t$, there will be $1/t$ pixels kept; for spatial
downsampling ratio of $s$, there will be $1/s^{2}$ pixels kept because of the
reduction in both height and width. We conduct a series of experiments with
different combinations of space and time downsampling factors and plot the
influence of pixel percentage in downsampled representation to the
reconstruction PSNR/SSIM calculated on Vimeo-90k’s testset in Fig. 17, where
the x-axis is the base-10 logarithmic scale of the pixel percentage, and the
y-axis refers to PSNR and SSIM. All the experiments are done with the same
network structure and trained with the same hyperparameters on Vimeo-90k.

From plot (a), we can observe that the reconstruction PSNR is almost linearly
correlated to the log of pixel percentage. The SSIM plot increases slower when
the percentage becomes higher, and it will eventually reach the top-right
corner when the percentage is $100\%$ and SSIM=1.0 (original video). These
plots reflect some interesting facts:

1\. Under our STAA framework with jointly learned downsampler and upsampler,
the reconstruction performance only relates to the percentage of pixels kept
in the downsampled representation, regardless of whether the reduction happens
on time or space dimension. This trend validates our method’s effectiveness in
maintaining the spatio-temporal characteristics of the video;

2\. With this plot, we can estimate a rough range of reconstruction
performance given a specific downsampling ratio. This trade-off can guide the
design of the video restoration system by answering the following question:
what is the lowest percentage of pixels that satisfies the reconstruction
quality requirement;

3\. This plot can serve as a benchmark of the network’s restoration
capability: a better network should move the plot to the top left (reconstruct
the best quality with the least percentage of pixels). Since the plot is
acquired through different space/time downsampling settings, it reflects the
general restoration ability beyond specific tasks.