# Real-World Single Image Super-Resolution:
A Brief Review

Honggang Chen, Xiaohai He, Linbo Qing, Yuanyuan Wu, Chao Ren, and Ce Zhu This
work was supported by the National Natural Science Foundation of China under
Grant 62001316, Grant 61801316, and Grant 61871279. (_Corresponding author:
Chao Ren._)Honggang Chen, Xiaohai He, Linbo Qing, and Chao Ren are with the
College of Electronics and Information Engineering, Sichuan University,
Chengdu 610065, China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>chaoren@scu.edu.cn). Yuanyuan Wu is with the College of
Information Science and Technology, Chengdu University of Technology, Chengdu,
China, 610059 (e-mail: wuyuanyuan@cdut.edu.cn). Ce Zhu is with the School of
Information and Communication Engineering, University of Electronic Science
and Technology of China, Chengdu 611731, China (e-mail<EMAIL_ADDRESS>
###### Abstract

Single image super-resolution (SISR), which aims to reconstruct a high-
resolution (HR) image from a low-resolution (LR) observation, has been an
active research topic in the area of image processing in recent decades.
Particularly, deep learning-based super-resolution (SR) approaches have drawn
much attention and have greatly improved the reconstruction performance on
synthetic data. Recent studies show that simulation results on synthetic data
usually overestimate the capacity to super-resolve real-world images. In this
context, more and more researchers devote themselves to develop SR approaches
for realistic images. This article aims to make a comprehensive review on
real-world single image super-resolution (RSISR). More specifically, this
review covers the critical publically available datasets and assessment
metrics for RSISR, and four major categories of RSISR methods, namely the
degradation modeling-based RSISR, image pairs-based RSISR, domain translation-
based RSISR, and self-learning-based RSISR. Comparisons are also made among
representative RSISR methods on benchmark datasets, in terms of both
reconstruction quality and computational efficiency. Besides, we discuss
challenges and promising research topics on RSISR.

###### Index Terms:

Super-resolution, Real-world image, Deep learning, Datasets, Assessment
Metrics, Review

##  I Introduction

High-resolution (HR) images are desired urgently in many application areas
such as intelligent surveillance, medical imaging, and remote sensing. To
obtain images with higher resolution, a natural idea is to upgrade the
hardware (_e.g._ , the imaging system). Although recent years have witnessed
the obvious progress of imaging devices and techniques, this kind of approach
has two main limitations: (i) It is inflexible and costly because the demand
in practical applications is constantly changing. (ii) It can be used only for
capturing new HR images, but not for enhancing the resolution of existing low-
resolution (LR) images. Compared to the hardware upgrade-based “hard”
solution, the signal processing-based “soft” image resolution enhancement
known as super-resolution (SR) is more flexible and economical. With the SR
techniques that reconstruct a higher resolution output from the LR
observation, we can obtain images with the resolution beyond the limit of
imaging systems, thereby benefiting the subsequent analysis and understanding
tasks such as segmentation [1, 2, 3, 4], detection [5, 6, 7], and recognition
[8, 9, 10].

Figure 1: The taxonomy of existing super-resolution techniques.

In general, as presented in Fig. 1, existing SR techniques can be grouped into
two categories according to the LR input and the reconstructed HR output,
_i.e._ , video super-resolution (VSR) [11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
21, 22, 23, 24, 25, 26, 27] and image super-resolution (ISR) [28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69,
70, 71, 72, 73, 74, 75, 76, 77, 78]. On the whole, VSR aims to improve the
spatial resolution (known as spatial VSR) [11, 12, 13, 14, 15, 16, 17, 18, 19]
or the frame rate (known as temporal VSR) [20, 21, 22, 23, 24, 25, 26, 27] of
the observed video. ISR can be further classified into multi-frame image
super-resolution (MISR) [28, 29, 30, 31, 32, 33, 34, 35] and single image
super-resolution (SISR) [36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48,
49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,
68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78]. MISR refers to reconstructing an
HR image via fusing the complementary information in a series of correlated
images of the same scene [28, 29, 30, 31, 32, 33, 34, 35], while SISR
generates an HR image from one LR observation [36, 37, 38, 39, 40, 41, 42, 43,
44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62,
63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78]. In terms of
application scenarios, SISR is more practical than MISR and VSR because it is
much less demanding on the input, which is one reason why SISR attracts wider
attention. A variety of SISR methods have been proposed in the past decades,
mainly including reconstruction-based [36, 37, 38, 39, 40, 41, 42, 43, 44,
45], example-based [46, 47, 48, 49, 50, 51, 52], sparse representation-based
[53, 54, 55, 56, 57, 58, 59, 60], regression-based [61, 62, 63, 64, 65, 66],
and deep learning-based approaches [67, 68, 69, 70, 71, 72, 73, 74, 75, 76,
77, 78], _etc_. Particularly, the deep learning-based SISR methods [67, 68,
69, 70, 71, 72, 73, 74, 75, 76, 77, 78] developed in recent years take the SR
performance on synthetic LR images (_e.g._ , bicubically downsampled images)
to a new level.

Nevertheless, previous research [79] shows that the actual SR ability of most
existing SISR methods may be overestimated based only on synthetic data due to
the domain gap between synthetic and realistic data. In other words, compared
to the promising SR results on synthetic test images, the SR performance would
degrade significantly on real-world images, thus hindering the practical
applications of SISR algorithms. To address this problem, some researchers
have shifted their focus to real-world single image SR (RSISR) over the past
couple of years, and a series of studies involving real-world dataset
collection [80, 81, 82, 83, 84, 79, 85, 86, 87], SR models for real-world
images [88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 80, 83, 86, 87, 82, 84,
79, 81, 85, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111,
112, 113, 114, 115], and SR result assessment [116, 117, 118, 119] have been
conducted. Meanwhile, several challenges on RSISR have been organized in
conjunction with the _IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)_ , _IEEE International Conference on Computer Vision
(ICCV)_ , and _European Conference on Computer Vision (ECCV)_ to attract more
attention and promote the development of RSISR techniques [120, 121, 122,
123]. It is exciting to see that the studies on RSISR are becoming more
targeted and the SR performance on real-world images is improved increasingly.

In this work, we mainly give an overview of recent RSISR algorithms and
relevant studies. There are several works concerning the overview of image and
video SR techniques. For example, Yue _et al._ [124] make a summary of the
techniques, applications and future of image SR. Considering that deep
learning has been widely employed to address SR over the past few years, more
recently Yang _et al._ [125], Wang _et al._ [126], and Liu _et al._ [127]
review deep learning-based image/video SR methods. However, this work is the
first attempt to make an overview of RSISR techniques to the best of our
knowledge. The main contributions of this review are four-fold: (i) We
comprehensively review the studies on RSISR, including datasets, assessment
metrics, technologies and methods, _etc_. (ii) We present a taxonomy for
existing RSISR methods according to their primary principles. (iii) We compare
the reconstruction accuracy and efficiency of representative RSISR algorithms
on benchmark datasets. (iv) We further discuss current challenges and future
research directions for RSISR.

The rest of this review is organized as follows. The background of RSISR is
briefly introduced in Section II. In Section III, the datasets and assessment
metrics for RSISR are described. Section IV reviews RSISR technologies and
methods by category. The comparisons among representative RSISR algorithms are
presented in Section V. In Section VI, we analyze current challenges and
future research directions of RSISR. Finally, Section VII concludes this work.

##  II Background

SISR refers to reconstructing an HR image from an LR observation. Given an LR
image ${\bf{Y}}$, it is generally assumed to be degraded from a corresponding
HR image ${\bf{X}}$, which can be represented as

${\bf{Y}}=D({\bf{X}},{\theta_{D}})$ (1)

where $D(\cdot)$ denotes the degradation process defined by the parameter set
${\theta_{D}}$. Note that in a real scenario, the degradation parameter
${\theta_{D}}$ is unknown, and all we have is the LR image ${\bf{Y}}$. SISR
aims at recovering a good estimate of the potential HR image via reversing the
degradation process shown in Eq. (1), which can be formulated as

${\bf{\hat{X}}}=R({\bf{Y}},{\theta_{R}})$ (2)

where $R(\cdot)$ represents the SR function and ${\theta_{R}}$ is the
corresponding parameter set. ${\bf{\hat{X}}}$ is the super-resolved image from
${\bf{Y}}$, _i.e._ , an estimate of the real HR image ${\bf{X}}$.

Apparently, the SR process and degradation process are the inverses of each
other. Thus, for obtaining excellent reconstruction performance, the SR
function $R({\bf{Y}},{\theta_{R}})$ should be adapted to the degradation
$D({\bf{X}},{\theta_{D}})$. In the literature, some researchers [36, 37, 38,
39, 40, 41, 42, 43, 44, 45] approximate the degradation via blurring,
downsampling, and noise injection. Mathematically, the simulated degradation
process is as follows

${\bf{Y}}={\bf{SBX}}+{\bf{n}}$ (3)

where $\bf{B}$ and $\bf{S}$ denote the operations of blurring and
downsampling, respectively. In general, the blurring is realized via
convolving the HR image with a Gaussian kernel. $\bf{n}$ represents the
additive noise, which is usually assumed to be white Gaussian noise. Part of
works [61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78]
adopt a simpler degradation model, _i.e._ , directly downscaling an HR image
using the “bicubic” kernel to generate corresponding LR image. For comparison
and evaluation, most existing SR methods are developed and validated on
synthetic LR images generated by degradation simulations. Overall, the SR
reconstruction performance on synthetic LR images is rather good, especially
for deep learning-based SISR approaches such as RCAN [73], SAN [75], and
RFANet [77].

Compared with the commonly used degradation model in simulations, the actual
degradation in real-world scenarios is more complex and varying because it can
be affected by a number of factors (_e.g._ , imaging system and imaging
environment). In other words, the degradation model used in simulations may
not match that suffered by real-world images, which results in the domain gap
between synthetic LR images and realistic LR observations. For this primary
reason, the reconstruction performance of most existing SISR algorithms drops
significantly on real-world images. To enhance quality of super-resolving
real-world images, some researchers have been working on RSISR from different
perspectives in the past several years, including realistic dataset building,
SR model development, SR performance assessment,  _etc_.

##  III Datasets and Assessment Metrics

Training/testing datasets and assessment metrics are the cornerstones of the
SISR. In this section, we briefly introduce the relevant datasets and
assessment metrics.

###  III-A Datasets for RSISR

For the training and testing of SISR models, the widely used datasets include
DIV2K [128], BSDS500 [129], T91 [53], Set5 [130], Set14 [54], Urban100 [50],
Manga109 [131], _etc_. Most of these datasets only contain HR images. In this
case, we need to generate LR counterparts based on the assumed degradation
model (_e.g._ , “bicubic” kernel-based downsampling), both for model training
and testing. Therefore, these datasets are not quite suitable for the study of
RSISR due to the significant discrepancy between the assumed degradation model
and the real one. To address this problem, some more targeted datasets for
RSISR have been constructed in different ways, including DIV2KRK [94], RealSR
[80], DRealSR [81], City100 [82], SR-RAW [83], TextZoom [84], SupER [79],
ImagePairs [85],  _etc_. Table I summarizes the above datasets.

TABLE I: An overview of datasets for RSISR.

Datasets |  Published |  Synthetic / Realistic |  Scale Factors |  Keywords
---|---|---|---|---
DIV2KRK |  NeurIPS-2019 [94] |  Synthetic |  $\times$2, $\times$4 |  DIV2K, Random kernels, Uniform multiplicative noise
RealSR |  ICCV-2019 [80] |  Realistic |  $\times$2, $\times$3, $\times$4 |  Focal length adjusting
DRealSR |  ECCV-2020 [81] |  Realistic |  $\times$2, $\times$3, $\times$4 |  Focal length adjusting
City100 |  CVPR-2019 [82] |  Realistic |  $\times$2.9, $\times$2.4 |  Focal length adjusting, Shooting distance changing
SR-RAW |  CVPR-2019 [83] |  Realistic |  $\times$4, $\times$8 |  Focal length adjusting, RAW data
TextZoom |  ECCV-2020 [84] |  Realistic |  $\times$2 |  Text, Recognition
SupER |  TPAMI-2020 [79] |  Realistic |  $\times$2, $\times$3, $\times$4 |  Hardware binning, Image sequences
ImagePairs |  CVPRW-2020 [85] |  Realistic |  $\times$2 |  Beam-splitter cube, RAW data

####  III-A1 DIV2KRK [94]

This is a synthetic testing dataset built by Bell-Kligler _et al._ [94] for
blind SR. 111DIV2KRK [94] is available at
http://www.wisdom.weizmann.ac.il/~vision/kernelgan/ The DIV2KRK [94] is
derived from DIV2K [128] that contains diverse 2K resolution images. DIV2K is
proposed by Agustsson _et al._ [128] for NTIRE 2017 SR challenge. More
specifically, the 100 HR images in the validation set of DIV2K [128] are
blurred and downsampled with random kernels to generate LR counterparts.
Random blur kernels are $11\times 11$ anisotropic gaussians, and each of them
is with two independently distributed lengths ${\lambda_{1}},{\lambda_{2}}\sim
u(0.6,5)$ and a rotation angle $\theta\sim u\left[{-\pi,\pi}\right]$. Further,
uniform multiplicative noise is applied to the blur kernel before normalizing
its sum to one. DIV2KRK [94] is still a synthetic dataset, although the
degradation model is more complex and random.

####  III-A2 RealSR [80]

RealSR is a real-world dataset collected by Cai  _et al._ [80] for training
and testing RSISR models. 222RealSR [80] is available at
https://github.com/csjcai/RealSR RealSR consists of 595 LR-HR image pairs
obtained by adjusting the lens of two digital single lens reflex (DSLR)
cameras (_i.e._ , Nikon D810 and Canon 5D3). To study the RSISR with different
scaling factors (_e.g._ , $\times$2, $\times$3, $\times$4), images are taken
at four focal lengths, _i.e._ , 28mm, 35mm, 50mm, and 105mm. Naturally, for
the same scene, the images captured at 105mm are used as HR images, and the
images captured at 28mm, 35mm, and 50mm are seen as LR counterparts for
$\times$4, $\times$3, and $\times$2 upsampling, respectively. Considering the
differences (_e.g._ , lens distortion and exposures) among the images captured
at different focal lengths, Cai _et al._ [80] further propose a progressive
image registration framework to achieve pixel-wise registration of images
taken at 28mm, 35mm, 50mm, and 105mm. First, PhotoShop is used to correct the
lens distortion and the regions of interest around the center of the corrected
images are cropped. Then, the regions cropped from the images captured at
105mm are used as HR references, and the corresponding regions cropped from
the images taken at 28mm, 35mm, and 50mm are aligned to generate LR
counterparts via iteratively optimizing the parameters of affine
transformation and luminance adjustment. After convergence, real-world LR-HR
image pairs can be obtained.

####  III-A3 DRealSR [81]

The real-world dataset DRealSR built by Wei  _et al._ [81] is similar to
RealSR [80], with a larger scale. 333DRealSR [81] is available at
https://github.com/xiezw5/Component-Divide-and-Conquer-for-Real-World-Image-
Super-Resolution More specifically, five DLSR cameras (_i.e._ , Sony, Canon,
Olympus, Nikon, and Panasonic) are used to capture images at four resolutions
in outdoor and indoor scenes (_e.g._ , buildings, offices, plants, posters,
_etc._). The SIFT algorithm [132] is adopted to align the images with
different resolutions. In total, DRealSR [81] contains 884, 783, and 840 LR-HR
image pairs for $\times$2, $\times$3, and $\times$4 SR, respectively.

####  III-A4 City100 [82]

The City100 dataset proposed by Chen _et al._ [82] includes City100_NikonD5500
and City100_iPhoneX, which characterize the resolution and field-of-view (FoV)
degradation under DSLR and smartphone cameras, respectively. 444City100 [82]
is available at https://github.com/ngchc/CameraSR As is well known, there is
an tradeoff between the resolution and FoV for imaging systems. An image with
a larger FoV but a lower resolution can be obtained when zooming out the lens,
while the image resolution can be enhanced at the expense of a reduced FoV
when zooming in the lens. Therefore, Chen _et al._ propose [82] to adjust the
focal length or shooting distance to capture images of the same scene with
different resolutions. 100 postcards of different city scenes are used as
imaging subjects. For the City100_NikonD5500 dataset taken by NikonD5500, HR
and LR images are captured at the focal length of 55mm and 18mm, respectively.
The HR and LR images of the same scene are first spatially aligned based on
SIFT key-points [132] and RANSAC [133]. Further, intensity and color
rectification is conducted to improve the accuracy of alignment. The main
distinction between City100_iPhoneX and City100_NikonD5500 is that
City100_iPhoneX is captured by iPhone X via changing the shooting distance.

####  III-A5 SR-RAW [83]

The SR-RAW dataset proposed by Zhang  _et al._ [83] is composed of pairs of
RAW images taken at different levels of optical zoom. 555SR-RAW [83] is
available at https://ceciliavision.github.io/project-pages/project-zoom.html
The SR-RAW [83] is similar to the RealSR [80] mentioned above in terms of the
way to capture images of the same scene with different resolutions, _i.e._ ,
adjusting the focal length. For SR-RAW [83], seven images of each scene are
taken under seven different optical zoom settings using a 24-240mm zoom lens
(_i.e._ , Sony FE 24-240mm). In total, 500 seven-image sequences are collected
in outdoor and indoor scenes. For image registration, the Euclidean motion
model is applied to describe the relationship between the images with
different resolutions and it is optimized by minimizing the enhanced
correlation coefficient as in [134]. Unlike the RealSR built by Cai _et al._
[80], SR-RAW [83] contains both raw sensor data and RGB images because it is
used for SR from raw data. That is, instead of the LR RGB image, its raw
sensor data is used as the input to reconstruct the corresponding HR RGB
image. Actually, the reconstruction process covers demosaicing, denoising, SR,
_etc_. Compared with 8-bit RGB images, as pointed out in [83], raw sensor data
generally contains more useful information for SR.

####  III-A6 TextZoom [84]

The TextZoom derived from RealSR [80] and SR-RAW [83] is the first real scene
text SR dataset constructed by Wang _et al._ [84]. 666TextZoom [84] is
available at https://github.com/JasonBoy1/TextZoom. More specifically, the
text images in TextZoom [84] are cropped from the images in RealSR [80] and
SR-RAW [83], including various natural scenes such as shops, street views, and
vehicle interiors. The content, direction, and focal length of each LR-HR text
image pair in TextZoom are provided in the annotation process. Moreover,
TextZoom [84] comprises three subsets according to difficulty levels, namely
easy, medium, and hard. Thanks to the well-organized annotations, TextZoom
[84] can be utilized to study text image SR as well as text recognition.

####  III-A7 SupER [79]

The SupER is built by Köhler _et al._ [79] via hardware binning. 777SupER [79]
is available at https://www.lms.tf.fau.eu/research/downloads/superresolution/
More specifically, more than 80,000 images are taken from 14 lab scenes at
four imaging resolutions and five compression levels, using a Basler
acA2000-50gm CMOS camera with a $f$/1.8, 16mm fixed-focus lens. What is
special about SupER [79] is that the imaging resolution is adjusted by
changing the binning factor, which naturally guarantees the perfect alignment
between LR and HR images. Three binning factors (_i.e._ , 2, 3, and 4) are
used to acquire LR images corresponding to the same HR image at three
different resolution levels. Meanwhile, to enhance the comprehensiveness, four
motion types, two photometric conditions, and five H.265 compression levels
are considered while building SupER [79]. Additionally, unlike most existing
datasets for SISR, image sequences instead of independent images are captured
in SupER [79], which is also applicable in the study on MISR algorithms.

####  III-A8 ImagePairs [85]

The ImagePairs proposed by Joze _et al._ [85] includes 11,421 LR-HR image
pairs of diverse scenes, in which LR and HR images are captured by an LR
camera (5MP) and an HR camera (20.1MP), respectively. 888ImagePairs [85] is
available at www.microsoft.com/en-us/research/project/imagepairs More
specifically, a beam-splitter cube is used to make the two cameras capture
images of the same scene simultaneously. Due to the difference in focal
length, the LR and HR cameras have different perspectives. Therefore, Joze _et
al._ [85] propose to generate pixel-wise aligned LR-HR image pairs via
applying the following four steps: ISP, image undistortion, pair alignment,
and margin cropping. The raw data collected by LR and HR cameras are first
converted to color images in the ISP process. Then, distortions (_e.g._ ,
tangential and radial distortions) caused by cameras are reduced via camera
calibration. Further, LR and HR images are aligned globally and locally.
Finally, 10% of the border is removed from each image to improve the matching
accuracy of image pairs. In addition to SR, ImagePairs [85] may be used for
ISP and other tasks as it includes raw images.

TABLE II: An overview of widely used assessment metrics for RSISR.

Metrics |  Published |  Full/No-reference |  Keywords
---|---|---|---
PSNR |  - |  Full-reference |  Mean squared error
SSIM |  TIP-2004 [135] |  Full-reference |  Structure similarity, Luminance, Contrast, Structures
IFC |  TIP-2005 [136] |  Full-reference |  Nature scene statistics, Gaussian scale mixtures
LPIPS |  CVPR-2018 [137] |  Full-reference |  Deep features, Human perceptual similarity
NIQE |  SPL-2012 [138] |  No-reference |  Quality-aware features, Multivariate Gaussian model
PIQE |  NCC-2015 [139] |  No-reference |  Perceptually significant spatial regions, Block level distortion map
NRQM |  CVIU-2017 [116] |  No-reference |  Statistical features, Regression forests, Linear regression model

###  III-B Assessment Metrics for Super-Resolved Images

In general, the quality assessment of super-resolved images is two-fold,
_i.e._ , human perception-based subjective evaluation and quality metrics-
based objective evaluation. Overall, the former is a more direct way and it is
more in agreement with the practical need. However, subjective evaluation
suffers the following limitations. (i) The assessment result is readily
affected by personal preferences. (ii) The evaluation process is often costly
and cannot be automated. By contrast, objective evaluation is more convenient
to use, although the results by different assessment metrics may not be
necessarily consistent with each other as well as subjective evaluation. Table
II reports commonly used metrics for evaluating the objective quality of
super-resolved images, including PSNR, SSIM [135], IFC [136], LPIPS [137],
NIQE [138], PIQE [139], and NRQM [116]. For the description, let
${\bf{X}}\in\mathbb{R}{{}^{H\times W\times C}}$ and
${\bf{\hat{X}}}\in\mathbb{R}{{}^{H\times W\times C}}$ denote the ground truth
image and the super-resolved image, respectively. $H$, $W$, and $C$ denote
width, height, and number of components, respectively.

####  III-B1 PSNR

Peak signal-to-noise ratio (PSNR) is the most widely used full-reference
objective quality assessment metric for image restoration (_e.g._ , SR,
denoising, deblocking, and deblurring). Given ${\bf{\hat{X}}}$ and ${\bf{X}}$,
the PSNR is defined as

${\rm{PSNR}}=10\cdot{\log_{10}}(\frac{{{L^{2}}}}{{{\rm{MSE}}}})$ (4)

where
${\rm{MSE=}}\frac{1}{{HWC}}\left\|{{\bf{X}}-{\bf{\hat{X}}}}\right\|_{2}^{2}$
denotes the mean squared error (MSE) between ${\bf{\hat{X}}}$ and ${\bf{X}}$,
and $L$ represents the maximum pixel value (_i.e._ , 255 for 8-bit images). It
can be seen from Eq. (4) that PSNR is more concerned with the proximity
between corresponding pixels in ${\bf{\hat{X}}}$ and ${\bf{X}}$, which results
in the low consistency with perceptual quality in some cases.

####  III-B2 SSIM [135]

The structure similarity index (SSIM) [135] is a full-reference objective
quality assessment metric that measures structural similarity. 999The source
code of SSIM [135] is available at
https://live.ece.utexas.edu/research/Quality/index_algorithms.htm More
specifically, the comparisons are jointly performed in the aspects of
luminance, contrast, and structures as

${\rm{SSIM=}}{\left[{l({\bf{X}},{\bf{\hat{X}}})}\right]^{\alpha}}{\left[{c({\bf{X}},{\bf{\hat{X}}})}\right]^{\beta}}{\left[{s({\bf{X}},{\bf{\hat{X}}})}\right]^{\gamma}}$
(5)

where
$l({\bf{X}},{\bf{\hat{X}}})=\frac{{2{\mu_{\bf{X}}}{\mu_{{\bf{\hat{X}}}}}+{C_{1}}}}{{{\mu_{\bf{X}}}^{2}+{\mu_{{\bf{\hat{X}}}}}^{2}+{C_{1}}}}$,
$c({\bf{X}},{\bf{\hat{X}}})=\frac{{2{\sigma_{\bf{X}}}{\sigma_{{\bf{\hat{X}}}}}+{C_{2}}}}{{{\sigma_{\bf{X}}}^{2}+{\sigma_{{\bf{\hat{X}}}}}^{2}+{C_{2}}}}$,
and
$s({\bf{X}},{\bf{\hat{X}}})=\frac{{{\sigma_{{\bf{X\hat{X}}}}}+{C_{3}}}}{{{\sigma_{\bf{X}}}{\sigma_{{\bf{\hat{X}}}}}+{C_{3}}}}$.
$\alpha$, $\beta$, and $\gamma$ are weighting parameters. ${{\mu_{\bf{X}}}}$
and ${{\sigma_{\bf{X}}}}$ denote the mean and standard deviation of
${\bf{X}}$, respectively. Similarly, ${{\mu_{\bf{\hat{X}}}}}$ and
${{\sigma_{\bf{\hat{X}}}}}$ denote the mean and standard deviation of
${\bf{\hat{X}}}$, respectively. ${{\sigma_{{\bf{X\hat{X}}}}}}$ is the
covariance between ${\bf{\hat{X}}}$ and ${\bf{X}}$. ${C_{1}}$, ${C_{2}}$, and
${C_{3}}$ are constants. Further, Eq. (5) can be simplified when
$\alpha=\beta=\gamma=1$ and ${C_{3}}=\frac{{{C_{2}}}}{2}$ as

${\rm{SSIM=}}\frac{{(2{\mu_{\bf{X}}}{\mu_{{\bf{\hat{X}}}}}+{C_{1}})(2{\sigma_{{\bf{X\hat{X}}}}}+{C_{2}})}}{{({\mu_{\bf{X}}}^{2}+{\mu_{{\bf{\hat{X}}}}}^{2}+{C_{1}})({\sigma_{\bf{X}}}^{2}+{\sigma_{{\bf{\hat{X}}}}}^{2}+{C_{2}})}}$
(6)

In comparison, SSIM [135] is reported to reflect visual quality better than
PSNR. Generally, PSNR and SSIM [135] are combined to assess the quality of the
restored image when the corresponding ground truth image is available.

####  III-B3 IFC [136]

The information fidelity criterion (IFC) [136] is a full-reference metric that
assesses the quality of images based on natural scene statistics. 101010The
source code of IFC [136] is available at
https://live.ece.utexas.edu/research/Quality/index_algorithms.htm Research
shows that the statistics of the space formed by natural images can be
characterized using various models (_e.g._ , the Gaussian scale mixtures).
Generally, distortions would disturb the statistics of natural scenes and make
images unnatural. According to these observations, Sheikh _et al._ [136]
propose to measure the visual quality of an image via jointly using the
natural scene and distortion models to quantify the mutual information between
the test image and reference. Overall, the IFC [136] performs well for the
quality assessment of super-resolved images [140].

####  III-B4 LPIPS [137]

The learned perceptual image patch similarity (LPIPS) [137] is a learned
metric for reference-based image quality assessment. 111111The source code of
LPIPS [137] is available at https://github.com/richzhang/PerceptualSimilarity
More specifically, LPIPS [137] is obtained via computing the ${l_{2}}$
distance between the reference and the test image in a deep feature space,
which shows a good agreement with human judgments.

####  III-B5 NIQE [138]

The natural image quality evaluator (NIQE) is a completely blind metric
without the knowledge of human judgments or distortions [138]. 121212The
source code of NIQE [138] is available at
https://live.ece.utexas.edu/research/Quality/index_algorithms.htm The
multivariate Gaussian (MVG) model is used to fit the “quality-aware” features
extracted from images. More specifically, the features include the parameters
of the generalized Gaussian distribution (GGD) and the asymmetric generalized
Gaussian distribution (AGGD) that characterize the behavior of image patches.
Then, the quality of an image is measured using the distance between the two
MVG models fitting natural images and the evaluated image.

####  III-B6 PIQE [139]

The perception-based quality evaluator (PIQE) is a no-reference image quality
assessment metric [139]. 131313The implementation of PIQE [139] is included in
Matlab as https://www.mathworks.com/help/images/ref/piqe.html Considering that
the attention of human visual system (HVS) is highly directed towards
spatially active regions, the test image is divided into non-overlapping
blocks and block-level analysis is conducted to identify distortion and grade
quality. Therefore, PIQE [139] can provide a spatial quality map. The overall
quality of the evaluated image can be obtained by pooling the block level
quality scores.

####  III-B7 NRQM [116]

This is a learned no-reference quality metric (NRQM) for assessing super-
resolved images [116]. 141414The source code of NRQM [116] is available at
https://github.com/chaoma99/sr-metric To predict the perceptual scores of
super-resolved images, three groups of statistical features including local
frequency features, global frequency features, and spatial features are
extracted. The selected features cover the distribution of discrete cosine
transform coefficients, the distribution of wavelet coefficients, the spatial
discontinuity property of pixel intensity, _etc_. On this basis, three
regression forests are used to model these features independently and their
results are combined linearly to estimate the final perceptual score. These
three forests and the linear regression model are trained on a large-scale
dataset of super-resolved images with perceptual scores. Overall, the visual
quality predicted by NRQM [116] matches subjective evaluation well on SR
results.

Figure 2: The taxonomy of existing real-world single image super-resolution
techniques.

##  IV Technologies and Methods

Researchers have been studying the SISR methods for practical applications.
Especially with the SR performance on synthetic data becoming better and
better, more and more attention has been paid to RSISR. Fig. 2 presents the
overall taxonomy of existing RSISR techniques. Note that we focus more on deep
learning-based methods. According to the primary principles and
characteristics of existing RSISR methods, we group them into four categories,
_i.e._ , degradation modeling-based methods [88, 89, 90, 91, 92, 93, 94, 95,
96, 97, 98], image pairs-based methods [80, 83, 82, 84, 79, 81, 85, 86, 87],
domain translation-based methods [99, 100, 101, 102, 103, 104, 105, 106, 107,
108, 109, 110], and self-learning-based methods [111, 94, 112, 113, 114, 115].
Fig. 3 and Table III summarize existing RSISR methods. It is worth noting that
one method may belong to different categories when it is viewed from different
perspectives. The following sections introduce these methods in detail.

Figure 3: Milestones of RSISR methods. TABLE III: An overview of existing
works on RSISR.

Methods |  Published |  Category |  Keywords
---|---|---|---
SupER |  TPAMI-2020 [79] |  Image pairs-based |  Benchmarking super-resolution on real data
LP-KPN (RealSR) |  ICCV-2019 [80] |  Image pairs-based |  Laplacian pyramid-based kernel prediction network
CDC (DRealSR) |  ECCV-2020 [81] |  Image pairs-based |  Component divide-and-conquer model, Gradient weighted loss
CameraSR (City100) |  CVPR-2019 [82] |  Image pairs-based |  SR from the perspective of camera lenses
ZLLZ (SR-RAW) |  CVPR-2019 [83] |  Image pairs-based |  RAW sensor data, Contextual bilateral loss
TSRN (TextZoom) |  ECCV-2020 [84] |  Image pairs-based |  Sequential residual block, Central alignment module, Gradient profile loss
ImagePairs |  CVPRW-2020 [85] |  Image pairs-based |  A new data acquisition technique for gathering real image data
RawSR |  CVPR-2019 [86], TPAMI-2020 [87] |  Image pairs-based |  Two-branch structure, Raw image, Color correction
SARNBSR |  ICIG-2015 [88] |  Degradation modeling-based |  Blur kernel estimation, Bi-$l_{0}$-$l_{2}$-norm regularization
AHTNBSR |  JMIV-2019 [89] |  Degradation modeling-based |  Adaptive heavy-tailed image priors, Nonparametric blur kernel estimation
IKC |  CVPR-2019 [90] |  Degradation modeling-based |  Iterative kernel correction, Spatial feature transform
SVDBSR |  TOG-2019 [91] |  Degradation modeling-based |  Degradation-aware SR network, Kernel discriminator network
DAN |  NeurIPS-2020 [92] |  Degradation modeling-based |  SR image restorer, Blur kernel estimator, End-to-end trainable
NBSR |  ICCV-2013 [93] |  Degradation modeling-based |  Kernel estimation, Recurrence of small image patches
KernelGAN |  NeurIPS-2019 [94] |  Degradation modeling-based |  Deep internal learning, Cross-scale recurrence property, GAN, Image-specific SR-kernel
FISR |  ECCV-2018 [95] |  Degradation modeling-based |  High-to-low GAN, Low-to-high GAN, Unpaired LR and HR images
KMSR |  ICCV-2019 [96] |  Degradation modeling-based |  Realistic blur kernels, Blur kernel pool augment, GAN
DML |  ACCV-2020 [97] |  Degradation modeling-based |  Realistic HR-LR image pair synthesis, Pixel-wise spatially variant degradation kernel
RSRKN |  CVPRW-2020 [98] |  Degradation modeling-based |  Degradation framework, Kernel estimation, Noise injection, GAN
CinCGAN |  CVPRW-2018 [99] |  Domain translation-based |  Unsupervised learning, Cycle-in-cycle network structure, GAN
MCinCGAN |  TIP-2020 [100] |  Domain translation-based |  Unsupervised learning, Multiple cycle-in-cycle network structure, GAN
DDGAN |  CVPRW-2020 [101] |  Domain translation-based |  Unsupervised learning, Unknown degradation, Cycle-in-cycle GAN, Domain discriminator
PSUSR |  CVPR-2020 [102] |  Domain translation-based |  Unpaired kernel/noise correction network, Pseudo-paired SR network, GAN
USISResNet |  CVPRW-2020 [103] |  Domain translation-based |  Unsupervised learning, GAN, Mean Opinion Score-based loss function
DNSR |  arXiv-2018 [104] |  Domain translation-based |  Unsupervised degradation learning, Bidirectional structural consistency, Bi-cycle network
GAN-CIRCLE |  TMI-2020 [105] |  Domain translation-based |  Computed tomography, CycleGAN
DSGAN |  ICCVW-2019 [106] |  Domain translation-based |  Frequency separation, Unsupervised learning, GAN
SRResCGAN |  CVPRW-2020 [107] |  Domain translation-based |  Image observation model, Domain learning, GAN
RBSR |  WACV-2021 [108] |  Domain translation-based |  Bicubically down-sampled images, GAN
ULSR |  ICCVW-2019 [109] |  Domain translation-based |  Unsupervised learning, Unpaired data, GAN
ISPUSR |  CVPRW-2020 [110] |  Domain translation-based |  Unsupervised image translation, Supervised SR, Collaborative training
ZSSR |  CVPR-2018 [111] |  Self-learning-based |  Zero-shot, Internal recurrence, Deep internal learning, Image-specific CNN
DBPI |  SPL-2020 [112] |  Self-learning-based |  Unified internal learning, Downscaling/SR network, Dual back-projection loss
DualSR |  WACV-2021 [113] |  Self-learning-based |  Zero-shot, Dual-path architecture, GAN, Masked interpolation loss
MZSR |  CVPR-2020 [114] |  Self-learning-based |  Meta-transfer learning, Large-scale training, External and internal information
MLSR |  ECCV-2020 [115] |  Self-learning-based |  Meta-learning, Patch-recurrence property

###  IV-A Degradation Modeling-based Methods

Compared with the SR of synthetic LR images, one of the main challenges for
RSISR is that the degradation model (_i.e._ , the $D(\cdot)$ in Eq. (1)) is
unknown. Generally, the degradation parameter set is necessary to design the
objective function for reconstruction-based SISR methods derived from the
_Maximum a Posteriori_ (MAP) estimation. For learning-based methods that aim
to obtain the mapping from LR images to their HR counterparts, the degradation
parameter set is crucial for building training datasets. Therefore, as
presented in Fig. 4, an intuitive way is to estimate the degradation parameter
of the LR input prior to super-resolving or to iteratively optimize the
degradation parameter and the super-resolved image. Eq. (3)) shows the
commonly used degradation model, in which the blur kernel $\bf{B}$ and noise
$\bf{n}$ are unknowns. The exploration by Efrat _et al._ [141] shows that an
accurate blur model is more important to the success of SR than a
sophisticated image prior, in particular for real-world images. As highlighted
in [141], most of the existing degradation modeling-based methods focus on the
recovery of the blur kernel. 151515In some cases, the noise is not explicitly
characterized. Natural image priors are generally incorporated to suppress
noise.

Figure 4: The general idea of degradation modeling-based methods.

RSISR is a severely ill-posed problem. Shao _et al._ [88] formulate this kind
of task as an energy minimization problem and jointly optimize the
nonparametric blur kernel and the intermediate super-resolved image.
Mathematically, the objective function for blur kernel estimation is defined
as

$\mathop{\min}\limits_{{\bf{X}},{\bf{b}}}{\kern
2.0pt}\lambda\left\|{{\bf{SBX}}-{\bf{Y}}}\right\|_{2}^{2}+\Re({\bf{X}},{\bf{b}})+\eta\left\|{{\bf{BX}}-{\bf{\tilde{X}}}}\right\|_{2}^{2}$
(7)

where $\lambda$ and $\eta$ are parameters for balancing different terms.
${\bf{\tilde{X}}}$ denotes the super-resolved image generated by a non-blind
learning-based method, and $\bf{b}$ denotes the blur kernel corresponding to
the blur matrix $\bf{B}$. $\Re({\bf{X}},{\bf{b}})$ represents the direct
bi-${l_{0}}$-${l_{2}}$-norm regularization term for the intermediate super-
resolved image $\bf{X}$ and the blur kernel $\bf{b}$, which is beneficial to
the accurate estimation of $\bf{b}$. After iteratively optimizing the $\bf{X}$
and $\bf{b}$ in Eq. 7, the estimated blur kernel $\bf{\hat{b}}$ can be
combined with non-blind SR methods to produce an HR estimate. In [89], Shao
_et al._ introduce the ${l_{\alpha}}$-norm-based adaptive heavy-tailed image
prior to further improve the above approach. The studies in [88] and [89]
demonstrate that, with the aid of effective constraints, the iterative
optimization of the blur kernel and the super-resolved image is good for the
accuracy of blur kernel estimation.

Different from the above numerical optimization-based approaches [88, 89], the
blur kernel and the super-resolved image can also be jointly optimized using
deep neural networks [90, 91, 92]. Generally, the mismatch of blur kernels
would result in artifacts in the super-resolved image, _e.g._ , over-smoothing
or over-sharpening. Based on this observation, Gu _et al._ [90] and Cornillere
_et al._ [91] propose to progressively correct the inaccurate kernel according
to the quality of the super-resolved image. 161616The source code of IKC [90]
is available at https://github.com/yuanjunchai/IKC More specifically, they
develop degradation-aware SR networks to produce HR images, in which the blur
kernel is utilized as auxiliary information for SR. Meanwhile, corresponding
deep neural networks are designed to correct the kernel with the guidance of
the intermediate SR result. Unlike previous approaches [90, 91] that combine
two or more networks, Huang _et al._ [92] develop a deep alternating network
(DAN) for RSISR, in which the iterative optimization process between the
super-resolved image and the blur kernel is unfolded to an end-to-end
trainable network. 171717The source code of DAN [92] is available at
https://github.com/greatlog/DAN DAN [92] consists of a chain of alternately
stacked restorers and estimators, which are responsible for the restoration of
the HR image and the estimation of the blur kernel, respectively. For the
above methods, the super-resolved image and the blur kernel corresponding to
the LR input are iteratively refined. After joint optimization, the refined
blur kernel and the super-resolved image are supposed to be more accurate.

Previous studies have shown that natural image priors such as patch recurrence
property are useful for degradation modeling. In [93], Michaeli _et al._ point
out that the Point Spread Function (PSF) is not the optimal blur kernel, and
they further propose to obtain the principled MAP estimate of the blur kernel
via maximizing the similarity of recurring image patches across scales of the
LR input. 181818The project homepage of NBSR [93] is at
http://www.wisdom.weizmann.ac.il/~vision/BlindSR.html The estimated blur
kernel can be used to degrade the LR input or natural HR images artificially.
In this way, the blur kernel estimation approach is smoothly plugged into both
self-example-based and external-example-based SR approaches [47, 54]. The
results in [93] show that the accuracy enhancement of blur kernel estimation
leads to an obvious improvement of SR performance on synthetic as well as
real-world images.

Degradation modeling is also vitally important for deep learning-based SR
approaches. Deep convolutional neural networks (CNN)-based SISR approaches
usually achieve state-of-the-art (SOTA) results on standard benchmarks.
Nevertheless, their performance is limited when applied to real-world images.
The main reason is that the kernel (_e.g._ , “bicubic” kernel) used to
generate training data is significantly different from the blur in a real
scenario. To address this problem, some recently presented deep learning-based
RSISR methods [94, 95, 96, 97, 98] adopt the pre-estimated degradation
parameters to generate samples for model training. For example, inspired by
[93], Bell-Kligler _et al._ [94] develop an image-specific internal-GAN
(_i.e._ , KernelGAN) to learn the internal distribution of patches. 191919The
source code of KernelGAN [94] is available at
https://github.com/sefibk/KernelGAN The KernelGAN [94] is trained solely using
the LR test image, making its discriminator unable to differentiate the patch
distribution of the original LR input from that of the degraded version of the
LR image produced by the generator. After the joint training with the
discriminator, the generator can well characterize the degradation process
with an image-specific kernel. Then, the LR test image and its degraded
version generated by the generator form paired data for SR model training.
Bulat _et al._ [95] train a generative adversarial network (GAN)-based
degradation model from unpaired HR and LR face images, and then use the
learned network to generated image pairs for SR network training. 202020The
source code of FISR [95] is available at https://github.com/jingyang2017/Face-
and-Image-super-resolution Zhou _et al._ [96] propose to obtain a group of
realistic blur kernels from real-world photographs and a GAN is trained on
them to augment the pool of realistic blur kernels. 212121The source code of
KMSR [96] is available at https://github.com/IVRL/Kernel-Modeling-Super-
Resolution With the augmented kernel pool, more realistic and diverse LR-HR
image pairs can be constructed to train the SR model. Analogously, Xiao _et
al._ [97] model spatially variant degradation via learning a set of basis blur
kernels and corresponding pixel-wise weights from real-world image pairs. The
learned realistic degradation model is then used to generate pseudo-realistic
LR-HR image pairs. More recently, Ji _et al._ [98] take this idea one step
further and develop an effective degradation framework using various realistic
blur kernels and noise distributions, winning the _NTIRE 2020 Challenge on
Real-World Image Super-Resolution_ [122]. 222222The source code of RSRKN [98]
is available at https://github.com/jixiaozhong/RealSR The outstanding
performance of degradation modeling-based RSISR methods demonstrates that
degradation modeling is meaningful and this kind of approach is a feasible
solution to the SR of real-world images.

Figure 5: Existing solutions to construct realistic LR-HR image pairs.

###  IV-B Image Pairs-based Methods

Although paired LR-HR training data can be synthesized from high-quality
images according to pre-defined degradation models [96, 98], deriving explicit
realistic degradation models from real-world images is challenging. To deal
with this problem, researchers [80, 83, 82, 84, 79, 81, 85] propose to
directly collect the images of the same scene with different resolutions,
which are used to construct realistic LR-HR image pairs for RSISR model
training. Overall, as illustrated in Fig. 5, currently there are three main
ways to collect real-world images for dataset building, including the focal
length adjusting-based approach [80, 83, 82, 84, 81], the hardware binning-
based approach [79], and the beam splitter-based approach [85]. The
representative realistic datasets built with the above image collection
approaches are described in Section III-A. Therefore, this section focuses on
the RSISR models developed on these real-world datasets. Intuitively, given
LR-HR image pairs, nearly all existing supervised SR methods (_e.g._ , SRCNN
[67], VDSR [68], EDSR [70], RDN [74], SRGAN [69], _etc_.) can be adopted to
learn the mapping from LR images to their HR counterparts. The mapping learned
from realistic datasets is supposed to apply to the SR of real-world images.
However, in fact challenges remain.

For instance, the degradation kernels of real-world images are generally non-
uniform, varying with the depth in a real scene. Thus, training an SR model
that treats all pixels the same way as most previous deep CNN-based SR
approaches may not be the optimal solution. For this problem, Cai _et al._
[80] propose the LP-KPN, which combines the Laplacian pyramid with the pixel-
wise kernel prediction network (KPN), achieving good SR performance and high
efficiency. 232323The source code of LP-KPN [80] is available at
https://github.com/csjcai/RealSR Another challenge that cannot be ignored is
the misalignment between LR and HR image pairs in the collected realistic
datasets. Although image registration is performed to align realistic image
pairs, misalignment is unavoidable. As a consequence of misalignment, blurring
artifacts may be introduced in the reconstructed HR images when using these
datasets to train the SR model with pixel-to-pixel losses (_e.g._ , $l_{1}$
and $l_{2}$). Inspired by the Contextual Loss [142] and the edge-preserving
bilateral filter [143], Zhang _et al._ [83] propose the Contextual Bilateral
loss (CoBi) to resolve this issue. 242424The source code of ZLLZ [83] is
available at https://github.com/ceciliavision/zoom-learn-zoom CoBi integrates
the pixel-level information and spatial pixel coordinates to measure image
similarity. Moreover, two spaces including RGB image patches and pre-trained
perceptual-features (_e.g._ , VGG-19 [144]) are jointly considered in CoBi to
improve performance further. It is demonstrated that CoBi is robust to the
mild misalignment in the realistic image pairs for supervised SR model
training. Considering that pixel-wise losses are generally more focused on
smoothing flat regions and sharping edges while neglecting the recovery of
realistic details of textures to some extent, Wei _et al._ [81] develop a
Component Divide-and-Conquer (CDC) SR model for real-world images. 252525The
source code of CDC [81] is available at https://github.com/xiezw5/Component-
Divide-and-Conquer-for-Real-World-Image-Super-Resolution More specifically,
the flat, edge, and corner components are first predicted by three component-
attentive blocks respectively in CDC and then they are aggregated to produce
the final SR image based on the learned component-attentive maps. To achieve
this goal, a gradient-weighted loss that can adapt the model training to the
reconstruction difficulties of different image components is applied. The
results on RealSR [80] and DRealSR [81] prove the superiority and
generalization capability of CDC [81]. Given a dataset containing raw and
color images, learning a mapping from LR raw images to HR color images using
deep neural networks is an intuitive approach to exploit raw images for SR.
However, one raw image could correspond to a set of color images because it
does not have the information for the processes (_e.g._ , color correction)
within the image signal processing system, making the above naive way do not
work well. To address this problem, Xu _et al._ [86, 87] design a two-branch
CNN which jointly exploits the LR raw data and corresponding LR color image to
recover fine structures and high-fidelity color appearances.

In addition to the above challenges shared by almost all real-world images,
the SR of certain kinds of images (_e.g._ , text images, remote sensing
images, and medical images) usually has a particularity. Therefore, specific
SR models should be designed for these scenarios. For example, the TSRN
developed by Wang _et al._ [84] is an SR network for real scene text images.
262626The source code of TSRN [84] is available at
https://github.com/JasonBoy1/TextZoom To leverage the strong sequential
characteristics of text images, the Bi-directional LSTM (BLSTM) mechanism is
added to basic residual blocks. In order to address the misalignment problem
in the realistic text image dataset TextZoom, TSRN [84] introduces a spatial
transform network-based central alignment module in the front of the network.
Furthermore, aiming at enhancing the shape boundary of characters, a gradient
prior loss is combined with the $l_{2}$ loss to train TSRN. It is demonstrated
that the SR of real-world text images using TSRN does increase the recognition
accuracy. Predictably, in some cases, SR can also benefit other computer
vision tasks, _e.g._ , objection detection [5] and semantic segmentation [4].

###  IV-C Domain Translation-based Methods

As introduced in Sections III-A and IV-B, it is hard to obtain real-world
datasets with well-aligned LR-HR image pairs. More often than not, what we
have are only LR images for model training in practical applications. Or
better yet, a set of HR images are available for reference besides LR training
images, but there is no one-to-one correspondence between LR and HR images.
The supervised approaches no longer apply in these cases due to the lack of
paired samples. Previous studies [99, 100, 101, 102, 103, 104, 105, 106, 107,
107, 108, 109, 110] demonstrate that domain translation is a feasible solution
to deal with this problem. For this kind of RSISR approach, real-world LR
images, synthetic LR images (also known as clean LR images or ideal LR
images), and HR images are thought to be in different domains. Consequently,
the SR of real-world images converts to the translation from the real-world LR
image domain (RLRD) to the HR image domain (HRD). Overall, as presented in
Fig. 6, there are two main ways to cross RLRD to HRD, _i.e._ , the two-stage
and the one-stage approaches, and the most notable difference lies in whether
the synthetic LR image domain (SLRD) is used as a relay station.

Figure 6: The general idea of domain translation-based methods.

Most existing SISR approaches are trained using synthetic data, thereby
achieving excellent performance on clean LR images. As is known to all, there
is a noticeable domain gap between SLRD and RLRD, causing the degradation of
SR accuracy on real-world images. Intuitively, one can mitigate this
performance degradation via reducing the domain gap, which is how the two-
stage domain translation-based RSISR approaches work. As shown in Fig. 6, the
two-stage approaches generally include two main steps, _i.e._ , domain
translation and SR. Usually, generic SR methods are adopted in the SR
reconstruction phase, so as to benefit from pre-trained SR models learned from
large-scale datasets. The Cycle-in-Cycle GAN (CinCGAN) developed by Yuan _et
al._ [99] is a representative of this kind of RSISR method. More specifically,
CinCGAN [99] first uses a domain translation network to map realistic LR
images in RLRD into the SLRD, and then a pre-trained deep SR network with the
ideal degradation assumption is stacked to upscale the translation result to
the desired size. Finally, the domain translation and SR networks are end-to-
end fine-tuned. Inspired by CycleGAN [145], the domain translation network in
CinCGAN [99] is trained to map realistic LR inputs to synthetic LR images
(_i.e._ , “bicubic”-downsampled images) using unpaired training data, which
usually can suppress the artifacts such as noise in real-world LR images to
make them more suitable for the following SR network. Results show that the
CinCGAN [99] trained with unpaired data achieves comparable SR performance as
supervised methods. In MCinCGAN [100], the CinCGAN [99] is improved via
introducing a progressive multi-cycle framework for large-scale upsampling and
a new constraint to suppress the color fluctuation in training. For more
stable model training and better SR performance, the DDGAN developed by Kim
_et al._ [101] further combines the pixel-wise loss, the VGG feature loss, and
the SSIM loss to measure similarity. 272727The source code of DDGAN [101] is
available at https://github.com/GT-KIM/unsupervised-super-resolution-domain-
discriminator Meanwhile, a domain discriminator that takes the noise, texture,
and color into consideration simultaneously is proposed to make the generated
image more consistent with the target domain distribution. More recently,
Maeda _et al._ [102] propose an end-to-end trainable framework UISRPS to
jointly optimize the domain translation network and the SR network, achieving
excellent SR results on real-world face images and aerial images. Benefiting
the architecture, it is convenient to integrate existing SR networks and
pixel-wise loss functions into UISRPS [102].

Different from the two-stage methods mentioned above, the one-stage domain
translation-based RSISR frameworks aim to produce a super-resolved image
directly from the real-world LR input, as presented in Fig. 6. How to learn
the translation mapping from RLRD to HRD without real LR-HR image pairs is the
crucial issue. Prajapati _et al._ [103] propose to train a GAN-based network
USISRNet to upsample real-world images. 282828The source code of USISResNet
[103] is available at https://github.com/kalpeshjp89/USISResNet Since only
unpaired LR and HR training images are available, USISRNet is optimized
through unsupervised learning. Beyond the standard GAN losses, a pixel-wise
content loss, a Total-Variation loss, and a quality assessment loss are
combined to optimize USISRNet. The content loss makes the SR result be not far
from the bicubically upsampled version, thus preserving the primary content of
the LR image. The Total-Variation loss is integrated to suppress the noise and
artifacts. For better perceptual quality of the super-resolved image, the
learned Mean Opinion Score is used to construct the quality assessment loss.
Thanks to the combination of multiple losses, USISRNet [103] achieves good
generalization capacity. Inspired by CycleGAN [145], some researchers propose
to learn the direct relationship between RLRD to HRD using cycle consistency-
constrained GANs [104, 105]. 292929The source code of GAN-CIRCLE [105] is
available at https://github.com/charlesyou999648/GAN-CIRCLE Given an LR image,
the LR-to-HR generator (_i.e._ , RLRD to HRD) is trained to reconstruct a
vivid HR image that can be returned to the LR input by the corresponding HR-
to-LR generator (_i.e._ , HRD to RLRD). Conversely, an HR image should be well
recovered by the LR-to-HR generator from its downsampled version produced by
the HR-to-LR generator. After the joint training of the two generators and
corresponding discriminators, the LR-to-HR generator models the direct mapping
from RLRD to HRD and it is used to reconstruct HR images from realistic LR
images.

Considering that learning the end-to-end translation between RLRD to HRD from
unpaired LR-HR data is challenging, part of the one-stage domain translation-
based RSISR methods [109, 106, 108, 110, 107] also use the synthetic LR image
as a bridge in the training phase. Given unpaired realistic LR and HR images,
Fritsche _et al._ [106] first bicubically downsample HR images. Bicubically
downsampled results are then translated into the realistic domain to make them
follow real scene characteristics, using a standard GAN-based domain
translation network trained on the bicubically downsampled images and
realistic LR images in an unsupervised fashion. Taking the pseudo-realistic LR
images and corresponding HR images as training sample pairs, the ESRGAN [146]
is trained for upsampling in a supervised manner. In order to generate images
well matching the target distribution, both domain translation and SR networks
are optimized with frequency separation-based loss functions. 303030The source
code of DSGAN [106] is available at https://github.com/ManuelFritsche/real-
world-sr More specifically, the color loss, the texture loss, and the
perceptual loss are employed to the low-frequency component, the high-
frequency component, and the whole image, respectively. Note that only the SR
model is needed to upscale real-world images in the testing phase because it
is trained on the image pairs that follow real-world image distribution. On
this basis, recently Umer _et al._ [107] improve the GAN-based SR model
following the real-world image observation model, thus exploiting the powerful
regularization and optimization techniques simultaneously. 313131The source
code of SRResCGAN [107] is available at https://github.com/RaoUmer/SRResCGAN
Rad _et al._ [108] convert realistic LR images to bicubic look-alike images
based on their copying mechanism and bicubic perceptual loss. Different from
the above works [106, 107, 108] which use a single direction domain
translation model, Lugmayr _et al._ [109] and Chen _et al._[110] propose to
train a bi-directional domain translation model with the cycle consistency
constraints for better robustness.

Overall, domain translation is an effective way to reduce the domain gap
between synthetic and realistic data, thus improving the generalization
capability of SR models on ever-changing real-world images. In contrast, the
two-stage domain translation-based RSISR approaches can integrate synthetic
data-trained SR models more elegantly, while the one-stage methods generally
have lower complexity in the testing phase.

Figure 7: The general idea of self-learning-based SR methods. Note that the
pre-trained model is optional.

###  IV-D Self-Learning-based Methods

Most existing RSISR methods use external dataset (_i.e._ , paired or unpaired
training data) to train SR models. Therefore, the SR performance is tightly
bound to the consistency between testing data and training data. However,
real-world images do not always obey the characteristics of training data. In
order to reduce the impact of the training–testing discrepancy on SR
performance, researchers propose to exploit the internal information of the LR
input to learn image-specific SR model as shown in Fig. 7.

The “Zero-Shot” SR (ZSSR) developed by Shocher _et al._ [111] is one of the
representatives. 323232The source code of ZSSR [111] is available at
https://github.com/assafshocher/ZSSR The self-supervised approach ZSSR [111]
is based on the cross-scale internal recurrence of information, which is a
common property of natural images. More specifically, an eight-layer CNN is
trained to model image-specific LR-HR relations in the testing phase, using
the example pairs extracted from the LR test image and its degraded version.
In consideration of the insufficiency of training data (the test image only),
data augmentation is adopted when extracting image-specific LR-HR pairs. Since
ZSSR [111] can adapt itself to different testing images, it achieves excellent
SR performance on real-world images, whose degradation process is non-ideal
and unknown. Again based on the cross-scale recurrence property, Bell-Kligler
_et al._ [94] propose to train an image-specific GAN (KernelGAN) to model the
degradation process (_i.e._ , the blur kernel) of the input.19 Therefore, a
fully self-supervised image-specific RSISR framework can be achieved when the
blur kernel estimation module KernelGAN [94] is plugged into the
reconstruction module ZSSR [111]. To jointly train the image-specific
degradation and SR networks, Kim _et al._ [112] design a unified internal
learning-based SR framework DBPI, consisting of an SR network and a
downscaling network. 333333The source code of DBPI [112] is available at
https://github.com/prote376/DBPI-BlindSR In the self-supervised training phase
of DBPI, the SR network is optimized to reconstruct the LR input image from
the its downscaled version produced by the downscaling network. Meanwhile, the
downscaling network is trained to recover the LR input image from its super-
resolved version generated by the SR network. Similarly, Emad _et al._ [113]
propose the DualSR that jointly optimizes an image-specific downsampler and
corresponding upsampler. More specifically, DualSR [113] is trained with the
cycle-consistency loss, the masked interpolation loss, and the adversarial
loss using the patches from the test image. Results in [112, 113] show that
the complementary training of the image-specific degradation and SR networks
is beneficial to the reconstruction performance.

Although self-learning-based RSISR approaches such as ZSSR [111], KernelGAN
[94], and DBPI [112] can be easily adapted to LR input images, they generally
have two main shortcomings due to the self-supervised training strategy.
First, the optimization of SR models only utilizes the internal information of
the LR input, while a great deal of external information is neglected. Second,
these methods are usually time-consuming in the testing phase because of
online training. To overcome these disadvantages, meta-learning is introduced
into recent self-learning-based SR methods [114, 115]. Based on ZSSR [111],
Soh _et al._ [114] present the meta-transfer learning for zero-shot SR (MZSR),
which consists of three steps, _i.e._ , large-scale training, meta-transfer
learning, and meta-test. 343434The source code of MZSR [114] is available at
https://www.github.com/JWSoh/MZSR In order to ease the training of the SR
network and the meta-learning, the large-scale training step first trains an
eight-layer SR network with the pixel-wise $l_{1}$ loss on the large-scale
dataset DIV2K [128]. The meta-transfer learning process aims to find a generic
initial point for internal learning following the Model-Agnostic Meta-Learning
[147], making the model can be quickly adapted to new image conditions within
a few gradient updates. In the meta-test phase, the input test image is first
degraded to produce example pairs for model parameter update, and then it is
fed into the updated model to generate an SR result. Thanks to the meta-
transfer learning strategy, MZSR [114] achieves competitive performance in
terms of both the quality of the super-resolved image and the running time. In
[115], Park _et al._ also propose to improve the performance of SOTA SR
networks such as RCAN [73] using the meta-learning strategy, without changing
the original architectures. 353535The source code of MLSR [115] is available
at https://github.com/parkseobin/MLSR On the whole, meta-learning-based SR
approaches have strengths in reconstruction quality, generalization
capability, and processing efficiency.

(a) Ground-truth (b) Bicubic (c) ZSSR [111] (d) KernelGAN [94]

(e) MZSR (1) [114] (f) DBPI [112] (g) DAN [90] (h) SRResCGAN [107]

Figure 8: Super-resolution results comparison ($\times 2$) on the image taken
from DIV2KRK [94].

(a) Ground-truth (b) Bicubic (c) ZSSR [111] (d) KernelGAN [94]

(e) DBPI [112] (f) DAN [92] (g) IKC [92] (h) SRResCGAN [107]

Figure 9: Super-resolution results comparison ($\times 4$) on the image taken
from DIV2KRK [94].

(a) Ground-truth (b) Bicubic (c) ZSSR [111] (d) KernelGAN [94]

(e) MZSR (1) [114] (f) DBPI [112] (g) DAN [90] (h) SRResCGAN [107]

Figure 10: Super-resolution results comparison ($\times 2$) on the image taken
from RealSR [80].

(a) Ground-truth (b) Bicubic (c) ZSSR [111] (d) KernelGAN [94]

(e) DBPI [112] (f) DAN [92] (g) IKC [92] (h) SRResCGAN [107]

Figure 11: Super-resolution results comparison ($\times 4$) on the image taken
from RealSR [80].  TABLE IV: The performance of representative RSISR
algorithms on DIV2KRK [94] and RealSR [80] datasets.

_Method_ | _Scale_ | PSNR $\uparrow$ (dB) | SSIM $\uparrow$ [135] | IFC $\uparrow$ [136] | NIQE $\downarrow$ [138] | PIQE $\downarrow$ [139] | NRQM $\uparrow$ [116] | LPIPS $\downarrow$ [137]
---|---|---|---|---|---|---|---|---
_Performance on the synthetic dataset DIV2KRK[94]_
Bicubic | $\times 2$ | 27.24 | 0.7846 | 3.022 | 5.196 | 79.90 | 3.273 | 0.3631
$\times 4$ | 23.89 | 0.6478 | 1.092 | 6.310 | 95.26 | 3.043 | 0.5645
ZSSR [111] | $\times 2$ | 27.51 | 0.7925 | 3.000 | 5.068 | 80.06 | 3.435 | 0.3477
$\times 4$ | 24.05 | 0.6550 | 1.132 | 5.824 | 90.29 | 3.109 | 0.5257
KernelGAN [94] | $\times 2$ | 28.84 | 0.8379 | 3.720 | 5.099 | 72.17 | 3.920 | 0.2929
$\times 4$ | 24.76 | 0.6799 | 1.245 | 5.886 | 88.72 | 3.610 | 0.4980
MZSR (1) [114] | $\times 2$ | 26.69 | 0.7889 | 2.515 | 5.627 | 35.97 | 4.112 | 0.2630
DBPI [112] | $\times 2$ | 29.55 | 0.8657 | 4.084 | 5.239 | 50.61 | 4.844 | 0.2641
$\times 4$ | 24.92 | 0.7035 | 1.385 | 5.163 | 70.74 | 4.980 | 0.4039
DAN [92] | $\times 2$ | 31.06 | 0.8848 | 5.076 | 4.041 | 56.14 | 3.519 | 0.1667
$\times 4$ | 26.07 | 0.7305 | 1.758 | 5.570 | 82.28 | 3.157 | 0.4045
IKC [90] | $\times 4$ | 25.41 | 0.7255 | 1.691 | 5.140 | 79.72 | 3.869 | 0.3977
SRResCGAN [107] | $\times 2$ | 26.15 | 0.7486 | 1.808 | 3.717 | 35.88 | 4.976 | 0.2463
$\times 4$ | 24.00 | 0.6497 | 1.024 | 5.038 | 74.27 | 3.054 | 0.5054
_Performance on the real-world dataset RealSR[80]_
Bicubic | $\times 2$ | 30.27 | 0.8736 | 1.921 | 5.569 | 83.90 | 2.731 | 0.2095
$\times 4$ | 25.74 | 0.7413 | 0.890 | 6.228 | 92.53 | 2.802 | 0.4666
ZSSR [111] | $\times 2$ | 30.56 | 0.8786 | 1.949 | 5.376 | 81.09 | 2.749 | 0.1756
$\times 4$ | 25.83 | 0.7434 | 0.897 | 4.971 | 83.01 | 2.797 | 0.3503
KernelGAN [94] | $\times 2$ | 30.24 | 0.8907 | 2.106 | 5.384 | 77.10 | 2.769 | 0.1338
$\times 4$ | 24.09 | 0.7243 | 0.897 | 4.918 | 78.96 | 3.559 | 0.2981
MZSR (1) [114] | $\times 2$ | 27.96 | 0.8160 | 1.520 | 5.887 | 38.83 | 2.868 | 0.2105
DBPI [112] | $\times 2$ | 27.86 | 0.8285 | 1.876 | 5.698 | 54.87 | 2.995 | 0.1777
$\times 4$ | 22.36 | 0.6562 | 0.851 | 5.640 | 70.58 | 5.056 | 0.3106
DAN [92] | $\times 2$ | 30.63 | 0.8815 | 1.959 | 4.387 | 68.68 | 2.745 | 0.1314
$\times 4$ | 26.20 | 0.7598 | 0.966 | 6.096 | 90.57 | 2.834 | 0.4095
IKC [90] | $\times 4$ | 25.60 | 0.7488 | 0.944 | 4.845 | 82.75 | 2.927 | 0.3188
SRResCGAN [107] | $\times 2$ | 26.26 | 0.7983 | 1.573 | 3.786 | 39.40 | 4.129 | 0.2090
$\times 4$ | 25.84 | 0.7459 | 0.900 | 5.009 | 74.45 | 2.795 | 0.3746

##  V Comparisons among State-of-the-Arts

In this section, we compare representative RSISR methods on benchmark
datasets. More specifically, the selected competitors are ZSSR [111],
KernelGAN [94], MZSR [114], DBPI [112], DAN [92], IKC [90], and SRResCGAN
[107], covering multiple kinds of approaches mentioned in Section IV. For
these SR approaches, we use the official models provided by authors in the
comparison, and the upsampling factor is set to $2$ or $4$. Two benchmark
datasets are tested, including the DIV2KRK [94] and RealSR [80]. As introduced
in Section III-A, DIV2KRK [94] is a synthetic dataset with random kernels and
RealSR [80] is a realistic dataset collected by adjusting the focal length.
363636There are two subsets for testing in RealSR [80], _i.e._ , Canon and
Nikon. We only use the 50 testing images in Canon in this comparison.  In
addition to the subjective quality comparison presented in Figs. 8-11, the SR
results by different methods are objectively evaluated in Table IV on full-
reference and no-reference image quality assessment metrics, including PSNR,
SSIM [135], IFC [136], LPIPS [137], NIQE [138], PIQE [139], and NRQM [116].
Beyond reconstruction accuracy, the model size and execution speed are also
significant for SR algorithms. Therefore, the number of parameters and running
time are also presented in Fig. 12. It is worth noting that the results shown
in this section may be different from those in the original paper due to
different settings of test environments, hyper-parameters, _etc_. The aim of
these comparisons is not to find a winner in terms of accuracy or efficiency,
but to indicate the current state of the research on RSISR. In fact, as is
known to all, it is not easy to make a completely fair comparison among these
competitors due to the complexity of settings. As far as we know, in
particular, there is no uniform or universally accepted settings yet for the
comparison of RSISR models.

Figure 12: The number of parameters and the average running time on the images
taken from RealSR [80].

We can make the following observations from the comparison results on visual
quality, objective quality, and complexity. (i) Compared with conventional
interpolation (_e.g._ , “Bicubic”), SR is undoubtedly a more effective way to
obtain HR images. (ii) Overall, there is an obvious gap between the super-
resolved image and the corresponding ground-truth in terms of visual effects,
especially for texture and edge regions. For example, some SR results suffer
from over-smoothing or over-sharpening artifacts. (iii) There are some
differences between subjective and objective assessment results. Moreover, the
scores by different objective assessment metrics may not be necessarily
consistent with each other. (iv) In general, self-supervised learning-based SR
approaches have fewer parameters than the SR models trained on large-scale
datasets, but take longer to produce upsampled images.

##  VI Current Challenges and Future Directions

The numerous studies reviewed in Sections III and IV demonstrate the great
progress of the research on RSISR. In fact, however, there are still problems
requiring further exploration. We discuss some of the challenges and promising
directions for RSISR in this section.

###  VI-A Image Datasets

Overall, the data for SR model training may be equally as crucial as the SR
techniques for the study on RSISR, especially for deep learning-based
solutions. Several realistic datasets have been constructed in the past few
years, significantly boosting the reconstruction performance. However,
compared with the datasets for popular computer vision tasks such as
classification and detection, the lack of realistic datasets for RSISR is
still striking. Therefore, it is desired to build larger and more
representative/targeted realistic datasets for RSISR with considering the
limiting factors of imaging resolution in the future. Meanwhile, more accurate
alignment of the images captured from the same scene with different
resolutions is also needed.

###  VI-B SR Algorithms

Although the SR performance on real-world images is getting better and better,
there is still a long way to go before applying RSISR algorithms to practical
applications. First, the LR images captured in real scenarios are likely to
suffer from distinctly different degradations, challenging existing RSISR
algorithms. Therefore, it is necessary to make RSISR models can adapt
themselves to ever-changing real-world images. Second, most existing RSISR
approaches are with a large model (_e.g._ , a deep network), thereby requiring
vast computing resources/time for forward inference and plenty of space for
parameters storage. However, these resources are generally limited in real-
world applications. Hence, how to achieve lightweight design and
implementation of SR models without significant performance degradation is a
primary challenge. Third, it is hard to obtain paired training data, even
unpaired data or relevant HR references in some cases. Thus it is promising to
develop RSISR models that can be run with unpaired training data or even the
LR input only. Moreover, how to leverage fewer models to meet personalized and
multifunctional demands (_e.g._ , the need for arbitrary upscaling factors and
the preference to perceptual quality) of users deserves further investigation.

###  VI-C Evaluation Criteria

The evaluation criteria are vitally crucial for the research on computer
vision tasks. On the one hand, the design of objective functions is generally
guided by the evaluation criteria. For example, the $l_{2}$ loss is prevalent
for image/video restoration tasks because it is highly correlated to the
commonly used quality assessment metric PSNR. On the other hand, evaluation
criteria are needed to make comparisons among different approaches, thus
continuously advancing techniques. As previously mentioned, currently the PSNR
and SSIM are the two of the most popular evaluation metrics for SR. However,
previous studies show that they are unable to measure the visual quality of
super-resolved images accurately. In addition, PSNR and SSIM are full-
reference evaluation metrics that cannot be adopted in practical applications.
Therefore, developing more suitable evaluation criteria for RSISR is a crucial
and urgent research problem. On the whole, task-specific evaluation metrics
are needed. That is, we have to take the goals and characteristics of RSISR
into consideration while developing evaluation criteria. For example, the
common targets include smoothness preserving for flat areas, detail enhancing
for textures, sharpening for edges, _etc_. Correspondingly, an evaluation
criterion that covers these region-differentiated goals is desired. Meanwhile,
the SR is performed generally for better human visual perception. Therefore,
how to develop an automatic model that can measure visual quality more
accurately and conveniently remains a challenge. Furthermore, there are no HR
images for reference in practical applications. Thus the no-reference quality
assessment criteria for super-resolved images have great demands.

##  VII Conclusion

In recent years the super-resolution of real-world images has been getting
increased attention. This paper briefly reviews recent super-resolution
methods for realistic images, including degradation modeling-based algorithms,
image pairs-based algorithms, domain translation-based algorithms, and self-
learning-based algorithms. Meanwhile, we summarize the commonly used datasets
and assessment metrics for RSISR models training and evaluation. Moreover,
although some progress has been made on RSISR in the past few years, we point
out that there are still challenges to be further addressed, _e.g._ ,
realistic datasets for model training and testing, specific models for real-
world image super-resolution and reconstruction performance evaluation. These
unsolved problems also indicate the promising directions for future
exploration. We expect that this review can give a better understanding of
existing studies for researchers, and also hope that it can attract more
attention to advance the progress and application of real-world image super-
resolution techniques.

## References

  * [1] D. Dai, Y. Wang, Y. Chen, and L. Van Gool, “Is image super-resolution helpful for other vision tasks?” in _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2016, pp. 1–9.
  * [2] S. Lei, Z. Shi, X. Wu, B. Pan, X. Xu, and H. Hao, “Simultaneous super-resolution and segmentation for remote sensing images,” in _IEEE International Geoscience and Remote Sensing Symposium (IGARSS)_ , 2019, pp. 3121–3124.
  * [3] Z. Guo, G. Wu, X. Song, W. Yuan, Q. Chen, H. Zhang, X. Shi, M. Xu, Y. Xu, R. Shibasaki _et al._ , “Super-resolution integrated building semantic segmentation for multi-source remote sensing imagery,” _IEEE Access_ , vol. 7, pp. 99 381–99 397, 2019.
  * [4] L. Wang, D. Li, Y. Zhu, L. Tian, and Y. Shan, “Dual super-resolution learning for semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 3774–3783.
  * [5] M. Haris, G. Shakhnarovich, and N. Ukita, “Task-driven super resolution: Object detection in low-resolution images,” _arXiv preprint arXiv:1803.11316_ , 2018.
  * [6] Y. Pang, J. Cao, J. Wang, and J. Han, “JCS-Net: Joint classification and super-resolution network for small-scale pedestrian detection in surveillance images,” _IEEE Transactions on Information Forensics and Security_ , vol. 14, no. 12, pp. 3322–3331, 2019.
  * [7] Y. Zhang, Y. Bai, M. Ding, S. Xu, and B. Ghanem, “KGSnet: key-point-guided super-resolution network for pedestrian detection in the wild,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2020.
  * [8] Z. Wang, S. Chang, Y. Yang, D. Liu, and T. S. Huang, “Studying very low resolution recognition using deep networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 4792–4800.
  * [9] X. Yang, W. Wu, K. Liu, P. W. Kim, A. K. Sangaiah, and G. Jeon, “Long-distance object recognition with image super resolution: A comparative study,” _IEEE Access_ , vol. 6, pp. 13 429–13 438, 2018.
  * [10] T. Suprapto Siadari, M. Han, and H. Yoon, “GSR-MAR: Global super-resolution for person multi-attribute recognition,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW)_ , 2019.
  * [11] S. P. Belekos, N. P. Galatsanos, and A. K. Katsaggelos, “Maximum a posteriori video super-resolution using a new multichannel image prior,” _IEEE Transactions on Image Processing_ , vol. 19, no. 6, pp. 1451–1464, 2010.
  * [12] C. Liu and D. Sun, “On bayesian adaptive video super resolution,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 36, no. 2, pp. 346–360, 2013.
  * [13] K. Li, Y. Zhu, J. Yang, and J. Jiang, “Video super-resolution using an adaptive superpixel-guided auto-regressive model,” _Pattern Recognition_ , vol. 51, pp. 59–71, 2016.
  * [14] A. Kappeler, S. Yoo, Q. Dai, and A. K. Katsaggelos, “Video super-resolution with convolutional neural networks,” _IEEE Transactions on Computational Imaging_ , vol. 2, no. 2, pp. 109–122, 2016.
  * [15] J. Caballero, C. Ledig, A. Aitken, A. Acosta, J. Totz, Z. Wang, and W. Shi, “Real-time video super-resolution with spatio-temporal networks and motion compensation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 4778–4787.
  * [16] Y. Jo, S. Wug Oh, J. Kang, and S. Joo Kim, “Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 3224–3232.
  * [17] A. Lucas, S. Lopez-Tapia, R. Molina, and A. K. Katsaggelos, “Generative adversarial networks and perceptual losses for video super-resolution,” _IEEE Transactions on Image Processing_ , vol. 28, no. 7, pp. 3312–3327, 2019\.
  * [18] M. Haris, G. Shakhnarovich, and N. Ukita, “Recurrent back-projection network for video super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 3897–3906.
  * [19] Y. Tian, Y. Zhang, Y. Fu, and C. Xu, “TDAN: Temporally-deformable alignment network for video super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 3360–3369.
  * [20] S. Dikbas and Y. Altunbasak, “Novel true-motion estimation algorithm and its application to motion-compensated temporal frame interpolation,” _IEEE Transactions on Image Processing_ , vol. 22, no. 8, pp. 2931–2945, 2012.
  * [21] B.-D. Choi, J.-W. Han, C.-S. Kim, and S.-J. Ko, “Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 17, no. 4, pp. 407–416, 2007.
  * [22] S. Niklaus, L. Mai, and F. Liu, “Video frame interpolation via adaptive separable convolution,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 261–270.
  * [23] S. Niklaus and F. Liu, “Context-aware synthesis for video frame interpolation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 1701–1710.
  * [24] W. Bao, W.-S. Lai, C. Ma, X. Zhang, Z. Gao, and M.-H. Yang, “Depth-aware video frame interpolation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 3703–3712.
  * [25] T. Peleg, P. Szekely, D. Sabo, and O. Sendik, “IM-Net for high resolution video frame interpolation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 2398–2407.
  * [26] W. Bao, W.-S. Lai, X. Zhang, Z. Gao, and M.-H. Yang, “MEMC-Net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2019.
  * [27] X. Cheng and Z. Chen, “Video frame interpolation via deformable separable convolution.” in _AAAI_ , 2020, pp. 10 607–10 614.
  * [28] S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” _IEEE Transactions on Image Processing_ , vol. 13, no. 10, pp. 1327–1344, 2004.
  * [29] S. Farsiu, M. Elad, and P. Milanfar, “Multiframe demosaicing and super-resolution of color images,” _IEEE Transactions on Image Processing_ , vol. 15, no. 1, pp. 141–159, 2005.
  * [30] X. Li, Y. Hu, X. Gao, D. Tao, and B. Ning, “A multi-frame image super-resolution method,” _Signal Processing_ , vol. 90, no. 2, pp. 405–414, 2010.
  * [31] Q. Yuan, L. Zhang, and H. Shen, “Multiframe super-resolution employing a spatially weighted total variation model,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 22, no. 3, pp. 379–392, 2011\.
  * [32] L. Yue, H. Shen, Q. Yuan, and L. Zhang, “A locally adaptive l1- l2 norm for multi-frame super-resolution of images with mixed noise and outliers,” _Signal Processing_ , vol. 105, pp. 156–174, 2014.
  * [33] T. Köhler, X. Huang, F. Schebesch, A. Aichert, A. Maier, and J. Hornegger, “Robust multiframe super-resolution employing iteratively re-weighted minimization,” _IEEE Transactions on Computational Imaging_ , vol. 2, no. 1, pp. 42–58, 2016.
  * [34] X. Liu, L. Chen, W. Wang, and J. Zhao, “Robust multi-frame super-resolution based on spatially weighted half-quadratic estimation and adaptive btv regularization,” _IEEE Transactions on Image Processing_ , vol. 27, no. 10, pp. 4971–4986, 2018.
  * [35] A. Laghrib, A. Hadri, A. Hakim, and S. Raghay, “A new multiframe super-resolution based on nonlinear registration and a spatially weighted regularization,” _Information Sciences_ , vol. 493, pp. 34–56, 2019.
  * [36] J. Sun, Z. Xu, and H.-Y. Shum, “Image super-resolution using gradient profile prior,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2008, pp. 1–8.
  * [37] K. Zhang, X. Gao, D. Tao, and X. Li, “Single image super-resolution with non-local means and steering kernel regression,” _IEEE Transactions on Image Processing_ , vol. 21, no. 11, pp. 4544–4556, 2012.
  * [38] W. Dong, L. Zhang, G. Shi, and X. Li, “Nonlocally centralized sparse representation for image restoration,” _IEEE Transactions on Image Processing_ , vol. 22, no. 4, pp. 1620–1630, 2013.
  * [39] V. Papyan and M. Elad, “Multi-scale patch-based image restoration,” _IEEE Transactions on Image Processing_ , vol. 25, no. 1, pp. 249–261, 2015\.
  * [40] J. Jiang, X. Ma, C. Chen, T. Lu, Z. Wang, and J. Ma, “Single image super-resolution via locally regularized anchored neighborhood regression and nonlocal means,” _IEEE Transactions on Multimedia_ , vol. 19, no. 1, pp. 15–26, 2017.
  * [41] C. Ren, X. He, and T. Q. Nguyen, “Single image super-resolution via adaptive high-dimensional non-local total variation and adaptive geometric feature,” _IEEE Transactions on Image Processing_ , vol. 26, no. 1, pp. 90–106, 2017\.
  * [42] H. Chen, X. He, L. Qing, and Q. Teng, “Single image super-resolution via adaptive transform-based nonlocal self-similarity modeling and learning-based gradient regularization,” _IEEE Transactions on Multimedia_ , vol. 19, no. 8, pp. 1702–1717, 2017.
  * [43] K. Chang, X. Zhang, P. L. K. Ding, and B. Li, “Data-adaptive low-rank modeling and external gradient prior for single image super-resolution,” _Signal Processing_ , vol. 161, pp. 36–49, 2019.
  * [44] T. Li, X. Dong, and H. Chen, “Single image super-resolution incorporating example-based gradient profile estimation and weighted adaptive p-norm,” _Neurocomputing_ , vol. 355, pp. 105–120, 2019.
  * [45] J. Li and W. Guan, “Adaptive $l_{q}$-norm constrained general nonlocal self-similarity regularizer based sparse representation for single image super-resolution,” _Information Fusion_ , vol. 53, pp. 88–102, 2020.
  * [46] W. T. Freeman, T. R. Jones, and E. C. Pasztor, “Example-based super-resolution,” _IEEE Computer Graphics and Applications_ , vol. 22, no. 2, pp. 56–65, 2002.
  * [47] D. Glasner, S. Bagon, and M. Irani, “Super-resolution from a single image,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2009, pp. 349–356.
  * [48] Z. Xiong, D. Xu, X. Sun, and F. Wu, “Example-based super-resolution with soft information and decision,” _IEEE Transactions on Multimedia_ , vol. 15, no. 6, pp. 1458–1465, 2013.
  * [49] Y. Zhu, Y. Zhang, and A. L. Yuille, “Single image super-resolution using deformable patches,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014, pp. 2917–2924.
  * [50] J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 5197–5206.
  * [51] T. Li, X. He, Q. Teng, and X. Wu, “Rotation expanded dictionary-based single image super-resolution,” _Neurocomputing_ , vol. 216, pp. 1–17, 2016.
  * [52] J.-J. Huang, T. Liu, P. Luigi Dragotti, and T. Stathaki, “SRHRF+: Self-example enhanced single image super-resolution using hierarchical random forests,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2017, pp. 71–79.
  * [53] J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” _IEEE Transactions on Image Processing_ , vol. 19, no. 11, pp. 2861–2873, 2010.
  * [54] R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in _International Conference on Curves and Surfaces_ , 2010, pp. 711–730.
  * [55] S. Wang, L. Zhang, Y. Liang, and Q. Pan, “Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012, pp. 2216–2223.
  * [56] Z. Zhu, F. Guo, H. Yu, and C. Chen, “Fast single image super-resolution via self-example learning and sparse representation,” _IEEE Transactions on Multimedia_ , vol. 16, no. 8, pp. 2178–2190, 2014.
  * [57] L.-W. Kang, C.-C. Hsu, B. Zhuang, C.-W. Lin, and C.-H. Yeh, “Learning-based joint super-resolution and deblocking for a highly compressed image,” _IEEE Transactions on Multimedia_ , vol. 17, no. 7, pp. 921–934, 2015.
  * [58] X. Li, G. Cao, Y. Zhang, A. Shafique, and P. Fu, “Combining synthesis sparse with analysis sparse for single image super-resolution,” _Signal Processing: Image Communication_ , vol. 83, p. 115805, 2020.
  * [59] B. Li, Y. Zhou, Y. Zhang, and A. Wang, “Depth image super-resolution based on joint sparse coding,” _Pattern Recognition Letters_ , vol. 130, pp. 21–29, 2020.
  * [60] S. Ayas and M. Ekinci, “Single image super resolution using dictionary learning and sparse coding with multi-scale and multi-directional gabor feature representation,” _Information Sciences_ , vol. 512, pp. 1264–1278, 2020.
  * [61] R. Timofte, V. De Smet, and L. Van Gool, “Anchored neighborhood regression for fast example-based super-resolution,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2013, pp. 1920–1927.
  * [62] ——, “A+: Adjusted anchored neighborhood regression for fast super-resolution,” in _Asian Conference on Computer Vision (ACCV)_ , 2014, pp. 111–126.
  * [63] K. Zhang, B. Wang, W. Zuo, H. Zhang, and L. Zhang, “Joint learning of multiple regressors for single image super-resolution,” _IEEE Signal Processing Letters_ , vol. 23, no. 1, pp. 102–106, 2015.
  * [64] E. Agustsson, R. Timofte, and L. Van Gool, “Regressor basis learning for anchored super-resolution,” in _2016 23rd International Conference on Pattern Recognition (ICPR)_ , 2016, pp. 3850–3855.
  * [65] E. Perez-Pellitero, J. Salvador, J. Ruiz-Hidalgo, and B. Rosenhahn, “Antipodally invariant metrics for fast regression-based super-resolution,” _IEEE Transactions on Image Processing_ , vol. 25, no. 6, pp. 2456–2468, 2016\.
  * [66] K. Zhang, Z. Wang, J. Li, X. Gao, and Z. Xiong, “Learning recurrent residual regressors for single image super-resolution,” _Signal Processing_ , vol. 154, pp. 324–337, 2019.
  * [67] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 2, pp. 295–307, 2015.
  * [68] J. Kim, J. Kwon Lee, and K. Mu Lee, “Accurate image super-resolution using very deep convolutional networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 1646–1654.
  * [69] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang _et al._ , “Photo-realistic single image super-resolution using a generative adversarial network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 4681–4690.
  * [70] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2017, pp. 136–144.
  * [71] M. Haris, G. Shakhnarovich, and N. Ukita, “Deep back-projection networks for super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 1664–1673.
  * [72] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Fast and accurate image super-resolution with deep laplacian pyramid networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 11, pp. 2599–2613, 2018.
  * [73] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 286–301.
  * [74] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 2472–2481.
  * [75] T. Dai, J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 11 065–11 074.
  * [76] Y. Guo, J. Chen, J. Wang, Q. Chen, J. Cao, Z. Deng, Y. Xu, and M. Tan, “Closed-loop matters: Dual regression networks for single image super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 5407–5416.
  * [77] J. Liu, W. Zhang, Y. Tang, J. Tang, and G. Wu, “Residual feature aggregation network for image super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [78] X. Zhang, H. Dong, Z. Hu, W.-S. Lai, F. Wang, and M.-H. Yang, “Gated fusion network for degraded image super resolution,” _International Journal of Computer Vision_ , pp. 1–23, 2020.
  * [79] T. Köhler, M. Bätz, F. Naderi, A. Kaup, A. Maier, and C. Riess, “Toward bridging the simulated-to-real gap: Benchmarking super-resolution on real data,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 11, pp. 2944–2959, 2020.
  * [80] J. Cai, H. Zeng, H. Yong, Z. Cao, and L. Zhang, “Toward real-world single image super-resolution: A new benchmark and a new model,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 3086–3095.
  * [81] P. Wei, Z. Xie, H. Lu, Z. Zhan, Q. Ye, W. Zuo, and L. Lin, “Component divide-and-conquer for real-world image super-resolution,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [82] C. Chen, Z. Xiong, X. Tian, Z.-J. Zha, and F. Wu, “Camera lens super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 1652–1660.
  * [83] X. Zhang, Q. Chen, R. Ng, and V. Koltun, “Zoom to learn, learn to zoom,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 3762–3770.
  * [84] W. Wang, E. Xie, X. Liu, W. Wang, D. Liang, C. Shen, and X. Bai, “Scene text image super-resolution in the wild,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [85] H. Reza Vaezi Joze, I. Zharkov, K. Powell, C. Ringler, L. Liang, A. Roulston, M. Lutz, and V. Pradeep, “ImagePairs: Realistic super resolution dataset via beam splitter camera rig,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020, pp. 518–519.
  * [86] X. Xu, Y. Ma, and W. Sun, “Towards real scene super-resolution with raw images,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 1723–1731.
  * [87] X. Xu, Y. Ma, W. Sun, and M.-H. Yang, “Exploiting raw images for real-scene super-resolution,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [88] W.-Z. Shao and M. Elad, “Simple, accurate, and robust nonparametric blind super-resolution,” in _International Conference on Image and Graphics (ICIG)_ , 2015, pp. 333–348.
  * [89] W.-Z. Shao, Q. Ge, L.-Q. Wang, Y.-Z. Lin, H.-S. Deng, and H.-B. Li, “Nonparametric blind super-resolution using adaptive heavy-tailed priors,” _Journal of Mathematical Imaging and Vision_ , vol. 61, no. 6, pp. 885–917, 2019.
  * [90] J. Gu, H. Lu, W. Zuo, and C. Dong, “Blind super-resolution with iterative kernel correction,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 1604–1613.
  * [91] V. Cornillere, A. Djelouah, W. Yifan, O. Sorkine-Hornung, and C. Schroers, “Blind image super-resolution with spatially variant degradations,” _ACM Transactions on Graphics_ , vol. 38, no. 6, pp. 1–13, 2019.
  * [92] Y. Huang, S. Li, L. Wang, T. Tan _et al._ , “Unfolding the alternating optimization for blind super resolution,” in _34th Conference on Neural Information Processing Systems (NeurIPS)_ , 2020.
  * [93] T. Michaeli and M. Irani, “Nonparametric blind super-resolution,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2013, pp. 945–952.
  * [94] S. Bell-Kligler, A. Shocher, and M. Irani, “Blind super-resolution kernel estimation using an internal-gan,” in _33rd Conference on Neural Information Processing Systems (NeurIPS)_ , 2019, pp. 284–293.
  * [95] A. Bulat, J. Yang, and G. Tzimiropoulos, “To learn image super-resolution, use a gan to learn how to do image degradation first,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 185–200.
  * [96] R. Zhou and S. Susstrunk, “Kernel modeling super-resolution on real low-resolution images,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 2433–2443.
  * [97] J. Xiao, H. Yong, and L. Zhang, “Degradation model learning for real-world single image super-resolution,” in _Proceedings of the Asian Conference on Computer Vision (ACCV)_ , 2020.
  * [98] X. Ji, Y. Cao, Y. Tai, C. Wang, J. Li, and F. Huang, “Real-world super-resolution via kernel estimation and noise injection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020, pp. 466–467.
  * [99] Y. Yuan, S. Liu, J. Zhang, Y. Zhang, C. Dong, and L. Lin, “Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2018, pp. 701–710.
  * [100] Y. Zhang, S. Liu, C. Dong, X. Zhang, and Y. Yuan, “Multiple cycle-in-cycle generative adversarial networks for unsupervised image super-resolution,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 1101–1112, 2020.
  * [101] G. Kim, J. Park, K. Lee, J. Lee, J. Min, B. Lee, D. K. Han, and H. Ko, “Unsupervised real-world super resolution with cycle generative adversarial network and domain discriminator,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020, pp. 456–457.
  * [102] S. Maeda, “Unpaired image super-resolution using pseudo-supervision,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 291–300.
  * [103] K. Prajapati, V. Chudasama, H. Patel, K. Upla, R. Ramachandra, K. Raja, and C. Busch, “Unsupervised single image super-resolution network (USISResNet) for real-world data using generative adversarial network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020, pp. 464–465.
  * [104] T. Zhao, W. Ren, C. Zhang, D. Ren, and Q. Hu, “Unsupervised degradation learning for single image super-resolution,” _arXiv preprint arXiv:1812.04240_ , 2018.
  * [105] C. You, G. Li, Y. Zhang, X. Zhang, H. Shan, M. Li, S. Ju, Z. Zhao, Z. Zhang, W. Cong _et al._ , “CT super-resolution gan constrained by the identical, residual, and cycle learning ensemble (GAN-CIRCLE),” _IEEE Transactions on Medical Imaging_ , vol. 39, no. 1, pp. 188–203, 2020.
  * [106] M. Fritsche, S. Gu, and R. Timofte, “Frequency separation for real-world super-resolution,” in _IEEE International Conference on Computer Vision Workshop (ICCVW)_ , 2019, pp. 3599–3608.
  * [107] R. Muhammad Umer, G. Luca Foresti, and C. Micheloni, “Deep generative adversarial residual convolutional networks for real-world super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020, pp. 438–439.
  * [108] M. S. Rad, T. Yu, C. Musat, H. K. Ekenel, B. Bozorgtabar, and J.-P. Thiran, “Benefiting from bicubically down-sampled images for learning real-world image super-resolution,” in _Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2021, pp. 1590–1599.
  * [109] A. Lugmayr, M. Danelljan, and R. Timofte, “Unsupervised learning for real-world super-resolution,” in _IEEE International Conference on Computer Vision Workshop (ICCVW)_ , 2019, pp. 3408–3416.
  * [110] S. Chen, Z. Han, E. Dai, X. Jia, Z. Liu, L. Xing, X. Zou, C. Xu, J. Liu, and Q. Tian, “Unsupervised image super-resolution with an indirect supervised path,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020, pp. 468–469.
  * [111] A. Shocher, N. Cohen, and M. Irani, ““Zero-Shot” super-resolution using deep internal learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 3118–3126.
  * [112] J. Kim, C. Jung, and C. Kim, “Dual back-projection-based internal learning for blind super-resolution,” _IEEE Signal Processing Letters_ , vol. 27, pp. 1190–1194, 2020.
  * [113] M. Emad, M. Peemen, and H. Corporaal, “DualSR: Zero-shot dual learning for real-world super-resolution,” in _Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2021, pp. 1630–1639.
  * [114] J. W. Soh, S. Cho, and N. I. Cho, “Meta-transfer learning for zero-shot super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 3516–3525.
  * [115] S. Park, J. Yoo, D. Cho, J. Kim, and T. H. Kim, “Fast adaptation to super-resolution networks via meta-learning,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [116] C. Ma, C.-Y. Yang, X. Yang, and M.-H. Yang, “Learning a no-reference quality metric for single-image super-resolution,” _Computer Vision and Image Understanding_ , vol. 158, pp. 1–16, 2017.
  * [117] Y. Fang, C. Zhang, W. Yang, J. Liu, and Z. Guo, “Blind visual quality assessment for image super-resolution by convolutional neural network,” _Multimedia Tools and Applications_ , vol. 77, no. 22, pp. 29 829–29 846, 2018.
  * [118] B. Bare, K. Li, B. Yan, B. Feng, and C. Yao, “A deep learning based no-reference image quality assessment model for single-image super-resolution,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 1223–1227.
  * [119] M. Greeshma and V. Bindu, “Super-resolution quality criterion (srqc): a super-resolution image quality assessment metric,” _Multimedia Tools and Applications_ , pp. 1–22, 2020.
  * [120] J. Cai, S. Gu, R. Timofte, and L. Zhang, “NTIRE 2019 challenge on real image super-resolution: Methods and results,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2019, pp. 2211–2223.
  * [121] A. Lugmayr, M. Danelljan, R. Timofte, M. Fritsche, S. Gu, K. Purohit, P. Kandula, M. Suin, A. Rajagoapalan, N. H. Joon _et al._ , “AIM 2019 challenge on real-world image super-resolution: Methods and results,” in _IEEE International Conference on Computer Vision Workshop (ICCVW)_ , 2019, pp. 3575–3583.
  * [122] A. Lugmayr, M. Danelljan, and R. Timofte, “NTIRE 2020 challenge on real-world image super-resolution: Methods and results,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020, pp. 494–495.
  * [123] P. Wei, H. Lu, R. Timofte, L. Lin, W. Zuo, Z. Pan, B. Li, T. Xi, Y. Fan, G. Zhang _et al._ , “AIM 2020 challenge on real image super-resolution: Methods and results,” in _European Conference on Computer Vision Workshops (ECCVW)_ , 2020.
  * [124] L. Yue, H. Shen, J. Li, Q. Yuan, H. Zhang, and L. Zhang, “Image super-resolution: The techniques, applications, and future,” _Signal Processing_ , vol. 128, pp. 389–408, 2016.
  * [125] W. Yang, X. Zhang, Y. Tian, W. Wang, J.-H. Xue, and Q. Liao, “Deep learning for single image super-resolution: A brief review,” _IEEE Transactions on Multimedia_ , vol. 21, no. 12, pp. 3106–3121, 2019.
  * [126] Z. Wang, J. Chen, and S. C. Hoi, “Deep learning for image super-resolution: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [127] H. Liu, Z. Ruan, P. Zhao, F. Shang, L. Yang, and Y. Liu, “Video super resolution based on deep learning: A comprehensive survey,” _arXiv preprint arXiv:2007.12928_ , 2020.
  * [128] E. Agustsson and R. Timofte, “NTIRE 2017 challenge on single image super-resolution: Dataset and study,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2017, pp. 126–135.
  * [129] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 33, no. 5, pp. 898–916, 2011.
  * [130] M. Bevilacqua, A. Roumy, C. Guillemot, and M.-L. A. Morel, “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” in _British Machine Vision Conference (BMVC)_ , 2012.
  * [131] A. Fujimoto, T. Ogawa, K. Yamamoto, Y. Matsui, T. Yamasaki, and K. Aizawa, “Manga109 dataset and creation of metadata,” in _Proceedings of the 1st International Workshop on Comics Analysis, Processing and Understanding_ , 2016, pp. 1–5.
  * [132] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” _International Journal of Computer Vision_ , vol. 60, no. 2, pp. 91–110, 2004\.
  * [133] M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” _Communications of the ACM_ , vol. 24, no. 6, pp. 381–395, 1981.
  * [134] G. D. Evangelidis and E. Z. Psarakis, “Parametric image alignment using enhanced correlation coefficient maximization,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 30, no. 10, pp. 1858–1865, 2008\.
  * [135] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Transactions on Image Processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [136] H. R. Sheikh, A. C. Bovik, and G. De Veciana, “An information fidelity criterion for image quality assessment using natural scene statistics,” _IEEE Transactions on Image Processing_ , vol. 14, no. 12, pp. 2117–2128, 2005.
  * [137] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 586–595.
  * [138] A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a ‘completely blind’ image quality analyzer,” _IEEE Signal Processing Letters_ , vol. 20, no. 3, pp. 209–212, 2012.
  * [139] N. Venkatanath, D. Praneeth, M. C. Bh, S. S. Channappayya, and S. S. Medasani, “Blind image quality evaluation using perception based features,” in _2015 Twenty First National Conference on Communications (NCC)_ , 2015, pp. 1–6.
  * [140] C.-Y. Yang, C. Ma, and M.-H. Yang, “Single-image super-resolution: A benchmark,” in _European Conference on Computer Vision (ECCV)_ , 2014, pp. 372–386.
  * [141] N. Efrat, D. Glasner, A. Apartsin, B. Nadler, and A. Levin, “Accurate blur models vs. image priors in single image super-resolution,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2013, pp. 2832–2839.
  * [142] R. Mechrez, I. Talmi, and L. Zelnik-Manor, “The contextual loss for image transformation with non-aligned data,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 768–783.
  * [143] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in _IEEE International Conference on Computer Vision (ICCV)_ , 1998, pp. 839–846.
  * [144] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations (ICLR)_ , 2015.
  * [145] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 2223–2232.
  * [146] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “ESRGAN: Enhanced super-resolution generative adversarial networks,” in _Proceedings of the European Conference on Computer Vision Workshops (ECCVW)_ , 2018.
  * [147] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International Conference on Machine Learning (ICML)_ , 2017.