# DaLPSR: Leverage Degradation-Aligned Language Prompt for Real-World Image
Super-Resolution

Aiwen Jiang IEEE Member, Zhi Wei, Long Peng, Feiqiang Liu, Wenbo Li, Mingwen
Wang* IEEE Member Aiwen Jiang, Zhi Wei, Feiqiang Liu and Mingwen Wang are with
School of Computer and Information Engineering, Jiangxi Normal University,
Nanchang, China. Aiwen Jiang and Mingwen Wang are also with School of Digital
Industry, Jiangxi Normal University, Shangrao, China. Long Peng is with
Institute of Advanced Technology, University of Science and Technology of
China, Hefei, China. Wenbo Li is with Noah’s Ark Lab, Huawei Technologies
Ltd., HongKong, China.Manuscript received June, XXX, 2024; revised XXXX,XXXX.

###### Abstract

Image super-resolution pursuits reconstructing high-fidelity high-resolution
counterpart for low-resolution (LR) image. In recent years, diffusion-based
models have garnered significant attention due to their capabilities with rich
prior knowledge. The success of diffusion models based on general text prompts
has validated the effectiveness of textual control in the field of text2image.
However, given the severe degradation commonly presented in low-resolution
images, coupled with the randomness characteristics of diffusion models,
current models struggle to adequately discern semantic and degradation
information within severely degraded images. This often leads to obstacles
such as semantic loss, visual artifacts, and visual hallucinations, which pose
substantial challenges for practical use. To address these challenges, this
paper proposes to leverage degradation-aligned language prompt for accurate,
fine-grained, and high-fidelity image restoration. Complementary priors
including semantic content descriptions and degradation prompts are explored.
Specifically, on one hand, image-restoration prompt alignment decoder is
proposed to automatically discern the degradation degree of LR images, thereby
generating beneficial degradation priors for image restoration. On the other
hand, much richly tailored descriptions from pretrained multimodal large
language model elicit high-level semantic priors closely aligned with human
perception, ensuring fidelity control for image restoration. Comprehensive
comparisons with state-of-the-art methods have been done on several popular
synthetic and real-world benchmark datasets. The quantitative and qualitative
analysis have demonstrated that the proposed method achieves a new state-of-
the-art perceptual quality level, especially in real-world cases based on
reference-free metrics. Related source codes and pre-trained parameters will
be public in https://github.com/puppy210/DaLPSR.

###### Index Terms:

Diffusion Model, Image Super-Resolution, Visual-Language Model, Multimodal
Large Language Model, Degradation Prompt

##  I Introduction

Image super-resolution pursuits improving image clarity and overall visual
quality for low resolution (LR) image[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16]. It involves an intricate task of extracting nuanced
perceptual details from LR counterpart to reconstruct high-fidelity and high-
resolution image. In recent years, the popularity of deep learning has
promoted profound advancements in this field. However, traditional mainstream
methods[17, 18, 19] often exhibited a propensity towards training on delimited
degradation scenarios, thereby constraining their adaptability to unknown
complex conditions. As a consequence, real-world applications were often
ensnared by the challenges inherent in navigating through diverse and complex
degradation scenarios, encompassing denoising, deblurring, and compression
artifact removal among other intricacies[20, 21, 22, 23, 24, 25, 26, 27].

To recover realistic high-resolution (HR) images with perceptually clear
details, some researchers proposed to employ long-short camera lenses to
collect LR-HR image pairs from real world[28, 29, 30]. Some other researchers
employed more economical approach to simulate complex real-world image
degradation using random combinations of basic degradation operations[31, 32,
33, 34, 35]. Representative works in this regard include BSRGAN[20], Real-
ESRGAN[36], and their variants[6, 37, 38].

Recent endeavors have heralded the ascendancy of denoising diffusion
probabilistic models[39] within the realm of image generation. Establishing
works[40, 41, 42] have validated that diffusion-based super-resolution models
consistently outshone their counterparts relying on generative adversarial
networks on various public datasets. More recently, in response to the
exigencies posed by the labyrinthine issues of image degradation, particularly
in cases where characteristics of degradation remain nebulous, large-scale
pre-trained text-to-image models are increasingly inclined to serve as
beneficial helpers[43]. In this routine, ControlNet[44] introduced as an
innovative adaptor can effectively utilize supplementary conditions to steer
the generative capabilities of pretrained models. Prior research such as
DiffBIR[45] has demonstrated that integrating ControlNet into image super-
resolution process yields remarkable fidelity in reproducing realistic
details.

Nevertheless, it poses challenges to derive semantic priors in the form of
textual prompts into pretrained text-to-image models, given the complex
degradation patterns observed in LR images. PASD[46] and SeeSR[47] proposed to
employ existing labeling models to extract object labels as high-level
prompts. Unfortunately, these annotation cues often lack intricate details for
scene comprehension. Moreover, substantial degradation in LR image, such as
disruption of local structures, may lead to semantic ambiguities, causing the
reconstructed HR image to potentially exhibit semantic inaccuracies, thereby
deteriorating final super-resolution performance.

Multimodal large language models (MLLMs) [48, 49, 50, 51] have showcased
remarkable proficiency in visual comprehension, and obtained remarkable
success across a gamut of downstream tasks. SUPIR[52] was one of the few
latest pioneering works that empowered the capabilities of MLLMs for super-
resolution task. It initially integrated robust features from LR images into
LDM image decoder to produce a HR reference image. Subsequently, it leveraged
MLLMs to generate precise and detailed textual descriptions based on the HR
reference. Finally, the generated descriptions acted as semantic prompt to
guide pretrained SDXL[53] for image restoration. As it claimed that it was the
largest-ever image restoration method, inevitably, it substantially escalated
computational resource demands, which were unaffordable for ordinary
applications.

We deem that accurate and comprehensive prompts are crucial in T2I based
super-resolution schemes. Therefore, in this paper, we have proposed an
effective multimodal framework (DaLPSR) that leverages degradation-aligned
language prompt for real-world image super-resolution task. In the proposed
DaLPSR, two complementary priors are generated to overcome preceding mentioned
problems. Specifically, on one hand, image-restoration prompt alignment
decoder (IRPAD) is proposed and trained to automatically discern the
degradation degree of LR images. To facilitate the generation of image-
restoration prompt, we have constructed an auxiliary dataset with triplet data
that signifies the degradation process. We adopt to discretize degradation
degree into several intervals, and take the generation process of degradation
prompt as fine-grained retrieval process. The generated prompts from IRPAD
provide degradation priors for Stable Diffusion. On the other hand, MLLM is
utilized to acquire high-level semantic priors, ensuring fidelity control for
the generated content. To ensure that the high-level semantic priors generated
by MLLM correlate closely with semantic ground-truth, along with fine-tuning
image encoder for feature coherence between LR and HR image, we introduce
Recognize Anything Model (RAM)[54] to generate object labels as part of prompt
instruction for MLLM, guiding it to generate accurate high-level semantic
priors aligned with image subjects.

In summary, the main contributions of this work are as followings:

  1. 1.

We have proposed an effective and computational affordable one-stage
degradation aligned multimodal framework for real-world image super-resolution
task. Complementary priors including semantic content descriptions and
degradation prompts are leveraged for accurate, fine-grained, and high-
fidelity image restoration. Much richly tailored prompts for MLLMs elicit
high-level semantic priors closely aligned with human perception.

  2. 2.

We have proposed an effective image-restoration prompt alignment decoder that
is capable of automatically identifying degradation level of LR images,
thereby generating beneficial degradation priors for image restoration. To the
best of my knowledge, we are the first to explicitly concern degradation
prompts for image super-resolution.

  3. 3.

We have engineered an image-restoration prompt generation pipeline that
seamlessly incorporates textual degradation prompts into SR datasets. As a
result, we have innovatively contributed an auxiliary triplet dataset for
degradation learning.

  4. 4.

We have conducted comprehensive experiments on popular benchmark datasets,
including both synthetic and real-world cases. The experiment results
according to reference-based and reference-free metrics have demonstrated that
the proposed method achieves new state-of-the-art perceptual performance on
real-world cases.

##  II Related Work

###  II-A GAN-based Real-ISR methods

Ever since the emergence of SRCNN[55], deep learning based image super-
resolution (ISR) has garnered significant attention in computer vision area.
Various methods aiming at enhancing the reconstruction precision of ISR have
been proposed. Among these methods, most of them generally operated on
predetermined degradation that typically employed bicubic downsampling to
generate low-resolution image. As a result, when they confronting intricate
and unpredictable degradation in real-world scenarios, such straightforward
degradation assumption would constrain their efficacy.

Recent studies emphasized the imperative of delving into more intricate
degradation models to better emulate real-world conditions. For instance,
BSRGAN[20] introduced a stochastic degradation modeling strategy. Real-
ESRGAN[36] adopted sophisticated high-order degradation strategy. By
leveraging training datasets featuring more realistic degradation, both BSRGAN
and RealESRGAN harness generative adversarial networks (GANs) to reconstruct
requisite high-resolution images.

Albeit yielding more authentic details, the instability inherent in training
GANs often leads to ISR outputs plagued by unnatural visual artifacts. Being
aware the fact, subsequent endeavors such as LDL[37] and DeSRA[56]
concentrated on mitigating these artifacts. Specifically, LDL proposed to
explicitly discriminate visual artifacts from realistic details, and design
locally discriminative learning framework to regularize the adversarial
training. DeSRA proposed to locate problematic areas based on a defined
relative local variance distance and semantic-aware thresholds. After
detecting the artifact regions, they proposed to develop a finetune procedure
to improve GAN-based SR models. Nevertheless, they still frequently
encountered challenges in generating more naturally plausible details.

###  II-B Diffusion-based Real-ISR methods

The diffusion model is a type of generative model that utilizes diffusion
ideas from physical thermodynamics, mainly including forward diffusion and
backward diffusion processes. In the realm of image generation, denoising
diffusion probabilistic model (DDPM)[39] stood as a pioneering effort to
employ diffusion models, boasting commendable performance and surpassing GAN-
based methods. As following work after DDPM, denoising diffusion implicit
model (DDIM)[57] introduced a distinct yet effective sampling technique to
address the inference inefficiency of traditional diffusion process. Latent
diffusion model (LDM)[40] from another perspective extended diffusion models
into latent space of pre-trained autoencoders, enabling training large-scale
model with limited computational resources. Concurrently, in the domain of
text-to-image field, Stable Diffusion (SD), based on latent diffusion, have
garnered significant success.

In recent times, diffusion models have been astutely repurposed in the field
of Real-ISR to be leveraged as priors. There are two main categories for
diffusion-based SR methods. The first category[58, 59, 60, 61] involves
training diffusion model from scratch, where models use a combination of LR
images and noise as input at each diffusion step. The second category[42, 45,
46, 47] leverages prior knowledge from pre-trained diffusion models (e.g.,
Stable Diffusion). Methods of the second category required implementing
”relearning” of large models through adaptors such as ControlNet[44],
LoRA[62], etc. Specifically, StableSR[42] fine-tuned Stable Diffusion (SD)
model and employed feature warping to strike balance between fidelity and
perceptual quality. DiffBIR[45] adopted a two-stage methodology, which
reconstructed an initial HR estimate followed by utilizing SD to refine and
augment visual details. Objectively speaking, under the impetus of diffusion-
based methods, significant progress has been made on image super-resolution,
especially on precise restoration of image details. However, it still presents
great challenge when facing restoration of LR images featuring complex
distortions and spatially intertwined object locations.

###  II-C Semantic Guidance-based Real-ISR methods

Benefiting from robust generative priors, pretrained text-to-image diffusion
models are becoming increasingly popular for addressing Real-ISR challenges.
Previous ISR works have attempted to leverage existing visual perception
models (e.g., ResNet[63], BLIP[64], or RAM[54]) to comprehend image content
and provide multimodal semantic priors as prompts. Typically, PASD[46] and
SeeSR[47] introduced high-level semantic information (e.g., object labels) as
additional conditions, to harness the potential of pre-trained text-to-image
diffusion models. Nevertheless, these prompts were often insufficient since
they merely contained basic information from object recognition, lacking
details of spatial localization and scene understanding.

Figure 1: Overview of the proposed method. (a) Image-Restoration Prompt
Alignment Decoder (IRPAD) is designed to extract image-restoration prompts
from LR images. (b) The extracted prompts from IRPAD are integrated into image
generation process, enabling the model to leverage semantic priors and
restoration priors to enhance the quality and fidelity of the reconstructed HR
images.

Benefiting from scale expanding of multimodal training data and success of
large language model (LLMs), multimodal LLM (MLLMs) have demonstrated
significant capabilities in semantic understanding. They are empowered to well
capture and interpret complex semantic relationships across different
modalities, leading to more accurate and contextually relevant outputs.
However, in the current usage of MLLMs, it is still not an easy task to
determine appropriate instructions to guide MLLMs to generate outputs aligning
with human preference, since crafting these instructions involves not only
understanding the nuances of human language and preferences but also how to
translate these nuances into machine-understandable prompts. This process
necessitates sustained practice and in-depth research to optimize the design
of instructions, enabling them to effectively direct large model to generate
high-quality, semantically consistent outputs.

##  III Proposed Method

In text-to-image research field, popular diffusion model primarily leverages
semantic textual prompt to generate images that are both more expressive and
semantically meaningful. In this study, we deem that it is viable way to
enhance model’s capability on image super-resolution through utilizing textual
prompt as semantic prior during image generation process. Through the
incorporation of textual prompt, models can achieve a more precise
understanding of semantic content presented in LR image, leading to
reconstruction of more clearer super-resolution image with enriching
perceptual details.

In this section, we primary describe our proposed DaLPSR network in details.
The overview of the proposed network is provided in Figure 1.

###  III-A Preliminary

####  III-A1 Stable Diffusion

Stable Diffusion is a latent diffusion model comprising an auto-encoder with
Encoder $\mathcal{E}$ and Decoder $\mathcal{D}$, a prompt-conditioning
denoising U-Net $\mathcal{E}_{\theta}$ and a CLIP text-encoder
$\mathcal{T}_{E}$. Input image $x$ undergoes perceptual compression by Encoder
$\mathcal{E}$ into a lower-dimensional latent space $z$. Decoder $\mathcal{D}$
is responsible for converting the latent representation back to pixel space.
Text-encoder $\mathcal{T}_{E}$ generates latent text embeddings $\tau$ for
input text prompts. Denoising Network $\mathcal{E}_{\theta}$ is employed to
gradually process/diffuse information transformed into latent space.

The training of denoising U-Net $\mathcal{E}_{\theta}$ involves minimizing a
defined loss function as shown in Equation 1:

$\mathrm{L}=\mathbb{E}_{z,\tau,\epsilon\sim\mathcal{N}(0,I),t}[||\epsilon-\mathcal{E}_{\theta}(z_{t},t,\tau)||_{2}^{2}],$
(1)

where $t$ is diffusion timestep, $z_{t}$ represents a fully noised image or
encoded image generated through a gradual addition of Gaussian Noise
$\epsilon\sim\mathcal{N}(0,I)$, and $\tau$ is latent text embedding.

Stable diffusion employs cross-attention to incorporate textual prompts into
image generation process. This process involves encoding latent image features
$\mathit{F_{img}}$ and textual features $\mathit{F_{\tau}}$. It maps
respective features to query $Q={{W}_{Q}}\cdot{{F}_{img}}$, key
$K={{W}_{K}}\cdot{\mathit{F}_{\tau}}$, and value
$V={{W}_{V}}\cdot{\mathit{F}_{\tau}}$ through linear projection layers, where
$W_{Q}$, $W_{K}$, $W_{V}$ respectively represent the weight parameters of the
query projection layer, key projection layer and value projection layer. The
calculation of cross attention is subsequently executed by performing a
weighted summation of the value features $V$. The formula of cross-attention
is as shown in Equation 2.

$Attention(Q,K,V)=Softmax(\frac{Q{{K}^{T}}}{\sqrt{d}})V,$ (2)

where $d$ represents the output dimension of key and query features. The
output of cross-attention block is utilized to update the latent features in
Stable-Diffusion model.

####  III-A2 ControlNet

ControlNet[44] is specially developed for efficiently managing pretrained
large diffusion models while accommodating additional input conditioning. It
involves creating two distinct copies of the weights from pretrained diffusion
model. One is a trainable copy and the other is a locked copy. The trainable
copy is trained on task-specific datasets to learn some required condition,
while the locked copy remains fixed. These two copies are connected through
zero-convolutions, enabling efficient training that is in comparable speed
with fine-tuning, while avoiding introducing additional noise to deep features
of original pretrained network. The fundamental architecture of ControlNet is
shown in Figure 2.

Figure 2: The fundamental structure of ControlNet. $Z_{\text{LR}}$ is the
latent representation of LR image.

###  III-B Image-Restoration Prompt Alignment Decoder (IRPAD)

####  III-B1 Image-Restoration Prompt Generation Pipeline

To facilitate the generation of image-restoration prompt, we construct an
auxiliary dataset that signifies the degradation process. In the constructed
dataset, each data is a triplet [$I_{\text{HR}}$, $I_{\text{LR}}$, $p_{r}$],
where $p_{r}$ signifies the textual prompt characterizing degradation process,
while $I_{\text{HR}}$ and $I_{\text{LR}}$ respectively represents HR and LR
image in pair.

Specifically, the generation pipeline of image-restoration prompt is outlined
as in Figure 3. It consists of two primary components. One is the degradation
module and the other is the restoration prompt representation module. The
former is responsible for producing HR-LR image pairs. It follow the second-
order degradation process as in Real-ESRGAN[36] to model practical
degradations. The latter delineates the degradation process to generate
corresponding textual prompts.

Since degradation space is continuous, it is difficult to accurately identify
the true degree value for a specific degradation. For simplicity and practical
use, in this paper, we adopt a strategy that involves discretizing degradation
space into several intervals. As a result, textual descriptions are
facilitated to comprehensively and informatively capture nuanced variations
and intricacies in degradation process. As shown in Figure 3, we specifically
take 4 intervals, e.g. [0,0.25], (0.25,0.5], (0.5,0.75], and (0.75,1] for
degradation degree. Consequently, restoration prompts are generated as
descriptions ”Deblur with sigma $0.5\sim 0.75$, Denoise with sigma $0.25\sim
0.5$, Dejpeg with sigma $0\sim 0.25$”. We deem that it is essential for ill-
reverse applications like image restoration, helping narrow semantic gap. The
adopted strategy can ensure clarity and precision in conveying the intricacies
of degradation operations in textual way.

Figure 3: The image-restoration prompt generation pipeline. Discretization
strategy is employed on degradation level representation to delineate high-
order degradation process.  Figure 4: The example demonstrates LLaVA has the
capacity to produce high-level semantic prompts that align with human
perceptual comprehension through application of tailored prompt instructions.
RAM is employed to detect semantic tags to formulate tailored instructions for
LR images.

####  III-B2 Image-Restoration Prompt Alignment Decoder

Figure 1(a) delineates the image-prompt alignment process. The alignment
decoder comprises two successive blocks, each of which is consist of a cross-
attention layer and a feedforward layer. Embeddings of degradation prompts are
aligned with image features within the alignment decoder.

Specifically, in cross-attention layer, degradation prompts function as
queries, and image features encoded from precede block serve as keys and
values. During pretraining stage, we take the alignment process as fine-
grained retrieval process. For each degradation type, we select the most
relevant degradation prompt from the define four intervals.

Moreover, to avoid combination explosion problem, we respectively construct an
image-restoration prompt alignment decoder for each degradation type. And then
concatenate respective degradation prompt retrieved to form the final prompt
description. Consequently, in this way, we can prioritize efficiency in
computational resources, emphasize the role of degradation prompt and preserve
good scalability to more degradation types in future.

###  III-C Multi-Modality Semantic Priors Guidance

Benefiting from significantly scaling up the size and complexity of both data
and language model, multimodal large language models (MLLMs) have achieved
unprecedentedly amazing abilities on many high-level tasks, such as image
captioning. In this work, we leverage a state-of-the-art MLLM known as
LLaVA[50] to enhance the perception of LR images and extract informative
semantic priors. LLaVA’s robust capabilities enable it to capture intricate
relationships and patterns within visual data.

To enhance the coherence between the semantic content of LR images and the
descriptions generated by LLaVA, we propose to refine the feature extraction
ability of image encoder $\mathcal{E}$ on LR image. We conduct fine-tuning on
image encoder $\mathcal{E}$ through minimizing the loss
$\mathcal{L}_{\mathcal{E}}=\|\mathcal{E}(I_{\text{LR}})-\mathcal{E}(I_{\text{HR}})\|_{2}^{2}$,
thereby reducing the disparity between LR and HR images.

As we know, it is essential to provide clear and comprehensive instructions to
MLLM, ensuring that its generated outputs align with human intentions and
preferences. Considering the robust capability of RAM[54] on image tag
recognition, we propose to employ RAM to detect tags for LR images.
Subsequently, we leverage the identified tags to formulate tailored
instructions corresponding to different LR images. As depicted in Figure 4,
the tags extracted from LR image serve as part of instructive prompts for
LLaVA. Through the application of specific prompt instructions, LLaVA is
capable of producing high-level semantic image descriptions that align with
human perceptual understanding.

###  III-D Training Strategy

The training strategy of the proposed method is delineated in Algorithm 1.
During model training, we initially derive the latent representation
$\mathbf{z}_{0}$ of HR image. Subsequently, we incrementally introduce noise
to this latent representation, producing a noisy latent $\mathbf{z}_{t}$,
where $t$ represents a randomly sampled diffusion step. Considering various
conditions such as the diffusion step $t$, latent LR input
$\mathbf{z}_{\text{LR}}$, restoration prompt $p_{r}$, and high-level semantic
prompt $p_{s}$, we train the network $\epsilon_{\theta}$ to predict the noise
introduced into the noisy latent $\mathbf{z}_{t}$. The optimization objective
is:

$\mathcal{L}=\mathbb{E}_{z_{0},z_{\text{LR}},t,p_{r},p_{s},\epsilon\sim\mathcal{N}}[\|\epsilon-\epsilon_{\theta}(\mathbf{z}_{t},\mathbf{z}_{\text{LR}},t,p_{r},p_{s})\|_{2}^{2}].$
(3)

Algorithm 1 Training

Training datasets:
$(I_{\text{HR}},I_{\text{LR}},p_{r},p_{s}\sim{MLLM(I_{\text{LR}}))}$

sample:
$\mathbf{z}_{0},\mathbf{z}_{\text{LR}}\sim(I_{\text{HR}},I_{\text{LR}})$

while not converged do

$t\sim\text{Uniform}(\\{1,\ldots,T\\})$

sample: $\epsilon\sim\mathcal{N}(\mathbf{0},\kappa^{2}\eta_{t}\mathbf{I})$

$\mathbf{z}_{t}=\mathbf{z}_{0}+\epsilon$

The optimization objective is:

$\mathcal{L}=\mathbb{E}_{z_{0},z_{\text{LR}},t,p_{r},p_{s},\epsilon\sim\mathcal{N}}[\|\epsilon-\epsilon_{\theta}(\mathbf{z}_{t},\mathbf{z}_{\text{LR}},t,p_{r},p_{s})\|_{2}^{2}]$

end while

##  IV Experiments

###  IV-A Datasets and metrics

####  IV-A1 Training datasets

As similar settings to conventional training strategy, our training data are
collected from DIV2K[65], DIV8K[66], Flickr2K[67], OST[68], and a subset of
top 10K images from FFHQ[69]. Initially, HR images are randomly cropped to a
size of $512\times 512$. Then, image-restoration prompt generation pipeline is
employed to produce training dataset containing HR-LR image pairs and
corresponding restoration prompts. Subsequently, the proposed network is
trained using the generated training data.

####  IV-A2 Test Datasets

In order to thoroughly and reliably evaluate the proposed model, we conducted
experiments on several synthetic and real-world benchmark datasets that widely
adopted in super-resolution tasks.

In case of synthetic dataset, we utilized the validation set from DIV2K[65]
dataset. Following the same experiment settings as SeeSR[47], we utilize the
randomly cropped 3K image patches for
testing111https://huggingface.co/datasets/Iceclear/StableSR-TestSets. The HR
images are of size $512\times 512$. The corresponding LR images of size
$128\times 128$ are generated using the same degradation pipeline as training
stage.

TABLE I: Quantitative evaluation against state-of-the-art methods on real-world benchmarks. The best performances across each metric are highlighted in red and bold. Datasets | Metrics | BSRGAN | Real-ESRGAN | LDL | FeMaSR | StableSR | ResShift | PASD | DiffBIR | SeeSR | SUPIR* | Ours
---|---|---|---|---|---|---|---|---|---|---|---|---
|  | ICCV 2021 | ICCV 2021 | CVPR 2022 | MM 2022 | arxiv 2023 | NeurIPS 2023 | arxiv 2023 | arxiv 2023 | CVPR 2024 | CVPR 2024 |
|  LPIPS $\downarrow$ | 0.4136 | 0.3868 | 0.3995 | 0.3973 | 0.4055 | 0.4284 | 0.4410 | 0.4270 | 0.3876 | 0.3933 | 0.4085
|  FID $\downarrow$ | 64.28 | 53.46 | 58.94 | 53.70 | 36.57 | 55.77 | 40.77 | 40.42 | 32.79 | 31.73 | 34.92
DIV2K-Val |  MANIQA $\uparrow$ | 0.4834 | 0.5251 | 0.5127 | 0.4869 | 0.5914 | 0.5232 | 0.6049 | 0.6205 | 0.6236 | 0.5080 | 0.6146
|  MUSIQ $\uparrow$ | 59.11 | 58.64 | 57.90 | 58.10 | 62.95 | 58.23 | 66.85 | 65.23 | 68.29 | 59.68 | 70.70
|  CLIPIQA $\uparrow$ | 0.5183 | 0.5424 | 0.5313 | 0.5597 | 0.6486 | 0.5948 | 0.6799 | 0.6664 | 0.6834 | 0.6737 | 0.7556
|  LPIPS $\downarrow$ | 0.2670 | 0.2727 | 0.2766 | 0.2942 | 0.3018 | 0.3460 | 0.3435 | 0.3658 | 0.3133 | 0.3788 | 0.3952
|  FID $\downarrow$ | 141.28 | 135.18 | 142.71 | 141.05 | 128.51 | 141.71 | 129.76 | 128.99 | 134.90 | 133.17 | 148.65
RealSR |  MANIQA $\uparrow$ | 0.5399 | 0.5487 | 0.5485 | 0.4865 | 0.6221 | 0.5285 | 0.6493 | 0.6253 | 0.6506 | 0.5673 | 0.6303
|  MUSIQ $\uparrow$ | 63.21 | 60.18 | 60.82 | 58.95 | 65.78 | 58.43 | 68.69 | 64.85 | 69.67 | 65.41 | 71.48
|  CLIPIQA $\uparrow$ | 0.5001 | 0.4449 | 0.4477 | 0.5270 | 0.6178 | 0.5444 | 0.6590 | 0.6386 | 0.6630 | 0.6896 | 0.7199
|  LPIPS $\downarrow$ | 0.2883 | 0.2847 | 0.2815 | 0.3169 | 0.3315 | 0.4006 | 0.3931 | 0.4599 | 0.3299 | 0.4180 | 0.4048
|  FID $\downarrow$ | 155.63 | 147.62 | 155.53 | 157.78 | 148.98 | 172.26 | 159.24 | 166.79 | 151.88 | 164.64 | 164.66
DRealSR |  MANIQA $\uparrow$ | 0.4878 | 0.4907 | 0.4914 | 0.4420 | 0.5591 | 0.4586 | 0.5850 | 0.5923 | 0.6005 | 0.4776 | 0.5733
|  MUSIQ $\uparrow$ | 57.14 | 54.18 | 53.85 | 53.74 | 58.42 | 50.60 | 64.81 | 61.19 | 65.11 | 58.70 | 65.88
|  CLIPIQA $\uparrow$ | 0.4915 | 0.4422 | 0.4310 | 0.5464 | 0.6206 | 0.5342 | 0.6773 | 0.6346 | 0.6708 | 0.6701 | 0.7245
|  MANIQA $\uparrow$ | 0.5462 | 0.5582 | 0.5519 | 0.5295 | 0.5841 | 0.5417 | 0.6066 | 0.5902 | 0.6198 | 0.4795 | 0.6339
RealLR200 |  MUSIQ $\uparrow$ | 64.87 | 62.94 | 63.11 | 64.14 | 63.30 | 60.18 | 68.20 | 62.06 | 69.52 | 66.44 | 72.78
|  CLIPIQA $\uparrow$ | 0.5679 | 0.5389 | 0.5326 | 0.6522 | 0.6068 | 0.6486 | 0.6797 | 0.6509 | 0.6814 | 0.6275 | 0.7728

In case of real-world dataset, we employ three benchmark datasets,
DRealSR[30], RealSR[29] and RealLR200[47] for evaluation. Except for
RealLR200, all datasets undergo cropping HR image to dimensions of $512\times
512$ and subsequent LR images of size $128\times 128$. LR images in RealLR200
dataset are of various sizes for practical use.

####  IV-A3 Compared Methods

We perform benchmark comparisons for our proposed method against several
representative cutting-edge Real-ISR methods. The compared methods are
categorized into two primary groups. One group contains GAN-based methods,
such as BSRGAN[20], Real-ESRGAN[36], LDL[37], FeMaSR[2]. The other group
encompasses diffusion-based methods, like StableSR[42], ResShift[59],
PASD[46], DiffBIR[45], SeeSR[47] and SUPIR[52].

####  IV-A4 Evaluation indicators

Both established reference-based and reference-free metrics are utilized for
evaluation.

The reference-based perceptual quality assessment is conducted using LPIPS[70]
and FID[71] metrics. LPIPS gauges perceptual similarity between the restored
images and their references. FID quantifies the distributional dissimilarity
between the restored images and their references, providing insights into the
fidelity of image restoration. Notably, traditional pixel-level evaluation
metrics such as PSNR and SSIM[72] are excluded from our assessment due to
their limited efficacy in capturing nuanced aspects of human perceptual
quality.

For reference-free evaluation, we leverage MANIQA[73], MUSIQ[74], and
CLIPIQA[75] metrics. These metrics offer a holistic assessment of image
quality without relying on reference images, thus providing a more robust
evaluation framework for assessing the fidelity and perceptual quality of
generated images.

###  IV-B Implementation details

The SD 2.1-base model222https://github.com/Stability-AI/stablediffusion
servers as the foundational text-to-image model in our experiment. Adam with
batch size of 4 is employed as optimizer for training. We configure the
learning rate to 2e-5 and fine-tune complete model over 100,000 iterations.
During inference, we incorporate LR images with initial noise to establish an
enhanced starting point for diffusion process. All training experiments are
conducted on image patch of $512\times 512$ resolution on NVIDIA 3090 GPU.

###  IV-C Comparison with state-of-the-art methods

####  IV-C1 Quantitative Comparisons

The quantitative comparisons across both synthetic and real-world datasets are
presented in Table I. From the experiment results, we can observe that the
proposed method consistently outperforms competing methods across multiple
datasets, demonstrating superior performance in terms of various perceptual
metrics. Specifically, on RealLR200 dataset, our MANIQA score exhibits a
superiority of 2.27 % over the runner-up method, while our MUSIQ score
surpasses the second-best method by 4.69%, and our CLIPIQA score excels by
13.41% compared to the next best method.

Notably, compared with the most closely work such as SUPIR, the proposed
DaLPSR has several crucial differences. (1) Overall, DaLPSR is one-stage
framework while SUPIR utilized two-stage strategy. (2) The priors that SUPIR
utilized are only the generated descriptions by MLLM based on initially
estimated HR reference. In contrast, DaLPSR, on one hand, with assistance of
RAM, employs MLLM directly on encoded LR images to generate semantic priors.
As evidenced in the experiments, the generated semantic priors in DaLPSR are
more accurate and comprehensive. On the other hand, DaLPSR accounts for
degradation alignment, and learns degradation information as complementary
degradation priors. (3) The computation burdens of DaLPSR are much less than
SUPIR, which enables DaLPSR be more applicable in practical scenario.

We have also noted that GAN-based methods generally exhibit better performance
over diffusion-based approaches in reference-based metrics. We speculate that
diffusion model possesses capability to generate more intricate texture
details, albeit at the expense of marginal reduction in fidelity.
Nevertheless, compared to SOTA diffusion-based methods, the proposed method
achieves more superior scores in reference-free metrics. Especially on
RealLR200 dataset, it attains highest scores across all reference-free
metrics, thereby establishing itself as a valuable asset for practical
deployment.

Figure 5: Visual comparisons between the proposed model and other state-of-
the-art methods on DIV2K-Val dataset. For a clearer and more detailed view,
please zoom in on the images. Figure 6: Visual comparisons between the
proposed model and other state-of-the-art methods on Real-world datasets. For
a clearer and more detailed view, please zoom in on the images.

####  IV-C2 Qualitative Comparisons

We further provide visual comparisons of the proposed method against state-of-
the-art Real-ISR methods on both synthetic and real-world images, as depicted
in Figure 5 and Figure 6.

Figure 7: Qualitative comparisons with state-of-the-art methods on real-world
examples are presented. For a closer examination of the finer details and to
better appreciate the improvements, please zoom in on the images. Figure 8:
Visual comparison for ablation study. ”w/o IRPAD & w/o MLLM” represents model
directly applying ControlNet on SD without degradation prompt and MLLM-
generated semantic prompt. ”w/o IRPAD” represents model utilizing MLLMs to
generate semantic prompt, while without degradation prompt. ”w/o MLLM” employs
IRPAD to generate restoration prompts, while without MLLM-generated semantic
prompt. ”Ours” is final full model with both MLLM and IRPAD, resulting much
clearer and more semantically accurate outcome.

It is evident that all compared methods are capable of restoring visually
high-quality images. Notably, the results produced by SeeSR and SUPIR exhibit
finer details compared to those by ResShift, highlighting the significance of
textual guidance in diffusion-based image restoration models. In Figure 5, our
model tends to generate clearer and more regular contours, such as sharper
leaves, finer animal fur, and well-defined flower petals. Additionally, in
Figure 6, our model surpasses SeeSR and SUPIR in generating semantically
accurate and detail-rich results, which highlights the advantages of using
degradation-alignment prompt conditioning.

In summary, the proposed model demonstrates advantages in detail generation
and degradation removal. These examples underscore the benefits of utilizing
multimodal semantic priors over just relying on scattered object labels. More
visual examples that further illustrate the capabilities of our model can be
found in Figure 7.

####  IV-C3 User Study

To substantiate the superiority of the proposed DaLPSR in generating precise
and realistic LR images, we conducted a user study. This investigation
involved 10 real-world LR images randomly selected from established benchmark
datasets. We recruited 10 participants to evaluate six representative methods
(BSRGAN, Real-ESRGAN, ResShift, SeeSR, SUPIR, and DaLPSR). As a result, each
method can be valuated by 100 scores. During evaluation, the names of all
method was concealed. Participants are blindly to rate ”Which HR image best
matches the LR reference and provides the best SR result?”. Each HR image was
rated on a scale: 2 for very poor, 4 for poor, 6 for fair, 8 for good, and 10
for excellent. As depicted in Figure 9 and Table II, our DaLPSR demonstrates a
higher score compared to other methods when evaluated on real-world data. The
user-study has further confirmed the advantages of our proposed DaLPSR.

Figure 9: Comparisons about score distributions of user-study on real-world data for six representative state-of-the-art methods. The scores are divided into five levels. 2 for very poor, 4 for poor, 6 for fair, 8 for good, and 10 for excellent.  TABLE II: Statistical comparisons of user study on real-world data for six representative state-of-the-art methods.  Methods |  Average Score |  Best rates
---|---|---
BSRGAN |  4.94 |  1%
RealESRGAN |  4.74 |  3%
ResShift |  5.54 |  4%
SeeSR |  6.96 |  17%
SUPIR |  7.60 |  30%
DaLPSR |  8.46 |  45%

###  IV-D Ablation Studies

To validate the effectiveness of our proposed key components, we have
performed ablation studies on representative RealSR and RealLR200 datasets for
practical use.

####  IV-D1 Efficacy of Image-Restoration Prompt Alignment Decoder

As demonstrated in Table III, the Image-Restoration Prompt Alignment Decoder
(IRPAD) significantly enhances the perceptual quality of restored LR images.
This improvement is evident across various evaluation metrics. It underscores
the decoder’s effectiveness in producing visually superior results.

TABLE III: Ablation studies on RealSR and RealLR200 benchmarks. Datasets | RealSR | RealLR200
---|---|---
IRPAD | MLLM | MUSIQ$\uparrow$ | CLIPIQA$\uparrow$ | MUSIQ$\uparrow$ | CLIPIQA$\uparrow$
× | × | 66.72 | 0.6109 | 64.35 | 0.5916
× | ✓ | 68.54 | 0.6355 | 66.82 | 0.6495
✓ | × | 69.69 | 0.6671 | 69.25 | 0.6750
✓ | ✓ | 71.48 | 0.7199 | 72.78 | 0.7728

For more intuitive observation, we provide visual comparisons for ablation
experiments. As illustrated in Figure 8, the implementation of IRPAD markedly
diminishes image noise and mitigates aliasing artifacts. The resulted image
displays textures with well-defined edges. The improvements explicitly
demonstrate the decoder’s capability to produce visually appealing and high-
fidelity restorations.

####  IV-D2 Efficacy of Multi-Modality Semantic Priors

As the results presented in Table III, it is evident that the exclusion of
high-level semantic priors results in a significant reduction in fidelity
metrics. This decrease underscores the critical role that high-level priors
play in preserving the perceptual quality and structural integrity of restored
images.

Moreover, as illustrated in Figure 8, the effectiveness of utilizing multi-
modality semantic priors is further validated. The incorporation of multi-
modality semantic priors leads to images with clearer details and more refined
textures, resulting in higher level of realism and overall image quality.

Both ablation studies have demonstrated the significant advantage of
leveraging multi-modality prompts for superior image restoration outcomes.

TABLE IV: Classification accuracy for different degradation types. Degradation |  Blur |  Noise |  JPEG
---|---|---|---
Accuracy |  0.63 |  0.65 |  0.69

####  IV-D3 Discussion and Future Work

In this work, we have innovatively simplified the generation of degradation
prompt, and converted generation problem into isolate classification/retrieval
problem for respective degradation type. As illustrated in ablation study, it
plays significant role on model’s performance improvement. Nevertheless, as
detailed in Table IV, the accuracy of four-class classification is less than
70%, still unsatisfactory. We believe more accurate degradation prompt will be
more benefit for restoration. In the future, we will further optimize the way
of degradation prompt generation, and provide accurate controls to balance the
generation quality and fidelity of HR image.

##  V Conclusion

In this paper, we have proposed an effective and innovative multimodal
framework that leverages degradation aligned language prompts for real-world
image super-resolution. In the proposed framework, two complementary priors
including semantic content priors and image degradation priors are produced as
textual prompts to guide stable diffusion for high-fidelity high-resolution
image generation. Comprehensive experiments on several popular synthetic and
real-world benchmark datasets have demonstrated that the proposed method
achieves significant superior performance, and obtains new state-of-the-art
perceptual quality level, especially on real-world cases based on reference-
free metrics. The proposed method offers a viable solution to overcome
existing challenges with acceptable computation burdens.

## References

  * [1] H. Sun, W. Li, J. Liu, H. Chen, R. Pei, X. Zou, Y. Yan, and Y. Yang, “Coser: Bridging image and language for cognitive super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2024, pp. 25 868–25 878.
  * [2] C. Chen, X. Shi, Y. Qin, X. Li, X. Han, T. Yang, and S. Guo, “Real-world blind super-resolution via feature matching with implicit high-resolution priors,” in _Proceedings of the 30th ACM international conference on multimedia_ , 2022, pp. 1329–1338.
  * [3] Y. Liu, H. Zhao, J. Gu, Y. Qiao, and C. Dong, “Evaluating the generalization ability of super-resolution networks,” _IEEE Transactions on pattern analysis and machine intelligence_ , 2023.
  * [4] W. Zhang, X. Li, X. Chen, X. Zhang, Y. Qiao, X.-M. Wu, and C. Dong, “SEAL: A framework for systematic evaluation of real-world super-resolution,” in _The Twelfth International Conference on Learning Representations_ , 2024\.
  * [5] L. Peng, Y. Cao, R. Pei, W. Li, J. Guo, X. Fu, Y. Wang, and Z.-J. Zha, “Efficient real-world image super-resolution via adaptive directional gradient convolution,” _arXiv preprint arXiv:2405.07023_ , 2024.
  * [6] D. Chen, J. Liang, X. Zhang, M. Liu, H. Zeng, and L. Zhang, “Human guided ground-truth generation for realistic image super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 14 082–14 091.
  * [7] Y. Huang, S. Li, L. Wang, T. Tan _et al._ , “Unfolding the alternating optimization for blind super resolution,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 5632–5643, 2020.
  * [8] W. Li, K. Zhou, L. Qi, L. Lu, and J. Lu, “Best-buddy gans for highly detailed image super-resolution,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 2, 2022, pp. 1412–1420.
  * [9] A. Lugmayr, M. Danelljan, and R. Timofte, “Unsupervised learning for real-world super-resolution,” in _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_. IEEE, 2019, pp. 3408–3416.
  * [10] X. Ji, Y. Cao, Y. Tai, C. Wang, J. Li, and F. Huang, “Real-world super-resolution via kernel estimation and noise injection,” in _proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 2020, pp. 466–467.
  * [11] C. Mou, Y. Wu, X. Wang, C. Dong, J. Zhang, and Y. Shan, “Metric learning based interactive modulation for real-world super-resolution,” in _European Conference on Computer Vision_. Springer, 2022, pp. 723–740.
  * [12] A. Liu, Y. Liu, J. Gu, Y. Qiao, and C. Dong, “Blind image super-resolution: A survey and beyond,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 45, no. 5, pp. 5461–5480, 2022.
  * [13] M. Fritsche, S. Gu, and R. Timofte, “Frequency separation for real-world super-resolution,” in _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_. IEEE, 2019, pp. 3599–3608.
  * [14] Y. Zhou, W. Deng, T. Tong, and Q. Gao, “Guided frequency separation network for real-world super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 428–429.
  * [15] L. Wang, Y. Wang, X. Dong, Q. Xu, J. Yang, W. An, and Y. Guo, “Unsupervised degradation representation learning for blind super-resolution,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 10 581–10 590.
  * [16] Y. Li, B. Sixou, and F. Peyrin, “A review of the deep learning methods for medical images super resolution problems,” _Irbm_ , vol. 42, no. 2, pp. 120–133, 2021.
  * [17] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 38, no. 2, pp. 295–307, 2015.
  * [18] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1646–1654.
  * [19] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang _et al._ , “Photo-realistic single image super-resolution using a generative adversarial network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4681–4690.
  * [20] K. Zhang, J. Liang, L. Van Gool, and R. Timofte, “Designing a practical degradation model for deep blind image super-resolution,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 4791–4800.
  * [21] R. Zhang, J. Gu, H. Chen, C. Dong, Y. Zhang, and W. Yang, “Crafting training degradation distribution for the accuracy-generalization trade-off in real-world super-resolution,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 41 078–41 091.
  * [22] S. Park, D. Kim, S. Baik, and T. H. Kim, “Learning controllable degradation for real-world super-resolution via constrained flows,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 27 188–27 203.
  * [23] X. Li, C. Chen, X. Lin, W. Zuo, and L. Zhang, “From face to natural image: Learning real degradation for blind image super-resolution,” in _European Conference on Computer Vision_. Springer, 2022, pp. 376–392.
  * [24] J. Xiao, H. Yong, and L. Zhang, “Degradation model learning for real-world single image super-resolution,” in _Proceedings of the Asian Conference on Computer Vision_ , 2020.
  * [25] V. Wolf, A. Lugmayr, M. Danelljan, L. Van Gool, and R. Timofte, “Deflow: Learning complex image degradations from unpaired data with conditional flows,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 94–103.
  * [26] M. Elad and A. Feuer, “Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images,” _IEEE transactions on image processing_ , vol. 6, no. 12, pp. 1646–1658, 1997.
  * [27] L. Peng, Y. Cao, Y. Sun, and Y. Wang, “Lightweight adaptive feature de-drifting for compressed image classification,” _IEEE Transactions on Multimedia_ , 2024.
  * [28] C. Chen, Z. Xiong, X. Tian, Z.-J. Zha, and F. Wu, “Camera lens super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1652–1660.
  * [29] J. Cai, H. Zeng, H. Yong, Z. Cao, and L. Zhang, “Toward real-world single image super-resolution: A new benchmark and a new model,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 3086–3095.
  * [30] P. Wei, Z. Xie, H. Lu, Z. Zhan, Q. Ye, W. Zuo, and L. Lin, “Component divide-and-conquer for real-world image super-resolution,” in _Proceedings of European conference computer vision_ , 2020, pp. 101–117.
  * [31] T. Yang, P. Ren, L. Zhang _et al._ , “Synthesizing realistic image restoration training pairs: A diffusion approach,” _arXiv preprint arXiv:2303.06994_ , 2023.
  * [32] A. Bulat, J. Yang, and G. Tzimiropoulos, “To learn image super-resolution, use a gan to learn how to do image degradation first,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 185–200.
  * [33] W. Wu, Y. Zhao, H. Chen, Y. Gu, R. Zhao, Y. He, H. Zhou, M. Z. Shou, and C. Shen, “Datasetdm: Synthesizing data with perception annotations using diffusion models,” _Advances in Neural Information Processing Systems_ , vol. 36, pp. 54 683–54 695, 2023.
  * [34] V. Sehwag, C. Hazirbas, A. Gordo, F. Ozgenel, and C. Canton, “Generating high fidelity data from low-density regions using diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 492–11 501.
  * [35] S. Maeda, “Unpaired image super-resolution using pseudo-supervision,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 291–300.
  * [36] X. Wang, L. Xie, C. Dong, and Y. Shan, “Real-esrgan: Training real-world blind super-resolution with pure synthetic data,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 1905–1914.
  * [37] J. Liang, H. Zeng, and L. Zhang, “Details or artifacts: A locally discriminative learning approach to realistic image super-resolution,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 5657–5666.
  * [38] ——, “Efficient and degradation-adaptive network for real-world image super-resolution,” in _Proceedings of European conference on computer vision_ , 2022, pp. 574–591.
  * [39] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in neural information processing systems_ , vol. 33, pp. 6840–6851, 2020.
  * [40] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 684–10 695.
  * [41] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans _et al._ , “Photorealistic text-to-image diffusion models with deep language understanding,” _Advances in neural information processing systems_ , vol. 35, pp. 36 479–36 494, 2022.
  * [42] J. Wang, Z. Yue, S. Zhou, K. C. Chan, and C. C. Loy, “Exploiting diffusion prior for real-world image super-resolution,” _arXiv preprint arXiv:2305.07015_ , 2023.
  * [43] C. Mou, X. Wang, L. Xie, Y. Wu, J. Zhang, Z. Qi, and Y. Shan, “T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 38, no. 5, 2024, pp. 4296–4304.
  * [44] L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2023, pp. 3836–3847.
  * [45] X. Lin, J. He, Z. Chen, Z. Lyu, B. Fei, B. Dai, W. Ouyang, Y. Qiao, and C. Dong, “Diffbir: Towards blind image restoration with generative diffusion prior,” _arXiv preprint arXiv:2308.15070_ , 2023.
  * [46] T. Yang, P. Ren, X. Xie, and L. Zhang, “Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization,” _arXiv preprint arXiv:2308.14469_ , 2023.
  * [47] R. Wu, T. Yang, L. Sun, Z. Zhang, S. Li, and L. Zhang, “Seesr: Towards semantics-aware real-world image super-resolution,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2024, pp. 25 456–25 467.
  * [48] R. Zhang, J. Han, C. Liu, P. Gao, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, and Y. Qiao, “Llama-adapter: Efficient fine-tuning of language models with zero-init attention,” _arXiv preprint arXiv:2303.16199_ , 2023.
  * [49] J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in _International conference on machine learning_ , 2023, pp. 19 730–19 742.
  * [50] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” _Advances in neural information processing systems_ , vol. 36, 2024.
  * [51] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” _arXiv preprint arXiv:2304.10592_ , 2023.
  * [52] F. Yu, J. Gu, Z. Li, J. Hu, X. Kong, X. Wang, J. He, Y. Qiao, and C. Dong, “Scaling up to excellence: practicing model scaling for photo-realistic image restoration in the wild,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , June 2024, pp. 25 669–25 680.
  * [53] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, “SDXL: Improving latent diffusion models for high-resolution image synthesis,” in _The Twelfth International conference on learning representations_ , 2024.
  * [54] Y. Zhang, X. Huang, J. Ma, Z. Li, Z. Luo, Y. Xie, Y. Qin, T. Luo, Y. Li, S. Liu _et al._ , “Recognize anything: A strong image tagging model,” _arXiv preprint arXiv:2306.03514_ , 2023.
  * [55] C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in _Proceedings of European conference on computer vision_ , 2014, pp. 184–199.
  * [56] L. Xie, X. Wang, X. Chen, G. Li, Y. Shan, J. Zhou, and C. Dong, “Desra: detect and delete the artifacts of gan-based real-world super-resolution models,” _arXiv preprint arXiv:2307.02457_ , 2023.
  * [57] J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” _arXiv preprint arXiv:2010.02502_ , 2020.
  * [58] B. Xia, Y. Zhang, S. Wang, Y. Wang, X. Wu, Y. Tian, W. Yang, and L. Van Gool, “Diffir: Efficient diffusion model for image restoration,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2023, pp. 13 095–13 105.
  * [59] Z. Yue, J. Wang, and C. C. Loy, “Resshift: Efficient diffusion model for image super-resolution by residual shifting,” _Advances in neural information processing systems_ , vol. 36, 2024.
  * [60] Y. Wang, W. Yang, X. Chen, Y. Wang, L. Guo, L.-P. Chau, Z. Liu, Y. Qiao, A. C. Kot, and B. Wen, “Sinsr: Diffusion-based image super-resolution in a single step,” _arXiv preprint arXiv:2311.14760_ , 2023.
  * [61] L. Peng, W. Li, R. Pei, J. Ren, Y. Wang, Y. Cao, and Z.-J. Zha, “Towards realistic data generation for real-world super-resolution,” _arXiv preprint arXiv:2406.07255_ , 2024.
  * [62] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in _International conference on learning representations_ , 2022.
  * [63] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [64] J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in _Proceedings of International conference on machine learning_ , 2022, pp. 12 888–12 900.
  * [65] E. Agustsson and R. Timofte, “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 126–135.
  * [66] S. Gu, A. Lugmayr, M. Danelljan, M. Fritsche, J. Lamour, and R. Timofte, “Div8k: Diverse 8k resolution image dataset,” in _Proceedings of IEEE/CVF international conference on computer vision workshops_ , 2019, pp. 3512–3516.
  * [67] R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, and L. Zhang, “Ntire 2017 challenge on single image super-resolution: Methods and results,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 114–125.
  * [68] X. Wang, K. Yu, C. Dong, and C. C. Loy, “Recovering realistic texture in image super-resolution by deep spatial feature transform,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 606–615.
  * [69] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 4401–4410.
  * [70] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 586–595.
  * [71] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [72] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [73] S. Yang, T. Wu, S. Shi, S. Lao, Y. Gong, M. Cao, J. Wang, and Y. Yang, “Maniqa: Multi-dimension attention network for no-reference image quality assessment,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 1191–1200.
  * [74] J. Ke, Q. Wang, Y. Wang, P. Milanfar, and F. Yang, “Musiq: Multi-scale image quality transformer,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 5148–5157.
  * [75] J. Wang, K. C. Chan, and C. C. Loy, “Exploring clip for assessing the look and feel of images,” in _Proceedings of the AAAI conference on artificial intelligence_ , 2023, pp. 2555–2563.