# Text-guided Explorable Image Super-resolution

Kanchana Vaishnavi Gandikota* Paramanand Chandramouli*
Institute for Vision and Graphics, University of Siegen

###### Abstract

In this paper, we introduce the problem of zero-shot text-guided exploration
of the solutions to open-domain image super-resolution. Our goal is to allow
users to explore diverse, semantically accurate reconstructions that preserve
data consistency with the low-resolution inputs for different large
downsampling factors without explicitly training for these specific
degradations. We propose two approaches for zero-shot text-guided super-
resolution - i) modifying the generative process of text-to-image (_T2I_ )
diffusion models to promote consistency with low-resolution inputs, and ii)
incorporating language guidance into zero-shot diffusion-based restoration
methods. We show that the proposed approaches result in diverse solutions that
match the semantic meaning provided by the text prompt while preserving data
consistency with the degraded inputs. We evaluate the proposed baselines for
the task of extreme super-resolution and demonstrate advantages in terms of
restoration quality, diversity, and explorability of solutions.

retsrzdtigujiserdtf LR inputretsrzdtigujis |  DPS [12] using diffusion model trained on Imagenetretsrzig |  retsrzdtj Oursretsrzdtigujiserdt
---|---|---

’A statue of Walt Disney holding Mickey Mouse hands is showing in front of
Cinderellas castle.’
LR input $\xleftarrow{\hskip 65.04034pt}\text{DPS
\cite[cite]{[\@@bibref{Number}{chung2022diffusion}{}{}]}}\xrightarrow{\hskip
65.04034pt}\xleftarrow{\hskip 65.04034pt}\text{DDNM
\cite[cite]{[\@@bibref{Number}{wang2022zero}{}{}]}}\xrightarrow{\hskip
65.04034pt}$ ’A smiling girl’retsrzdtiigujj ’a smiling woman’retsrzdtiguj ’an
elderly woman’retsrzdti ’smiling elderly woman’

Figure 1:  Text guided image Super-resolution. We explore consistent
reconstructions to image super-resolution problems through text prompts while
achieving perfect data consistency with the given inputs for all solutions.
Shown are a) extreme super-resolution of natural images (top), b) face super-
resolution (bottom), with an upsampling factor of 16. ††footnotetext: *
indicates the authors contributed equally.

##  1 Introduction

The goal of image super-resolution is to recover a high-quality image, given a
low-resolution (LR) observation $\mathbf{y}$,

$\mathbf{y}=\mathbf{A}\mathbf{x}+\mathbf{n},$ (1)

where $\mathbf{A}$, $\mathbf{x}$, and $\mathbf{n}$ represent the down-sampling
operator, ground truth image, and measurement noise respectively. Image super-
resolution is highly ill-posed, especially at large super-resolution factors
with many valid solutions satisfying the data consistency accurately. While
most of the recent state-of-the-art supervised deep networks for super-
resolution [31, 6, 80] recover only a single image from this solution space,
there are also methods utilizing conditional or unconditional generative
models [2, 5, 46, 42, 38, 81], which allow sampling multiple solutions. A few
of these works also allow exploring the solution space, using graphical user
inputs [2] or semantic maps [5]. Yet, even these methods are tailored to
images of specific classes such as faces, or trained for specific super-
resolution factors. On the other hand, natural language provides a simpler and
more intuitive means of conveying semantic concepts. For instance, it is
easier to provide detailed descriptions or convey concepts such as age,
gender, emotion, and race through text rather than graphical inputs alone.
Therefore, a method that guides image super-resolution through text can
greatly aid the exploration of semantically meaningful solutions.

In this paper, we propose for the first time zero-shot open-domain image
super-resolution using simple and intuitive text prompts. Our goal is to
explore via text prompts, diverse and semantically accurate reconstructions
that preserve data consistency with the low-resolution inputs for different
large downsampling factors without explicitly training for these specific
degradations. Towards this goal, we exploit recent advances in text-to-image
(_T2I_ ) generative models [68, 64, 66], contrastive language image
pretraining (CLIP) [61], and unsupervised zero-shot approaches to image
recovery using diffusion-based generative models [12, 74, 81]. We explore two
paradigms for zero-shot text-guided super-resolution. In the first approach,
we adapt recent diffusion-based zero-shot super-resolution approaches to _T2I_
models by appropriately modifying the generative process. We consider recent
state-of-the-art text-to-image diffusion models, open-sourced versions of
DALL-e2 [64], and Imagen [68], and adapt these models for zero-shot super-
resolution using different zero-shot approaches for diffusion-based image
recovery [74, 12, 81]. As these _T2I_ models comprise of a cascade of
diffusion models at different resolutions, we modify the zero-shot approaches
accordingly to deal with multi-stage generation. In the second approach, we
modify an existing diffusion-based zero-shot restoration approach to
incorporate additional language guidance through CLIP. We focus on extreme
image super-resolution with large upscale factors, as this problem is severely
ill-posed, and allows exploration of a larger solution space.

Fig. 1 illustrates the benefits of text guidance in extreme super-resolution.
Existing zero-shot methods such as [12, 77] cannot recover realistic details
when the ground truth has complex content as seen in the super-resolution
results of [12] on challenging input. On the other hand, the use of text
enables the recovery of complex scene content with specific details matching
the text prompt. The use of text also effortlessly improves diversity in
solutions for face super-resolution in terms of age, expression, gender, race,
and other attributes over [12, 81] which recover images with limited
diversity. We evaluate the proposed baselines in terms of realism, fidelity
with low-resolution input, and agreement with text, in several qualitative,
quantitative, and human evaluations. Extensive experimental evaluations
demonstrate the benefit of using zero-shot text guidance in terms of
flexibility, diversity, and explorability of solutions to extreme super-
resolution. Our work opens up a promising direction of developing efficient
tools for text-guided exploration of image recovery.

##  2 Preliminaries

###  2.1 Denoising Diffusion Probabilistic Models (DDPM)

DDPM generative models [27] employ two diffusion processes: _i) A forward
process_ slowly noising a data sample $\boldsymbol{x}_{0}$ into Gaussian
distribution $\mathcal{N}$ in $T$ steps, with the evolution of a sample
$\mathbf{x}_{t}$ at time-step $t$ given by:

$\displaystyle\begin{split}q(\mathbf{x}_{t}|\mathbf{x}_{t-1}):=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I})\\\
i.e.,\quad\mathbf{x}_{t}=\sqrt{1-\beta_{t}}\mathbf{x}_{t-1}+\sqrt{\beta_{t}}\boldsymbol{\epsilon},\quad\boldsymbol{\epsilon}\sim\mathcal{N}(0,\mathbf{I}),\end{split}$
(2)

where ${\\{\beta_{t}\\}^{T}_{t=0}}$ is the noise variance schedule. _ii) A
learned reverse process_ using iterative denoising to generate samples from
the training data distribution $q(\mathbf{x})$ in $T$ steps given by:

$\begin{gathered}p(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0}):=\mathcal{N}(\mathbf{x}_{t-1};\boldsymbol{\mu}_{t}(\mathbf{x}_{t},\mathbf{x}_{0}),\sigma_{t}^{2}\mathbf{I}),\enskip\text{
where,}\\\
\boldsymbol{\mu}_{t}(\mathbf{x}_{t},\mathbf{x}_{0})=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\boldsymbol{\epsilon_{\theta}}(\mathbf{x}_{t},t)\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\right),\enskip\text{
and,}\\\
\mathbf{\sigma}_{t}^{2}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t},\text{\enskip
with}\enskip\alpha_{t}=1-\beta_{t},~{}\text{and}~{}\bar{\alpha}_{t}=\prod_{i=0}^{t}\alpha_{i},\end{gathered}$
(3)

and $\boldsymbol{\epsilon_{\theta}}$ is the learned neural network noise
approximator.

###  2.2 Range-Null Space Decomposition

When there is no measurement noise in (1), i.e.
$\mathbf{y}=\mathbf{A}\mathbf{x}$, pseudoinverse operation
$\mathbf{A^{\dagger}}\mathbf{y}$ produces the minimum norm solution with
perfect data consistency. Any other solution
$(\mathbf{A^{\dagger}}\mathbf{y}+\mathbf{x}_{\delta})$ is also data
consistent, as long as $\mathbf{x}_{\delta}$ lies in the null space of
$\mathbf{A}$. Note that $\mathbf{x}$ can be decomposed as:

$\mathbf{x}\equiv\mathbf{A^{\dagger}}\mathbf{A}\mathbf{x}+(\mathbf{I}-\mathbf{A^{\dagger}}\mathbf{A})\mathbf{x}.$
(4)

The component $(\mathbf{I}-\mathbf{A^{\dagger}}\mathbf{A})\mathbf{x}$ is in
the null space of $\mathbf{A}$, (with
$\mathbf{A}(\mathbf{I}-\mathbf{A^{\dagger}}\mathbf{A})\mathbf{x}\equiv\boldsymbol{0}$).
Given an approximate solution $\bar{\mathbf{x}}$, Eq. 4 can be used to
construct a data consistent solution [72, 2, 80, 81] given by
$\hat{\textbf{x}}$ as,

$\hat{\mathbf{x}}=\mathbf{A^{\dagger}}\mathbf{y}+(\mathbf{I}-\mathbf{A^{\dagger}}\mathbf{A})\bar{\mathbf{x}}.$
(5)

###  2.3 Zero-Shot Restoration using Diffusion Models

We now describe diffusion-based zero-shot restoration methods that we explore
in this paper for text-guided restoration. Given a noisy sample
$\mathbf{x}_{t}$ at step $t$, [73] obtain the clean estimate of
$\mathbf{x}_{0}$ as:

$\displaystyle\mathbf{x}_{0|t}:=\frac{1}{\sqrt{\bar{\alpha}_{t}}}\left(\mathbf{x}_{t}-\boldsymbol{\epsilon_{\theta}}(\mathbf{x}_{t},t)\sqrt{1-\bar{\alpha}_{t}}\right).$
(6)

The works [12, 74] also utilize $\mathbf{x}_{0|t}$ of a similar form by
approximating $p(\mathbf{x}_{0}|\mathbf{x}_{t})$ as a simple Gaussian
distribution. This estimate is used in a guidance function or in a projection
to incorporate measurement consistency in the following methods.
Diffusion Posterior Sampling DPS [12] utilize $\mathbf{x}_{0|t}$ in
reconstruction guidance as:

$\displaystyle\mathbf{x}_{t-1}\leftarrow\mathbf{x}^{\prime}_{t-1}-{\rho_{t}}\nabla_{\mathbf{x}_{t}}\|\mathbf{y}-\mathbf{A}(\mathbf{x}_{0|t})\|_{2}^{2},$
(7)

where, the intermediate estimate at the previous step
$\mathbf{x}^{\prime}_{t-1}$ is obtained using the usual reverse step, and the
gradient of the reconstruction loss for
$\|\mathbf{y}-\mathbf{A}(\mathbf{x}_{0|t})\|_{2}^{2}$ with respect to
${\mathbf{x}_{t}}$ is obtained by differentiating through the diffusion model.
Pseudoinverse-Guided Diffusion Models $\Pi$GDM [74] modify (7) by inverting
the measurement model using pseudoinverse as:

$\displaystyle\mathbf{x}_{t-1}\leftarrow\mathbf{x}^{\prime}_{t-1}-{\rho_{t}}\nabla_{\mathbf{x}_{t}}\|\mathbf{A^{\dagger}}\mathbf{y}-\mathbf{A^{\dagger}}\mathbf{A}(\mathbf{x}_{0|t})\|_{2}^{2},$
(8)

and demonstrate improved reconstructions with a reduced number of diffusion
steps.
Denoising Diffusion Null-Space Models DDNM [81] utilize the range space-null
space decomposition in the reverse diffusion process to rectify the clean
estimate at each step to satisfy data consistency as:

$\hat{\mathbf{x}}_{0|t}:=\mathbf{A^{\dagger}}\mathbf{y}+(\mathbf{I}-\mathbf{A^{\dagger}}\mathbf{A})\mathbf{x}_{0|t}.$
(9)

This rectified data consistent estimate $\hat{\mathbf{x}}_{0|t}$ is used in
subsequent sampling from
$p(\mathbf{x}_{t-1}|\mathbf{x}_{t},\hat{\mathbf{x}}_{0|t})$ in [81]. Compared
to [12, 74], this method is faster as it does not require differentiation
through diffusion model weights.

###  2.4 Text guided Image Generation with Diffusion Models

There are two approaches for text-guided image generation using diffusion
models- training text-conditioned diffusion models, or incorporating text
guidance into unconditional models using vision-language models such as CLIP
[61].

####  2.4.1 Text-to-Image _T2I_ Diffusion Models

We now describe the _T2I_ generative models we employ for text-guided super-
resolution.
DALL E-2 unCLIP  [64] consists of: _i) a diffusion-based prior_ to produce
CLIP image embeddings [61] from encodings of the input prompt, _ii) a
conditional diffusion-based decoder_ to generate images conditioned on CLIP
image embeddings and text prompts in a down-sampled pixel space, and _iii) a
diffusion-based super-resolution module_ to upsample the decoder output into a
high-resolution image.
Imagen  [68] utilizes a pretrained text encoder [62] to generate embeddings
from input text which condition a cascade of conditional diffusion models to
generate images of increasing resolutions. Different from unCLIP, Imagen uses
only text embeddings which are used to condition every stage of image
generation and super-resolution.

We consider unCLIP and Imagen diffusion models with two stages,
$\boldsymbol{\epsilon_{\theta}}$ operating on a down-sampled pixel space at
resolution $64\times 64$, and an upsampling stage
$\boldsymbol{\zeta_{\theta}}$ operating at resolution $256\times 256$.

####  2.4.2 Training-free text-guided generation

The idea of training free guidance [84, 3] is to incorporate desired
conditioning signal $c$ into generation process through appropriate energy
function $E$ which measures the distance between desired condition and clean
estimate $\mathbf{x}_{0|t}$ at every diffusion step:

$\displaystyle\mathbf{x}_{t-1}\leftarrow\mathbf{x}^{\prime}_{t-1}-{\rho_{t}}\nabla_{\mathbf{x}_{t}}E(c,\mathbf{x}_{0|t}).$
(10)

For text-guided generation, $E$ can be defined using the distance between the
text and image embeddings obtained through CLIP text and image encoders.

##  3 Methodology

Given a low-resolution image ${y}$ with known degradation operator $A$, our
goal is to generate data-consistent solutions $u$ whose attributes can be
varied using input text prompts $c$.

$\begin{split}\textit{Data Consistency}:\quad A\hat{u}\equiv{f},\\\
\textit{Semantic Consistency}:\quad\hat{u}\sim q({u|c}),\end{split}$ (11)

where $q({u|c})$ denotes the distribution of images $u$ with semantic meaning
provided by the text prompt ${c}$. To obtain solutions satisfying semantic
meaning provided by text prompt as well as measurement consistency, we explore
the following methods:

  1. 1.

Zero-shot super-resolution using _T2I_ models.

  2. 2.

Incorporating CLIP guidance into zero-shot diffusion-based restoration.

###  3.1 Text Guided Super-resolution using _T2I_ Models

We consider two recent diffusion-based text-to-image (T2I) generative models
that operate in the pixel domain DALL E-2 [64] and Imagen [68]. As these
models employ a multi-stage generation process, first in down-sampled pixel-
space, followed by upsampling stages, we correspondingly modify the sampling
process to incorporate guidance or null-space consistency in both stages of
the generation. Let $\mathbf{c_{1}}$ and $\mathbf{c_{2}}$ denote the
conditioning signals in the two stages, $\boldsymbol{\epsilon_{\theta}}$ and
$\boldsymbol{\zeta_{\theta}}$ denote conditioned diffusion models in the down-
sampled stage, and the super-resolution stage respectively. The current
estimate of the clean image at each step in the first stage and second stage
are respectively given by,

$\displaystyle\mathbf{x}_{{LR}_{0|t}}=\frac{1}{\sqrt{\bar{\alpha}_{t}}}\left(\mathbf{x}_{LR_{t}}-\boldsymbol{\epsilon_{\theta}}(\mathbf{x}_{LR_{t}},t|\mathbf{c}_{1})\sqrt{1-\bar{\alpha}_{t}}\right),$
(12)
$\mathbf{x}_{0|t}=\frac{1}{\sqrt{\bar{\alpha}}_{t}}\left(\mathbf{x}_{t}-\boldsymbol{\zeta_{\theta}}(\mathbf{x}_{t},t|\mathbf{c}_{2})\sqrt{1-\bar{\alpha}_{t}}\right).$
(13)

For Imagen, $\mathbf{c}_{1}$ corresponds to text embeddings from the text
encoder, and $\mathbf{c}_{2}$ contains $\mathbf{x}_{LR}$, in addition to text
embeddings. For unCLIP $\mathbf{c}_{1}$ corresponds to a combination of CLIP
image embeddings produced by the prior model and text embeddings, and
$\mathbf{c}_{2}$ is the output $\mathbf{x}_{LR}$ of the first stage.

We first recover a lower resolution version $\mathbf{x}_{LR}$ by using a
modified measurement $\mathbf{A}_{LR}$ which takes into account the
downsampling operation for text-conditioned diffusion in low resolution. For
the subsequent super-resolution using the model $\zeta_{\theta}$, we consider
the actual measurement operator $\mathbf{A}$. We adapt zero-shot methods
discussed in Sec. 2.3 as follows:

#####  _T2I_ -DPS:

We incorporate reconstruction guidance in both stages. In down-sampled pixel
space, reconstruction guidance is given by

$\displaystyle\mathbf{x}_{{LR}_{t-1}}\leftarrow\mathbf{x}^{\prime}_{{LR}_{t-1}}-{\rho_{t}}\nabla_{\mathbf{x}_{{LR}_{t}}}\|\mathbf{y}-\mathbf{A}_{LR}\mathbf{x}_{{LR}_{0|t}})\|_{2}^{2}.$
(14)

In the second stage, we incorporate reconstruction guidance following the
standard DPS method given in Eq. (9).
_T2I_ -$\mathbf{\Pi}$GDM:  We incorporate pseudoinverse guidance in both
stages. In down-sampled pixel space, pseudoinverse guidance is given by

$\begin{gathered}\mathbf{x}_{{LR}_{t-1}}\leftarrow\mathbf{x}^{\prime}_{{LR}_{t-1}}-{\rho_{t}}\nabla_{\mathbf{x}_{{LR}_{t}}}E(y,\mathbf{A^{\dagger}}_{LR},\mathbf{x}_{{LR}_{0|t}}),\text{
where,}\\\
E(y,\mathbf{A^{\dagger}}_{LR},\mathbf{x}_{{LR}_{0|t}})=\|\mathbf{A^{\dagger}}_{LR}\mathbf{y}-\mathbf{A^{\dagger}}_{LR}\mathbf{A}_{LR}(\mathbf{x}_{{LR}_{0|t}})\|_{2}^{2}.\end{gathered}$
(15)

The second stage incorporates pseudoinverse guidance following the standard
$\Pi$GDM approach given in Eq. (8).
_T2I_ -DDNM:  We impose null-space consistency in both stages. In down-sampled
pixel space, $\mathbf{x}_{{LR}_{0|t}}$ is rectified at each step as

$\hat{\mathbf{x}}_{{LR}_{0|t}}=\mathbf{A^{\dagger}}_{{}_{LR}}\mathbf{y}+(\mathbf{I}-\mathbf{A^{\dagger}}_{LR}\mathbf{A}_{LR})\mathbf{x}_{{LR}_{0|t}}.$
(16)

The second stage has the usual DDNM null space rectification given by Eq. (9).

In the supplementary material, we explore text-guided super-resolution using
Stable diffusion [66]. It is not straightforward to impose null-space
consistency on the intermediate estimates in the Stable Diffusion model
similar to DDNM, as the diffusion process happens in the latent space. We show
in the supplementary that this does not lead to desirable solutions.

###  3.2 CLIP guided Image Super-resolution

We incorporate CLIP guidance into DDNM-based image super-resolution. For a
given image $\mathbf{x}_{0}$ and text prompt $c$, we define an energy function
$E(c,\mathbf{x}_{0})$ which measures similarity between the given image and
text prompt using CLIP model, through cosine similarity between the CLIP image
embeddings and CLIP text embeddings. At each step $t$, we obtain a clean
estimate $\mathbf{x}_{0|t}$ using Eq. (6), and compute the gradient
$\nabla_{\mathbf{x}_{t}}E(c,\mathbf{x}_{0|t})$. We rectify $\mathbf{x}_{0|t}$
to satisfy null space consistency using Eq. (9) to obtain
$\hat{\mathbf{x}}_{0|t}$ and use it to compute an intermediate estimate of the
previous step:

$\hat{\mathbf{x}}_{t-1}\sim
p(\mathbf{x}_{t-1}|\mathbf{x}_{t},\hat{\mathbf{x}}_{0|t}).$ (17)

This intermediate previous step is then modified to incorporate CLIP guidance
as

$\displaystyle\mathbf{x}_{t-1}\leftarrow\hat{\mathbf{x}}_{t-1}-{\rho_{t}}\nabla_{\mathbf{x}_{t}}E(c,\mathbf{x}_{0|t}).$
(18)

While one could also define an energy function combining CLIP guidance with
reconstruction guidance or pseudo-inverse guidance, we observe that this
creates a trade-off between the two objectives of minimizing reconstruction
loss and maximizing similarity with text, and does not provide satisfactory
results.

##  4 Experimental Evaluation

Dataset | SR | Metric | DPS | DDNM |  [68]+DDNM |  [64]+DDNM | CLIP guided |  [68]+$\Pi$GDM
---|---|---|---|---|---|---|---|---
Faces | $8\times$ | LR PSNR(dB)($\uparrow$) | 50.42 | 75.40 | 51.68 | 67.02 | 50.16 | 51.08
NIQE($\downarrow$) | 5.59 | 8.41 | 6.17 | 5.54 | 6.12 | 6.86
$16\times$ | LR PSNR(dB)($\uparrow$) | 51.98 | 80.91 | 51.86 | 66.30 | 51.79 | 52.02
NIQE($\downarrow$) | 5.54 | 9.77 | 5.94 | 5.43 | 6.38 | 6.98
Nocaps | $8\times$ | LR PSNR(dB)($\uparrow$) | 47.01 | 72.94 | 50.34 | 66.33 | 47.86 | 48.75
NIQE($\downarrow$) | 9.66 | 10.27 | 4.62 | 4.88 | 5.37 | 5.10
CLIP($\uparrow$) | 0.2592 | 0.2326 | 0.3102 | 0.3344 | 0.2564 | 0.2811
$16\times$ | LR PSNR (dB)($\uparrow$) | 48.07 | 78.42 | 53.05 | 70.01 | 50.97 | 49.67
NIQE($\downarrow$) | 4.81 | 13.24 | 4.72 | 5.21 | 5.71 | 5.33
CLIP($\uparrow$) | 0.2418 | 0.2162 | 0.3037 | 0.3381 | 0.2517 | 0.2788

Table 1: Quantitative evaluation of baselines DPS[12],[81], and
Imagen[68]+DDNM, [64], CLIP guided DDNM, Imagen[68]+$\Pi$GDM.

We perform experiments on extreme super-resolution of images with large super-
resolution factors $\times 8$, $\times 16$, as this problem is severely ill-
posed and allows exploration of a larger solution space, and is therefore an
ideal setting to test our method on exploring diverse solutions. In contrast,
input imposes stronger constraints on the solutions for super-resolution at
smaller scale factors, limiting their diversity and explorability. We generate
low-resolution images using bicubic downsampling and compute the pseudoinverse
operator $\mathbf{A}^{\dagger}$ using SVD following [38, 81]. To evaluate
consistency between the generated result and the input text prompt, we use
CLIP score [61] using the ViT-B/16 CLIP model. For super resolutions with
large factors, PSNR/SSIM which measure consistency with ground truth are not
effective metrics to measure reconstruction performance, as multiple solutions
can lead to the same low-resolution image. We instead evaluate consistency by
calculating PSNR between the input LR image and the downsampled version of the
solution, as also used in recent challenges for learning super-resolution
space [24, 47]. We evaluate the reconstruction quality in terms of NIQE score
[54]. We also conduct a user study to evaluate how the users rate the
plausibility of reconstruction and semantic consistency with the text prompt.

We perform experiments on face images from CelebA-HQ dataset [35] (a subset of
200 images) and open domain images with captions from NoCaps dataset [1] (a
subset of 100 images). For face image super-resolution, we manually provide
different text prompts with varying personal attributes such as age, gender,
smile, glasses, and curly hair. All our experiments are performed for an
output resolution of $256\times 256$. For text-guided restoration with _T2I_
models, we use the open-source versions of Imagen Deep-Floyd IF [16], unCLIP
Karlo-unCLIP [40], and Stable Diffusion v1.4 [66]. We describe the detailed
settings and hyper-parameters for each method in the supplementary material.
We will make our code publicly available.

The Santa ornament hangs from a Christmas tree branch amongst the colorful
bright lights.

A woman wearing glasses and smiling has a green balloon hat with one purple
balloon flower and one blue balloon flower in front of a whiteboard that says
face painting

Two white tigers, one laying down, the other standing.
AAAAAALRAAAAADPS[12]A [64]+DDNM A [68]+DDNM

Figure 2: Visual comparison of $16\times$ SR on open domain images.

AAALRAAAAAAAAAADPS[12]AAAAAAAAAAADDNM[81]AAA$\xleftarrow{\hskip
56.37271pt}\text{CLIP guidance+DDNM}\xrightarrow{\hskip 56.37271pt}$

$\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{ramesh2022hierarchical}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{saharia2022photorealistic}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}$

Figure 3: Exploring solution for $16\times$ SR on a face image for the
prompts: ‘Elderly smiling man’, ‘ Man with curly hair’, ‘Man with glasses’,

We first compare the performance of proposed baselines against existing
restoration methods, vanilla DDNM and DPS using unconditional diffusion models
trained on CelebA-HQ faces [35] and Imagenet [17]. For this experiment, we
provide a neutral prompt for text-based methods ‘a high-resolution photograph
of a face’ for face images, and utilize images and captions from the NoCaps
dataset for open domain images. The results are summarized in Tab. 1. Among
the zero-shot diffusion-based image restoration methods, DDNM achieves the
best LR PSNR owing to exact consistency imposed by projection operation, where
DPS achieves visually sharper results, which is reflected in the lower NIQE
scores. The text-based reconstruction baselines from Sec. 3 achieve comparable
performance to specialized models trained on faces on neutral prompts.
Further, on open-domain images from the NoCaps dataset, we observe
significantly better image quality in terms of NIQE score, and semantic
matching as measured by CLIP score in comparison to DDNM and DPS using
diffusion model trained on Imagenet. Further, we note that using CLIP guidance
along with DDNM significantly improves both semantic matching with text as
well as NIQE, with a reduction in LR consistency when compared to vanilla DDNM
which yields rather blurred results. All the methods still maintain a good
consistency with the low-resolution measurement, with LR PSNR $>$ 45 dB. In
addition to the baselines in Tab. 1, we also experimented with the baselines
of [9, 31], we did not include these results as these methods do not achieve
the desired level of LR consistency. Among these [31] is trained for the task
of extreme super-resolution, yet, we find that it cannot handle very low-
resolution inputs well.

Fig. 2 qualitatively compares the super-resolution performance of vanilla DPS
with _T2I_ -DDNM approaches using unCLIP and Imagen. On this test set, DDNM
using an unconditional diffusion model trained on Imagenet produces blurry
results. While DPS recovers sharp images in where the image content is
simpler, Figs. 2 and 1 show that it struggles with complex scene content. On
the other hand, the use of powerful _T2I_ models in zero-shot restoration can
recover data consistent solutions matching complex text prompts.
_T2I_ $-\mathbf{\Pi}$GDM and _T2I_ -DPS  Among the _T2I_ model based
approaches, Imagen [68]+$\Pi$GDM has the least CLIP score in Tab. 1. This is
because this method is evaluated without classifier-free guidance (CFG) [26],
whereas DDNM-based methods included CFG. It is known that including CFG in
_T2I_ models improves adherence to text. However, we find that it conflicts
with gradient-based measurement guidance, reducing data consistency in _T2I_
$-\mathbf{\Pi}$GDM and _T2I_ -DPS. The use of CFG requires a lower stepsize
parameter, and as seen in Tab. 2, the use of the same number of diffusion
steps drastically decreases LR-PSNR while improving CLIP score on the NoCaps
dataset. We investigate this further by evaluating [68]+$\Pi$GDM with varying
numbers of reverse diffusion steps on a subset of 25 CelebAHQ images for the
prompt ‘a photograph of a woman with curly hair’. While increasing the number
of diffusion steps improves LR PSNR, it also reduces text adherence. We see
that improving LR PSNR always does not lead to desired results in Fig. 4. In
the supplementary material Sec. 8.1, we include a detailed study and include
results of similar experiments with Imagen [68]+DPS. We observe that
Imagen$-$DPS does not provide the desired level of LR consistency even without
classifier-free guidance. When classifier-free guidance is included, text
adherence improves, with a significant drop in LR PSNR.

##### Exploring solutions through text

Figs. 1 and 3 provide qualitative comparisons of the proposed text-based
baselines with DPS [12] and [81] on face images. While vanilla DPS and DDNM
using a diffusion model trained on faces achieve realistic and data-consistent
solutions, they offer little scope for exploration and produce solutions with
limited diversity. On the other hand, the proposed baselines can recover
images with great diversity in attributes such as curly hair, glasses,
expression, and age. As the ill-posedness of the recovery problem becomes more
severe at high SR factors $(\times 32)$, it is possible to recover a wide
variety of outputs with challenging attributes in age, race, and appearance.
We show more examples in the supplementary material.

| Step size | Steps | CFG | LR PSNR (dB) | CLIP score
---|---|---|---|---|---
Nocaps | 1.0 | 100 | ✗ | 49.67 | 0.2788
0.5 | 100 | ✓ | 24.31 | 0.2923
Faces | 1.0 | 100 | ✗ | 48.705 | 0.2560
0.5 | 100 | ✓ | 30.675 | 0.3011
0.5 | 200 | ✓ | 40.025 | 0.2742
0.5 | 300 | ✓ | 42.431 | 0.2773
| 0.5 | 500 | ✓ | 43.093 | 0.2695

Table 2: Effect of CFG, step size, and number of steps on consistency and text
adherence in _T2I_ -$\Pi$GDM .

LR 28.34dB 28.09dB 34.43dB 41.56dB

LR 27.25dB 41.28dB 41.81dB 41.59dB

LR 20.09dB 43.5dB 44.0dB 44.2dB
100steps 200steps 300steps 500steps

Figure 4: Effect of classifier-free guidance and stepsize in Imagen-$\Pi$GDM.

Figure 5: $\times 16$ SR results with (bottom) and without (top) averaging
trick with $\lambda$$=$$0.4$, and the text prompt ‘a high-res photo of a cat’.

##### Limitation of _T2I_ -DDNM

While _T2I_ -DDNM methods do not face a trade-off between text adherence and
measurement consistency, they can still result in unrealistic images that are
both consistent with the text as well as measurement. This is because any
image hallucinated by the _T2I_ model can still be made consistent with the
measurement through null-space component rectification. We sometimes encounter
this problem in unCLIP[64]-DDNM, when image embedding imagined by prior does
not structurally align with the observation. This can be mitigated to an
extent by modifying the image embedding which conditions the diffusion model.
We introduce an embeddings averaging trick by considering a convex combination
of embedding provided by the prior, and CLIP image embedding of the pseudo-
inverse solution. The extent of this averaging can be controlled by an
additional hyperparameter $\lambda$ which determines the weight of
pseudoinverse embedding. This embeddings averaging trick can improve the
structural consistency of the solution with the input observation, as seen in
Fig. 5.

##### User Study

We performed a user study to evaluate the realism and semantic matching with
text prompts on the results of Imagen-DDNM, unCLIP DDNM, and CLIP-guided DDNM
using an online survey platform. The survey included 50 reconstructions for
each method for the task of $\times 16$ super-resolution along with the
corresponding text prompts. The LR images were generated using 30 face images
from CelebA HQ, and 20 open-domain images from the NoCaps dataset. The users
were asked to evaluate separately whether each reconstruction semantically
matches with input text prompt and whether the solution appears photo-
realistic. The results of this survey are found in Tab. 3. Both Imagen DDNM
and unCLIP DDNM score better in terms of user preference in comparison to
CLIP-guided recovery. This is to be expected, as both [40] and [16] are
powerful _T2I_ models trained over webscale data. Among the three methods,
Imagen DDNM has the best user preference in terms of both semantic matching
with text as well as perceived realism of the recovered solution. This human
evaluation is in contrast with the quantitative evaluation in Tab. 1 where
unCLIP achieves higher semantic matching with text in terms of CLIP score. As
unCLIP was trained to invert CLIP embeddings, it possibly produces images with
higher CLIP scores.

| Imagen-DDNM | unCLIP-DDNM | CLIP guidance
---|---|---|---
| Faces | nocaps | Faces | nocaps | Faces | nocaps
Text-Similarity | 90.89% | 92.88% | 73.20% | 54.42% | 60.64% | 25.38%
Photo-realism | 69.35% | 83.07% | 30.89% | 35.57% | 30.77% | 10.38%

Table 3: Results of user survey on text guided super-resolution

##  5 Related Work

Diverse Solutions to Image Super-resolution Deep networks have become popular
tool for image super-resolution in the past decade[78, 39, 53, 9], where many
state of the art methods [31, 6, 80, 79] employ supervised training to recover
a single solution. Deep learning based solutions which allow sampling multiple
solutions to the ill-posed SR problem also exist, [2, 46, 5, 38, 11, 81, 74,
46, 33, 9, 53, 55]. These methods utilize conditional or unconditional
generative models to sample solutions. Among these conditional generative
model based approaches [2, 46, 69, 42, 60, 32, 47, 49] are trained for the
specific super-resolution task. Zero-shot approaches[53, 38, 11, 81, 74] on
the other hand, utilize image generative models directly for image recovery.

##### Explorable Image Super-resolution

A few prior works attempt to explore solutions space using graphical inputs
[2] or semantic maps [5]. However, they are still restricted to specific
classes e.g. faces, or trained for specific degradation, e.g. specific super-
resolution factors. A recent work [50] also combines text features into super-
resolution network architectures using attention, and trains separate models
for text-guided image super-resolution in an end-to-end manner for each
dataset and super-resolution factor. Yet, this approach cannot handle open
domain images and arbitrary super-resolution factors. To the best of our
knowledge, there is no existing method that allows zero-shot exploration of
solutions space for different restoration tasks on open-domain images through
text.

##### Diffusion Models for Image Super-resolution

One could utilize diffusion models for image super-resolution and other
restoration tasks either by training a conditional diffusion model for
specific tasks [69, 42, 82, 88], or by leveraging diffusion models for zero-
shot image recovery [29, 34, 9, 11, 10, 37, 81, 48, 38, 56]. We are concerned
with the latter variety, which exploit the knowledge of degradation operator
to modify the sampling process. Earlier works [29, 34] adopt Langevin dynamics
for linear inverse problems and incorporate measurement guidance through the
gradient of the least-squares data fidelity term. [9, 11] alternate between a
standard reverse diffusion step and a projection step promoting measurement
consistency. Recent works utilize an estimate of clean sample at each reverse
step to modify the sampling process via a consistency enforcing projection
operation [81, 48, 38] or through guidance through the gradient of the least-
squares data fidelity term [12], or least squares measurement guidance [74] or
both [10]. While projection-based approaches are faster and do not need to
backpropagate through diffusion model weights, they are restricted to inverse
problems where a pseudo-inverse or its approximation can be computed. On the
other hand, gradient-based measurement guidance can be applied for any inverse
problems, or even arbitrary guidance [3, 84], yet it is more expensive as it
requires back-propagation through the diffusion model weights at each
iteration. More recently, [52] adopt diffusion models in a regularization by
denoising (RED) framework, and [91] demonstrate their utility for plug-and-
play image restoration as an effective alternative to the standard Gaussian
denoisers.

#####  _T2I_ Generative Models

Starting from [51] many works proposed different methods to generate images
from text prompts. Initial works trained RNNs [51] and GANs [65, 85, 83, 85,
86, 90, 41, 87, 89] using attention on smaller captioned image datasets [76,
43]. Recent developments in image generation [18, 57, 21], contrastive
learning[7] and large-scale training on massive internet-scale datasets of
paired text prompts and images [71, 70] accelerated research in vision-
language learning [61, 30, 63, 58, 64, 68]. Many recent works train text-to-
image (T2I) models directly on large scale datasets using autoregressive
transformers [63, 19, 22] or diffusion based models[58, 64, 68]. Some of these
T2I models perform diffusion in a low dimensional latent space [25, 75, 66, 4,
20, 28], or in a down-sampled pixel space [64, 68] for computational
efficiency. An alternative paradigm is employed by [23, 15, 14, 44, 45, 13,
59] using text-image encoder CLIP [61] approaches to guide pretrained
generative models [21, 18] towards the input text prompt.

##  6 Discussion, Limitations, and Conclusions

In this paper, we introduced the challenging task of zero-shot open-domain
extreme super-resolution for different scale factors guided by text prompts.
We explored two approaches to deal with this challenge- utilizing pretrained
diffusion-based _T2I_ models for zero-shot recovery and by guiding an image
diffusion model with CLIP for zero-shot diffusion-based restoration. We showed
that the proposed methods improve adherence to input text prompts while
maintaining consistency with the observation. We demonstrated significantly
improved diversity in solutions using the proposed methods. Among these
methods, CLIP guidance is naturally outperformed by the more powerful _T2I_
diffusion model-based methods. Further, we found that gradient-based
reconstruction guidance could be in trade-off with text adherence. Moreover,
the generated results are not always realistic, and it can require several
attempts to realize the desired output, as also observed in text condition
image generation [36]. It must be noted that not every text prompt is
meaningful for every observation. When certain patterns or objects indicated
by text cannot be present in the image, the corresponding objects or patterns
cannot be recovered without severe artifacts or unrealistic images. In this
case, it is not the failure of the approach or the model, rather it can help
the users determine the plausibility of a solution. In view of this, the
evaluation of text-guided restoration is highly subjective, and any
quantitative evaluation in terms of image quality metrics is meaningful only
when the input text prompts are well aligned and plausible for the given
degraded measurement. The performance of all the proposed methods depends on
and is limited by the generative capabilities of the pre-trained generative
model. The method inherits the biases of the data used to train the _T2I_
model. Finally, our work opens up a promising direction of developing
efficient user-guided tools for text-based exploration of image recovery.

## References

  * Agrawal et al. [2019] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson.  Nocaps: Novel object captioning at scale.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 8948–8957, 2019.
  * Bahat and Michaeli [2020] Yuval Bahat and Tomer Michaeli.  Explorable super resolution.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2716–2725, 2020.
  * Bansal et al. [2023] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Universal guidance for diffusion models.  _arXiv preprint arXiv:2302.07121_ , 2023.
  * Bond-Taylor et al. [2022] Sam Bond-Taylor, Peter Hessey, Hiroshi Sasaki, Toby P Breckon, and Chris G Willcocks.  Unleashing transformers: parallel token prediction with discrete absorbing diffusion for fast high-resolution image generation from vector-quantized codes.  In _17th European Conference on Computer Vision_ , pages 170–188. Springer, 2022.
  * Buhler et al. [2020] Marcel C Buhler, Andrés Romero, and Radu Timofte.  Deepsee: Deep disentangled semantic explorative extreme super-resolution.  In _Proceedings of the Asian Conference on Computer Vision_ , 2020.
  * Chan et al. [2021] Kelvin CK Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy.  Glean: Generative latent bank for large-factor image super-resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 14245–14254, 2021.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton.  A simple framework for contrastive learning of visual representations.  _ArXiv_ , abs/2002.05709, 2020.
  * Cheng et al. [2018] Zhiyi Cheng, Xiatian Zhu, and Shaogang Gong.  Low-resolution face recognition.  In _14th Asian Conference on Computer Vision_ , pages 605–621. Springer, 2018.
  * Choi et al. [2021] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon.  Ilvr: Conditioning method for denoising diffusion probabilistic models.  _arXiv preprint arXiv:2108.02938_ , 2021.
  * Chung et al. [2022a] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye.  Improving diffusion models for inverse problems using manifold constraints.  In _Advances in Neural Information Processing Systems_ , 2022a.
  * Chung et al. [2022b] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye.  Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12413–12422, 2022b.
  * Chung et al. [2023] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye.  Diffusion posterior sampling for general noisy inverse problems.  In _International Conference on Learning Representations_ , 2023.
  * Couairon et al. [2022] Guillaume Couairon, Asya Grechka, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Flexit: Towards flexible semantic image translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18270–18279, 2022.
  * Crowson [2021] Katherine Crowson.  CLIP guided diffusion HQ 256x256.  Colab Notebook, 2021.
  * Crowson et al. [2022] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff.  Vqgan-clip: Open domain image generation and editing with natural language guidance.  In _European Conference on Computer Vision_ , 2022.
  * deepfloyd.ai [2023] deepfloyd.ai.  Deepfloyd if: A modular cascaded diffusion model, 2023.  https://github.com/deep-floyd/IF.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. IEEE, 2009.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat GANs on image synthesis.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Ding et al. [2021] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang.  CogView: Mastering text-to-image generation via transformers.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Esser et al. [2021a] Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer.  Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis.  _Advances in Neural Information Processing Systems_ , 34:3518–3532, 2021a.
  * Esser et al. [2021b] Patrick Esser, Robin Rombach, and Björn Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12873–12883, 2021b.
  * Gafni et al. [2022] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman.  Make-a-scene: Scene-based text-to-image generation with human priors.  In _17th European Conference on Computer Vision_ , pages 89–106. Springer, 2022.
  * Galatolo et al. [2021] Federico A Galatolo, Mario GCA Cimino, and Gigliola Vaglini.  Generating images from caption and vice versa via clip-guided generative latent space search.  _arXiv preprint arXiv:2102.01645_ , 2021.
  * Gu et al. [2022a] Jinjin Gu, Haoming Cai, Chao Dong, Jimmy S Ren, Radu Timofte, Yuan Gong, Shanshan Lao, Shuwei Shi, Jiahao Wang, Sidi Yang, et al.  Ntire 2022 challenge on perceptual image quality assessment.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 951–967, 2022a.
  * Gu et al. [2022b] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10696–10706, 2022b.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _Advances in Neural Information Processing Systems_ , pages 6840–6851. Curran Associates, Inc., 2020.
  * Hu et al. [2022] Minghui Hu, Yujie Wang, Tat-Jen Cham, Jianfei Yang, and P.N. Suganthan.  Global context with discrete diffusion in vector quantised modelling for image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11502–11511, 2022.
  * Jalal et al. [2021] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir.  Robust compressed sensing mri with deep generative priors.  In _Advances in Neural Information Processing Systems_ , pages 14938–14954. Curran Associates, Inc., 2021.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pages 4904–4916. PMLR, 2021.
  * Jo et al. [2020] Younghyun Jo, Sejong Yang, and Seon Joo Kim.  Investigating loss functions for extreme super-resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 424–425, 2020.
  * Jo et al. [2021a] Younghyun Jo, Seoung Wug Oh, Peter Vajda, and Seon Joo Kim.  Tackling the ill-posedness of super-resolution through adaptive target generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16236–16245, 2021a.
  * Jo et al. [2021b] Younghyun Jo, Sejong Yang, and Seon Joo Kim.  Srflow-da: Super-resolution using normalizing flow with deep convolutional block.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 364–372, 2021b.
  * Kadkhodaie and Simoncelli [2021] Zahra Kadkhodaie and Eero P Simoncelli.  Stochastic solutions for linear inverse problems using the prior implicit in a denoiser.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Karras et al. [2018] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  In _International Conference on Learning Representations_ , 2018.
  * Karthik et al. [2023] Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata.  If at first you don’t succeed, try, try again: Faithful diffusion-based text-to-image generation by selection, 2023.
  * Kawar et al. [2021] Bahjat Kawar, Gregory Vaksman, and Michael Elad.  SNIPS: Solving noisy inverse problems stochastically.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Kawar et al. [2022] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song.  Denoising diffusion restoration models.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Ledig et al. [2017] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4681–4690, 2017.
  * Lee et al. [2022] Donghoon Lee, Jiseob Kim, Jisu Choi, Jongmin Kim, Minwoo Byeon, Woonhyuk Baek, and Saehoon Kim.  Karlo-v1.0.alpha on coyo-100m and cc15m.  https://github.com/kakaobrain/karlo, 2022.
  * Li et al. [2019] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr.  Controllable text-to-image generation.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Li et al. [2022] Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen.  Srdiff: Single image super-resolution with diffusion probabilistic models.  _Neurocomputing_ , 479:47–59, 2022.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European Conference on Computer Vision_ , 2014.
  * Liu et al. [2021] Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu.  Fusedream: Training-free text-to-image generation with improved clip+ gan space optimization.  _arXiv preprint arXiv:2112.01573_ , 2021.
  * Liu et al. [2023] Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell.  More control for free! image synthesis with semantic diffusion guidance.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 289–299, 2023.
  * Lugmayr et al. [2020] Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Srflow: Learning the super-resolution space with normalizing flow.  In _Proceedings of 16th European Conference on Computer Vision (ECCV)_ , pages 715–732. Springer, 2020.
  * Lugmayr et al. [2021] Andreas Lugmayr, Martin Danelljan, and Radu Timofte.  Ntire 2021 learning the super-resolution space challenge.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 596–612, 2021.
  * Lugmayr et al. [2022a] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11461–11471, 2022a.
  * Lugmayr et al. [2022b] Andreas Lugmayr, Martin Danelljan, Radu Timofte, Kang-wook Kim, Younggeun Kim, Jae-young Lee, Zechao Li, Jinshan Pan, Dongseok Shim, Ki-Ung Song, Jinhui Tang, Cong Wang, and Zhihao Zhao.  Ntire 2022 challenge on learning the super-resolution space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 786–797, 2022b.
  * Ma et al. [2022] Chenxi Ma, Bo Yan, Qing Lin, Weimin Tan, and Siming Chen.  Rethinking super-resolution as text-guided details generation.  _arXiv preprint arXiv:2207.06604_ , 2022.
  * Mansimov et al. [2016] Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov.  Generating images from captions with attention.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2016.
  * Mardani et al. [2023] Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat.  A variational perspective on solving inverse problems with diffusion models.  _arXiv preprint arXiv:2305.04391_ , 2023.
  * Menon et al. [2020] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin.  Pulse: Self-supervised photo upsampling via latent space exploration of generative models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2437–2445, 2020.
  * Mittal et al. [2012] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik.  Making a “completely blind” image quality analyzer.  _IEEE Signal processing letters_ , 20(3):209–212, 2012.
  * Montanaro et al. [2022] Antonio Montanaro, Diego Valsesia, and Enrico Magli.  Exploring the solution space of linear inverse problems with gan latent geometry.  In _2022 IEEE International Conference on Image Processing (ICIP)_ , pages 1381–1385, 2022.
  * Nair et al. [2023] Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Akino, Vishal M. Patel, and Tim K. Marks.  Steered diffusion: A generalized framework for plug-and-play conditional image synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 20850–20860, 2023.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Nichol et al. [2022] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.  GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models.  In _Proceedings of the 39th International Conference on Machine Learning_ , pages 16784–16804. PMLR, 2022.
  * Paiss et al. [2022] Roni Paiss, Hila Chefer, and Lior Wolf.  No token left behind: Explainability-aided image classification and generation.  In _17th European Conference on Computer Vision (ECCV)_ , Berlin, Heidelberg, 2022. Springer-Verlag.
  * Peng and Li [2020] Shichong Peng and Ke Li.  Generating unobserved alternatives: A case study through super-resolution and decompression.  _OpenReview_ , 2020.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 8748–8763. PMLR, 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Reed et al. [2016] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.  Generative adversarial text to image synthesis.  In _Proceedings of The 33rd International Conference on Machine Learning_ , pages 1060–1069. PMLR, 2016.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Rout et al. [2023] Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai.  Solving linear inverse problems provably via posterior sampling with latent diffusion models.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Saharia et al. [2023] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(4):4713–4726, 2023.
  * Schuhmann et al. [2021] Christoph Schuhmann, Robert Kaczmarczyk, Aran Komatsuzaki, Aarush Katta, Richard Vencu, Romain Beaumont, Jenia Jitsev, Theo Coombes, and Clayton Mullis.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  In _NeurIPS Workshop Datacentric AI_. Jülich Supercomputing Center, 2021.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _ArXiv_ , abs/2210.08402, 2022.
  * Schwab et al. [2018] Johannes Schwab, Stephan Antholzer, and Markus Haltmeier.  Deep null space learning for inverse problems: convergence analysis and rates.  _Inverse Problems_ , 35, 2018.
  * Song et al. [2021] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _International Conference on Learning Representations_ , 2021.
  * Song et al. [2023] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz.  Pseudoinverse-guided diffusion models for inverse problems.  In _International Conference on Learning Representations_ , 2023.
  * Tang et al. [2022] Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen.  Improved vector quantized diffusion models.  _arXiv preprint arXiv:2205.16007_ , 2022.
  * Wah et al. [2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge J. Belongie.  The caltech-ucsd birds-200-2011 dataset, 2011.
  * Wang and Isola [2020] Tongzhou Wang and Phillip Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In _International Conference on Machine Learning_ , pages 9929–9939. PMLR, 2020.
  * Wang et al. [2018] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy.  Esrgan: Enhanced super-resolution generative adversarial networks.  In _Proceedings of the European conference on computer vision (ECCV) workshops_ , pages 0–0, 2018.
  * Wang et al. [2021] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan.  Real-esrgan: Training real-world blind super-resolution with pure synthetic data.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1905–1914, 2021.
  * Wang et al. [2022] Yinhuai Wang, Yujie Hu, and Jian Zhang.  Panini-net: Gan prior based degradation-aware feature interpolation for face restoration.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 2576–2584, 2022.
  * Wang et al. [2023] Yinhuai Wang, Jiwen Yu, and Jian Zhang.  Zero-shot image restoration using denoising diffusion null-space model.  In _International Conference on Learning Representations_ , 2023.
  * Whang et al. [2022] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar.  Deblurring via stochastic refinement.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16293–16303, 2022.
  * Xu et al. [2018] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1316–1324, 2018.
  * Yu et al. [2023] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang.  Freedom: Training-free energy-guided conditional diffusion model.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023.
  * Zhang et al. [2017] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas.  StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 5908–5916, 2017.
  * Zhang et al. [2018] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan++: Realistic image synthesis with stacked generative adversarial networks.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(8):1947–1962, 2018.
  * Zhang et al. [2021] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang.  Cross-modal contrastive learning for text-to-image generation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Zhao et al. [2023] Yang Zhao, Tingbo Hou, Yu-Chuan Su, Xuhui Jia, Yandong Li, and Matthias Grundmann.  Towards authentic face restoration with iterative diffusion models and beyond.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7312–7322, 2023.
  * Zhu et al. [2022] Junchen Zhu, Lianli Gao, Jingkuan Song, Yuan-Fang Li, Feng Zheng, Xuelong Li, and Heng Tao Shen.  Label-guided generative adversarial network for realistic image synthesis.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Zhu et al. [2019] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang.  Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5802–5810, 2019.
  * Zhu et al. [2023] Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool.  Denoising diffusion models for plug-and-play image restoration.  In _IEEE Conference on Computer Vision and Pattern Recognition Workshops (NTIRE)_ , 2023.

Supplementary Material

This supplementary material is organized as follows: Sec. 7 provides the
algorithms for text based super-resolution introduced in Sec. 3. We conduct
further experiments demonstrating the trade-off between text adherence and LR
PSNR in gradient-based guidance methods in Sec. 8.1, and show the results of
our experiments with other approaches. We provide detailed experimental
settings in LABEL:{sec:settings} and additional qualitative results in Sec.
10.

##  7 Algorithms for Text based Super-resolution

We provide the algorithms for text-based super-resolution methods described in
Sec. 3. In the following $p_{1}$ and $p_{2}$ denote reverse diffusion
processes in the text-conditioned downsampled stage and the subsequent super-
resolution stage in _T2I_ diffusion models. For Imagen [68], $\mathbf{c}_{1}$
corresponds to text embeddings from text encoder, and $\mathbf{c}_{2}$
contains $\mathbf{x}_{LR}$, in addition to text embeddings. For unCLIP [64]
$\mathbf{c}_{1}$ corresponds to a combination of CLIP image embeddings
produced by the prior model and text embeddings, and $\mathbf{c}_{2}$ is the
output $\mathbf{x}_{LR}$ of the first stage.

$\mathbf{x}_{{LR}_{T}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

for $t=T,...,1$ do

$\mathbf{x}_{{LR}_{0|t}}=\frac{1}{\sqrt{\bar{\alpha}_{t}}}\left(\mathbf{x}_{LR_{t}}-\boldsymbol{\epsilon_{\theta}}(\mathbf{x}_{LR_{t}},t|\mathbf{c}_{1})\sqrt{1-\bar{\alpha}_{t}}\right)$

$\hat{\mathbf{x}}_{{LR}_{0|t}}=\mathbf{A^{\dagger}}_{{}_{LR}}\mathbf{y}+(\mathbf{I}-\mathbf{A^{\dagger}}_{LR}\mathbf{A}_{LR})\mathbf{x}_{{LR}_{0|t}}$

$\mathbf{x}_{{LR}_{t-1}}\sim
p_{1}(\mathbf{x}_{{LR}_{t-1}}|\mathbf{x}_{{{LR}}_{t}},\hat{\mathbf{x}}_{{LR}_{0|t}})$

end for

$\mathbf{x}_{LR}\leftarrow\mathbf{x}_{LR_{0}}$

$\mathbf{x}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

for $t=T,...,1$ do

$\mathbf{x}_{0|t}=\frac{1}{\sqrt{\bar{\alpha}}_{t}}\left(\mathbf{x}_{t}-\boldsymbol{\zeta_{\theta}}(\mathbf{x}_{t},t|\mathbf{c}_{2})\sqrt{1-\bar{\alpha}_{t}}\right)$

$\hat{\mathbf{x}}_{0|t}=\mathbf{A}^{\dagger}\mathbf{y}+(\mathbf{I}-\mathbf{A}^{\dagger}\mathbf{A})\mathbf{x}_{0|t}$

$\mathbf{x}_{t-1}\sim
p_{2}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\hat{\mathbf{x}}_{0|t})$

end for

return $\mathbf{x}_{0}$

Algorithm 1 _T2I_ -DDNM sampling process

$\mathbf{x}_{{LR}_{T}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

for $t=T,...,1$ do

$\mathbf{x}_{{LR}_{0|t}}=\frac{1}{\sqrt{\bar{\alpha}_{t}}}\left(\mathbf{x}_{LR_{t}}-\boldsymbol{\epsilon_{\theta}}(\mathbf{x}_{LR_{t}},t|\mathbf{c}_{1})\sqrt{1-\bar{\alpha}_{t}}\right)$

$\mathbf{x}^{\prime}_{{LR}_{t-1}}\sim
p_{1}(\mathbf{x}_{{LR}_{t-1}}|\mathbf{x}_{{{LR}}_{t}},{\mathbf{x}}_{{LR}_{0|t}})$

$\mathbf{x}_{{LR}_{t-1}}=\mathbf{x}^{\prime}_{{LR}_{t-1}}-{\rho_{t}}\nabla_{\mathbf{x}_{{LR}_{t}}}\|\mathbf{y}-\mathbf{A}_{LR}\mathbf{x}_{{LR}_{0|t}})\|_{2}^{2}$

end for

$\mathbf{x}_{LR}\leftarrow\mathbf{x}_{LR_{0}}$

$\mathbf{x}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

for $t=T,...,1$ do

$\mathbf{x}_{0|t}=\frac{1}{\sqrt{\bar{\alpha}}_{t}}\left(\mathbf{x}_{t}-\boldsymbol{\zeta_{\theta}}(\mathbf{x}_{t},t|\mathbf{c}_{2})\sqrt{1-\bar{\alpha}_{t}}\right)$

$\mathbf{x}^{\prime}_{t-1}\sim
p_{2}(\mathbf{x}_{t-1}|\mathbf{x}_{t},{\mathbf{x}}_{0|t})$

$\mathbf{x}_{t-1}=\mathbf{x}^{\prime}_{t-1}-{\rho_{t}}\nabla_{\mathbf{x}_{t}}\|\mathbf{y}-\mathbf{A}(\mathbf{x}_{0|t})\|_{2}^{2}$

end for

return $\mathbf{x}_{0}$

Algorithm 2 _T2I_ -DPS sampling process

$\mathbf{x}_{{LR}_{T}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

for $t=T,...,1$ do

$\mathbf{x}_{{LR}_{0|t}}=\frac{1}{\sqrt{\bar{\alpha}_{t}}}\left(\mathbf{x}_{LR_{t}}-\boldsymbol{\epsilon_{\theta}}(\mathbf{x}_{LR_{t}},t|\mathbf{c}_{1})\sqrt{1-\bar{\alpha}_{t}}\right)$

$\mathbf{x}^{\prime}_{{LR}_{t-1}}\sim
p_{1}(\mathbf{x}_{{LR}_{t-1}}|\mathbf{x}_{{{LR}}_{t}},{\mathbf{x}}_{{LR}_{0|t}})$

$\mathbf{x}_{{LR}_{t-1}}=\mathbf{x}^{\prime}_{{LR}_{t-1}}-{\rho_{t}}\nabla_{\mathbf{x}_{{LR}_{t}}}\|\mathbf{A^{\dagger}}_{LR}\mathbf{y}-\mathbf{A^{\dagger}}_{LR}\mathbf{A}_{LR}(\mathbf{x}_{{LR}_{0|t}})\|_{2}^{2}$

end for

$\mathbf{x}_{LR}\leftarrow\mathbf{x}_{LR_{0}}$

$\mathbf{x}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

for $t=T,...,1$ do

$\mathbf{x}_{0|t}=\frac{1}{\sqrt{\bar{\alpha}}_{t}}\left(\mathbf{x}_{t}-\boldsymbol{\zeta_{\theta}}(\mathbf{x}_{t},t|\mathbf{c}_{2})\sqrt{1-\bar{\alpha}_{t}}\right)$

$\mathbf{x}^{\prime}_{t-1}\sim
p_{2}(\mathbf{x}_{t-1}|\mathbf{x}_{t},{\mathbf{x}}_{0|t})$

$\mathbf{x}_{t-1}=\mathbf{x}^{\prime}_{t-1}-{\rho_{t}}\nabla_{\mathbf{x}_{t}}\|\mathbf{A^{\dagger}}\mathbf{y}-\mathbf{A^{\dagger}}\mathbf{A}(\mathbf{x}_{0|t})\|_{2}^{2}$

end for

return $\mathbf{x}_{0}$

Algorithm 3 _T2I_ -$\Pi$GDM sampling process

$\mathbf{x}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

for $t=T,...,1$ do

$\mathbf{x}_{0|t}=\frac{1}{\sqrt{\bar{\alpha}}_{t}}\left(\mathbf{x}_{t}-\boldsymbol{\epsilon_{\theta}}(\mathbf{x}_{t},t)\sqrt{1-\bar{\alpha}_{t}}\right)$

$\hat{\mathbf{x}}_{0|t}=\mathbf{A}^{\dagger}\mathbf{y}+(\mathbf{I}-\mathbf{A}^{\dagger}\mathbf{A})\mathbf{x}_{0|t}$

$\hat{\mathbf{x}}_{t-1}\sim
p(\mathbf{x}_{t-1}|\mathbf{x}_{t},\hat{\mathbf{x}}_{0|t})$

$\mathbf{x}_{t-1}\leftarrow\hat{\mathbf{x}}_{t-1}-{\rho_{t}}\nabla_{\mathbf{x}_{t}}E(c,\mathbf{x}_{0|t})$

end for

return $\mathbf{x}_{0}$

Algorithm 4 CLIP guidance with DDNM

##  8 Further Analysis of Text based SR Methods

###  8.1 Gradient based guidance in T2I models

We observed in Tab. 2 of our paper that gradient-based measurement guidance
could be in trade-off with the text adherence. We found that increasing the
number of diffusion steps to 1000 still did not improve LR consistency to the
desired level ($>45$ dB) when we include classifier-free guidance for $\times
16$ super-resolution of faces. We also performed experiments with Imagen+DPS,
on a subset of 25 face images, for the text prompt ’a photograph of a woman
with curly hair’, the results are provided in Tab. 4. While Imagen-$\Pi$GDM
achieved the desired LR consistency without classifier-free guidance, we were
unable to reach the desired level of LR consistency with Imagen$-$DPS, even
without classifier-free guidance. When 1000 diffusion steps are used, we
obtain an LR PSNR of around 40.9 dB. When classifier-free guidance is
included, text adherence improves, with a significant drop in LR PSNR.

| Step size | Steps | CFG | LR PSNR (dB) | CLIP score
---|---|---|---|---|---
Faces$16\times$ | 1.0 | 500 | ✗ | 39.71 | 0.2433
1.0 | 800 | ✗ | 39.83 | 0.2409
1.0 | 1000 | ✗ | 40.94 | 0.2353
0.5 | 500 | ✓ | 28.46 | 0.3130
| 0.5 | 1000 | ✓ | 36.51 | 0.2798

Table 4: Effect of CFG, step size, and number of steps on consistency and text
adherence in ${\emph{T2I}~{}}-DPS$ .

Fig. 6 shows sample results of this experiment for both Imagen$-\Pi$GDM and
Imagen$-$DPS. Imagen$-\Pi$GDM obtains a higher PSNR than Imagen$-$DPS. Figs. 4
and 6 indicate that these methods can also achieve consistency with text at
the cost of lower LR PSNR. We did not find solutions satisfying both the
constraints in 3 independent runs. Fig. 7 shows sample reconstructions of
Imagen$-\Pi$GDM and Imagen$-$DPS when there is no classifier-free guidance for
the same text prompt. We find that Imagen$-\Pi$GDM achieves significantly
higher LR PSNR. While both methods achieve photo-realistic reconstructions,
they fail to adhere to text prompts without classifier-free guidance.

_T2I_ $-\Pi$GDM
---
LR 39.93dB 41.72dB 41.78dB
_T2I_ $-\Pi$GDM
LR 44.43dB 43.28dB 43.19dB
_T2I_ $-$DPS
LR 38.44dB 38.94dB 38.97dB
_T2I_ $-$DPS
LR 39.00dB 39.02dB 38.98dB

Figure 6: Results of multiple runs of Imagen-$\Pi$GDM (top) and Imagen-DPS
(bottom) with 1000 steps including classifier-free-guidance and corresponding
LR PSNR values. Text prompt ‘Photograph of a face of a woman with curly hair’.

Imagen$-\Pi$GDM Imagen$-$DPS

LR 50.59dB 39.78dB
LR 49.65dB 40.37dB
LR 53.12dB 41.44dB
LR 51.59dB 40.65dB
LR 52.12dB 41.68dB

Figure 7: Example reconstructions of Imagen-$\Pi$GDM and Imagen-DPS without
classifier-free-guidance, and corresponding LR PSNR values. Text prompt
‘Photograph of the face of a woman with curly hair’.

##### Influence of image embedding in unCLIP-DDNM

In our paper, we introduced an ‘embeddings averaging trick’ and showed that
this can reduce the mismatch between image embedding and the observation. We
now illustrate the impact of image embedding on unCLIP-DDNM solutions with an
example in Fig. 8. The first row shows the LR image, and the images generated
by unCLIP through the usual DDPM sampling process with no text prompt, i.e.
decoder is conditioned only on image embeddings generated by the prior with
null-text. We fix the random seed throughout this experiment. This embedding
produces an image of a T-shirt. Even starting the reverse diffusion process at
an earlier step initializing with a pseudo-inverse solution retains the same
concept. In the second row, we observe that using the prior image embedding as
it is leads to a solution where the pseudoinverse is super-imposed on an image
of T-shirts. On the other hand, embeddings averaging results align the concept
provided by prior embedding with the pseudoinverse solution. The benefit of
embeddings averaging for text-conditioned super-resolution is further
illustrated in Fig. 9. Even for text prompts that are seemingly incoherent
with the low-resolution measurement such as ‘a photograph of a man’,
embeddings averaging can reduce misalignments and weird backgrounds and
improve photorealism.

unCLIP DDPM
---
LR start 0 null-text start 10 null-text pinv embed
unCLIP DDNM
LR no embed avg embed avg $\lambda=0.4$ embed avg $\lambda=1.0$

Figure 8: Effect of image embedding in unCLIP-DDNM.

No averaging trick

with averaging trick

No averaging trick

with averaging trick

‘A photograph of a face of{*}’ for $*\in$ {a man, a woman}

Figure 9: Embeddings averaging trick improves photorealism in difficult
examples $\times 16$ face super-resolution.

###  8.2 Experiments with Stable Diffusion

##### Using Text inputs in PSLD [67]

Recently [67] extends posterior sampling for inverse problems [12] to latent
diffusion models such as Stable diffusion [66]. While this method also
addresses super-resolution, they consider super-resolution with small SR
factors $2\times,~{}3\times$, and $4\times$, and do not provide text prompts.
Further, as Stablediffusion produces images at higher resolution $512\times
512$, [67] apply the forward operator after upsampling inputs to this
resolution, run posterior sampling, and then downsample images to resolution
$256\times 256$. Therefore, even $4\times$ SR has a higher resolution input.
We instead experimented with a much lower input resolution of $32\times 32$,
which we interpolate to $64\times 64$ and provide as input to PSLD for
$8\times$ SR task. While the original setting of $4\times$ SR with $128\times
128$ resolution provided satisfactory outputs, we could not find a
satisfactory hyper-parameter setting which results in a sharp output image
with very low-resolution inputs.

##### Incorporating Null-space Consistency

We also studied the applicability of Stable diffusion [66] for text-guided
image super-resolution using DDNM. In the case of Stable Diffusion, the
diffusion process happens in the latent space of a variational auto-encoder,
and it is not straightforward to adapt DDNM to this model.

AAALRAAAA 40 DDNM steps AA15 DDNM steps

AAALRAAAAA SD output AAAl After projection

Figure 10: (Top) Stable Diffusion with DDNM using VAE decoder & encoder.
(Bottom) Stable Diffusion results without DDNM in reverse diffusion. Results
for the text prompt ‘a photograph of a face of an elderly woman’ for $\times
8$ SR

We attempted to enforce DDNM consistency in the text-conditioned Stable
diffusion. At each step in the reverse diffusion process, we estimate the
clean latent variable $z_{0}$, and decode it to image space and enforce data
consistency. This data-consistent image is then encoded again to latent space
to resume reverse diffusion. While this approach achieves data-consistency, we
find that it is highly unstable due to the lossy nature of the variational
autoencoder. As the number of inference steps increases, it results in
unrealistic images with heavy artifacts. When the number of inference steps is
low, the artifacts reduce, however the resulting images are blurry, see the
top row of Fig. 10. On the other hand, we observed that using the interpolated
low-resolution image as an initial estimate in the diffusion process can lead
to a totally different image without any guidance or consistency enforcement
in the intermediate steps. While the output of the Stable Diffusion model in
this case is well-aligned with the text prompt, it is not consistent with the
measurement. As a result, the corresponding null-space contents are not
aligned with the pseudo-inverse solution. An example is illustrated in the
bottom row of Fig. 10, where the null space projection adds high-frequency
details of the elderly woman onto the pseudo inverse reconstruction. In
contrast, using the pixel domain _T2I_ models allows us to perform DDNM for
text-guided reverse diffusion in the (down-sampled) pixel space, which easily
generates data-consistent images that are aligned with text.

###  8.3 Noisy SR and Real-world SR:

We provide the results of $8\times$SR with Gaussian noise ($\sigma$=0.05) for
the neutral prompt on CelebA HQ ‘a high-resolution photograph of a face’. The
methods $T2I$-DPS/$\Pi$GDM can directly handle noisy inputs. For $T2I$-DDNM we
use DDNM+ following [81] and report LR PSNR, NIQE andCLIP score in Tab. 5.

Metric | DPS |  [68]+DDNM+ |  [68]+$\Pi$GDM |  [68]+DPS
---|---|---|---|---
LR PSNR | 30.05 | 31.40 | 31.52 | 31.33
NIQE | 5.92 | 7.09 | 6.07 | 5.57
CLIP score | 0.290 | 0.297 | 0.295 | 0.290

Table 5: Results of noisy super-resolution on CelebA HQ for $\times 8$ SR.

Zero-shot diffusion-based methods [12, 81, 74] require the knowledge of the
degradation operator which is not available directly in real-world SR, but
needs to be estimated. When the real LR images are well approximated by our
downsampling operator we can still recover good quality solutions, see Fig. 11
for results on real LR images from the QMUL tiny face dataset [8] using
[68]+DDNM+.

---
LR ‘man’ ‘elderly man’ ‘woman’

Figure 11: Real world super-resolution on sample image from [8].

##  9 Experiment Settings

#####  _T2I_ +DDNM

We investigate the use of Imagen [68] and unCLIP [64]with DDNM sampling face
super-resolution and open-domain image super-resolution. We use the open-
source versions of Imagen Deep-Floyd IF [16] and unCLIP Karlo-unCLIP [40]. We
use these models for super-resolution without further fine-tuning. For
Imagen+DDNM, we use 200 reverse diffusion steps at the resolution of $64\times
64$, followed by 50 reverse diffusion steps at resolution $256\times 256$.
Instead of starting at random noise, we start reverse diffusion at timestamp
$t_{stop}$ as 850, where noisy estimate at $t_{stop}$ is obtained by adding
noise of suitable variance to the pseudo-inverse solution. We utilize
classifier-free guidance with guidance scales of 7.0 and 4.0 respectively in
the first and second stages of the Imagen Deep Floyd model. When compared to
CFG using null-text, we found use of negative prompts {ugly, disfigured,
broken, caricature} improves the quality of reconstruction. The guidance scale
controls the strength of the attribute indicated by text and can be varied to
obtain attributes of varying intensity. We found that not including
classifier-free guidance can still produce realistic results, but the text
adherence reduces significantly.

For experiments with unCLIP+DDNM, we start reverse diffusion at timestamp
$t_{stop}$ as 850 out of the total 1000 steps and use fewer steps between
$[1,t_{stop}]$ in both the reverse diffusion process (80 for text conditioned
decoding and 7 for super-resolution). The noisy estimate at $t_{stop}$ is
obtained by adding noise of suitable variance to the pseudo-inverse solution.
For obtaining the image embeddings we perform 25 reverse diffusion steps using
the prior model. We utilize an optional embeddings average trick when there is
consistent misalignment between measurement and embeddings. For this, we
utilize the CLIP image encoder used to train the unCLIP model. We utilize
classifier-free guidance both for the prior and the unCLIP decoder. The super-
resolution stage is conditioned only on the output of the first stage, and
therefore, there is no classifier-free guidance in the SR module. We use
classifier-free guidance of scale 4.0 in the prior. We find that a high value
of classifier-free guidance scale for text conditioned decoder results in
images which adhere well to text, yet they are often smoothed out, and appear
unrealistic. We found lower values of classifier-free guidance between 1.0 and
3.0 leads to more photo-realistic results. Further, similar to Imagen+DDNM, we
find better reconstructions when negative prompts are used for classifier-free
guidance.

While we provided hyperparameters used in experiments, these are not fixed and
can be varied by the user to explore solutions. Oftentimes, a high number of
reverse diffusion steps (200) is not required for Imagen+DDNM, and most
solutions satisfying both text and low-resolution consistency can be obtained
for fewer steps($\sim$100). All experiments with _T2I_ +DDNM were run on a
single GPU office machine, containing an NVIDIA RTX 3090 GPU.

##### CLIP guidance+DDNM

We investigate the use of CLIP guidance along with DDNM for both face super-
resolution and open-domain image super-resolution. For the task of face super-
resolution, we use a diffusion model trained at a resolution of $256\times
256$ on CelebAMask-HQ dataset from available from SDEdit
repository††https://github.com/ermongroup/SDEdit. For the task of open domain
image super-resolution, we use an unconditional imagenet model trained at a
resolution of $256\times 256$ from Guided
Diffusion††https://github.com/openai/guided-diffusion. We use the ViT-B/16
CLIP model for CLIP guidance. We incorporate DDNM null-space rectification
into CLIP guided generation of [84] using their publicly available
code††https://github.com/vvictoryuki/FreeDoM. We retain their strategy for
learning rate and time-travel strategy when using imagenet model for open
domain super-resolution. We use 300 ddim steps with a learning rate
hyperparameter of $0.05$. Instead of $\ell_{2}$ distance used in [84], we
utilize cosine distance measurement in the energy function for CLIP guidance.
All experiments with CLIP guidance were run on a single GPU office machine,
containing an NVIDIA RTX 3090 GPU.

A man with salt and pepper hair, a tie, and glasses is sitting behind a table
with a sign that says Mexico in front of him.

A large white Lufthansa plane is sitting on the runway.

A red panda is sitting on a tree squeezing its eyes shut and sticking out its
tongue.

A stately grey and white house with an American flag sits among green trees.

A black Land Rover is next to a building.

A gorgeous red headed woodpeker perched on a branch with red flowers.
LR DPS[12] DDNM+CLIP guidance  unCLIP[64]+DDNM  Imagen[68]+DDNM

Figure 12: Visual comparison of $16\times$ SR on open domain images.

The woman is wearing a colorful kimono and carrying a floral print purple
umbrella.

A clock with a gold bezel reading ten minutes to 2-o-clock.

A man in a hat and blue striped shirt carrying wood

Two bicyclists are racing through a blurry background.

A man in a blue shirt and orange shorts ride a motorized scooter.

A smiling girl with glasses and a hat with a clock on it.
LR DPS[12] DDNM+CLIP guidance  unCLIP[64]+DDNM  Imagen[68]+DDNM

Figure 13: Visual comparison of $16\times$ SR on open domain images.

LR input |
---|---
$\xleftarrow{\hskip 65.04034pt}\text{DPS
\cite[cite]{[\@@bibref{Number}{chung2022diffusion}{}{}]}}\xrightarrow{\hskip
65.04034pt}\xleftarrow{\hskip 65.04034pt}\text{DDNM
\cite[cite]{[\@@bibref{Number}{wang2022zero}{}{}]}}\xrightarrow{\hskip
65.04034pt}$
Imagen-DDNM
unCLIP DDNM
retsrzii’a woman’retsrzdtguji ’a woman with curly hair’retsrz ’an elderly
woman’retsrzd ’a woman with glasses’
LR input |
$\xleftarrow{\hskip 65.04034pt}\text{DPS
\cite[cite]{[\@@bibref{Number}{chung2022diffusion}{}{}]}}\xrightarrow{\hskip
65.04034pt}\xleftarrow{\hskip 65.04034pt}\text{DDNM
\cite[cite]{[\@@bibref{Number}{wang2022zero}{}{}]}}\xrightarrow{\hskip
65.04034pt}$
Imagen-DDNM
unCLIP DDNM
retsrzii’a woman’retsrzdtgi ’a woman with glasses’retsrzdtuji ’an Asian
man’retsrzd ’a man with curly hair’

Figure 14: Exploring solutions to $8\times$ face super-resolution

LR input |
---|---
$\xleftarrow{\hskip 65.04034pt}\text{DPS
\cite[cite]{[\@@bibref{Number}{chung2022diffusion}{}{}]}}\xrightarrow{\hskip
65.04034pt}\xleftarrow{\hskip 65.04034pt}\text{DDNM
\cite[cite]{[\@@bibref{Number}{wang2022zero}{}{}]}}\xrightarrow{\hskip
65.04034pt}$
Imagen-DDNM
unCLIP DDNM
retsrzii’a man’retsrzdtgujiguji ’an elderly man’retsrzdujit ’man+checkered
shirt ’retujis’man+striped shirt+glasses’
LR input |
$\xleftarrow{\hskip 65.04034pt}\text{DPS
\cite[cite]{[\@@bibref{Number}{chung2022diffusion}{}{}]}}\xrightarrow{\hskip
65.04034pt}\xleftarrow{\hskip 65.04034pt}\text{DDNM
\cite[cite]{[\@@bibref{Number}{wang2022zero}{}{}]}}\xrightarrow{\hskip
65.04034pt}$
Imagen-DDNM
unCLIP DDNM
retsrzii’a man’retsrzdtgujiguji ’an Asian man’retsrzdujit ’a man with curly
hair ’retujis’an elderly man’

Figure 15: Exploring solutions to $8\times$ face super-resolution.

AAALRAAAAAAAAAADPS[12]AAAAAAAAAAADDNM[81]AAA$\xleftarrow{\hskip
56.37271pt}\text{CLIP guidance+DDNM}\xrightarrow{\hskip 56.37271pt}$

$\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{ramesh2022hierarchical}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{saharia2022photorealistic}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}$
Text prompts: ‘Smiling woman’, ‘ Man’, ‘Elderly man’

AAALRAAAAAAAAAADPS[12]AAAAAAAAAAADDNM[81]AAA$\xleftarrow{\hskip
56.37271pt}\text{CLIP guidance+DDNM}\xrightarrow{\hskip 56.37271pt}$

$\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{ramesh2022hierarchical}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{saharia2022photorealistic}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}$
Text prompts: ‘Smiling woman’, ‘Smiling elderly woman’,‘Smiling woman with
glasses’

AAALRAAAAAAAAAADPS[12]AAAAAAAAAAADDNM[81]AAA$\xleftarrow{\hskip
56.37271pt}\text{CLIP guidance+DDNM}\xrightarrow{\hskip 56.37271pt}$

$\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{ramesh2022hierarchical}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{saharia2022photorealistic}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}$
Text prompts: ‘Smiling woman’, ‘ Smiling child’, ‘Smiling woman with glasses’

Figure 16: Exploring solutions for $16\times$ SR of face images.

AAALRAAAAAAAAAADPS[12]AAAAAAAAAAADDNM[81]AAA$\xleftarrow{\hskip
56.37271pt}\text{CLIP guidance+DDNM}\xrightarrow{\hskip 56.37271pt}$

$\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{ramesh2022hierarchical}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{saharia2022photorealistic}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}$
Text prompts: ‘Smiling woman’, ‘ Woman with glasses’, ‘Smiling woman with
curly hair’

AAALRAAAAAAAAAADPS[12]AAAAAAAAAAADDNM[81]AAA$\xleftarrow{\hskip
56.37271pt}\text{CLIP guidance+DDNM}\xrightarrow{\hskip 56.37271pt}$

$\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{ramesh2022hierarchical}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{saharia2022photorealistic}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}$
Text prompts: ‘Elderly woman’, ‘ Smiling woman with glasses’, ‘Woman with
curly hair’

AAALRAAAAAAAAAADPS[12]AAAAAAAAAAADDNM[81]AAA$\xleftarrow{\hskip
56.37271pt}\text{CLIP guidance+DDNM}\xrightarrow{\hskip 56.37271pt}$

$\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{ramesh2022hierarchical}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}\xleftarrow{\hskip
73.7146pt}\text{\cite[cite]{[\@@bibref{Number}{saharia2022photorealistic}{}{}]}+DDNM}\xrightarrow{\hskip
73.7146pt}$
Text prompts: ‘Boy’, ‘ Smiling man with curly hair’, ‘Elderly man’

Figure 17: Exploring solutions for $16\times$ SR of face images.

LR input $\xleftarrow{\hskip 60.70653pt}\text{\bf
Imagen$-$DDNM}\xrightarrow{\hskip 60.70653pt}\xleftarrow{\hskip
60.70653pt}\text{\bf unCLIP$-$DDNM}\xrightarrow{\hskip 60.70653pt}$

Text prompts: ‘woman’, ‘elderly woman’, ‘elderly smiling man’

Text prompts: ‘smiling child’, ‘woman’, ‘smiling woman with curly hair’

Text prompts: ‘man’, ‘young woman’, ‘woman with glasses’

Text prompts: ‘smiling boy’, ‘smiling woman’, ‘elderly smiling man’

Text prompts: ‘smiling woman’, ‘elderly woman’, ‘man with glasses’

Text prompts: ‘woman’, ‘woman with glasses’, ‘elderly smiling man’

Text prompts: ‘boy’, ‘man with curly hair’, ‘elderly smiling woman’

Text prompts: ‘boy’, ‘man’, ‘elderly smiling man’

Figure 18: Exploring solutions for $32\times$ SR of face images.

##  10 Additional Results

###  10.1 Open domain Image super-resolution

We provide additional qualitative results of open domain text guided super-
resolution in Figs. 12 and 13. While the solutions provided by DPS [12] are
consistent with the measurements, the lack of additional information to guide
the reconstruction results in unsatisfactory results while recovering complex
scene content. The use of CLIP guidance with DDNM improves results over
vanilla DPS, and _T2I_ -DDNM improves adherence to text even further.
Imagen-[68]+DDNM can nearly recover images adhering to complex prompts
involving scene text, for instance, ‘A man with salt and pepper hair, a tie,
and glasses is sitting behind a table with a sign that says Mexico in front of
him.’, or expression ‘A red panda is sitting on a tree squeezing its eyes shut
and sticking out its tongue’. It is also better at compositionality, for
instance for text prompt ‘The woman is wearing a colorful kimono and carrying
a floral print purple umbrella’, Imagen-[68]+DDNM can recover a floral print
purple umbrella, where as unCLIP[64]+DDNM recovers a purple umbrella without
floral patterns, which appear instead on the woman’s head and clothing.

###  10.2 Exploring SR solutions through text

We provide qualitative results on exploring the solutions to face super-
resolution to upsampling factors $8\times,~{}16\times$ and $32\times$. Figs.
14 and 15 show examples of exploring the solutions of $8\times$ super-
resolution. The reconstructions of DPS [12], DDNM [81] and _T2I_ -DDNM methods
are compared. The results demonstrate improved diversity using _T2I_ -DDNM by
exploring a variety of attributes. In contrast, when compared to vanilla DPS
and DDNM which exhibit limited diversity. Similarly in Figs. 16 and 17 and
Fig. 18 we illustrate examples of exploring the solutions of $16\times$ and
$32\times$ super-resolution respectively. We can see in Fig. 18 that severe
ill-posedness of $32\times$ SR task allows a wide range of solutions with
varying personal attributes such as perceived age, gender, race, accessories
etc.