# Initial Steps Towards Tackling High-dimensional Surrogate Modeling for
Neuroevolution Using Kriging Partial Least Squares

Fergal Stapleton<EMAIL_ADDRESS>0000-0002-5347-1573 Hamilton
Institute, Maynooth University,
Naturally Inspired Computation Res. Group,Ireland and  Edgar Galván
<EMAIL_ADDRESS>Dept. of CS, Hamilton Institute, IVI, Maynooth University,
Naturally Inspired Computation Res. Group,Ireland

###### Abstract.

Surrogate-assisted evolutionary algorithms (SAEAs) aim to use efficient
computational models with the goal of approximating the fitness function in
evolutionary computation systems. This area of research has received
significant attention from the specialised research community in different
areas, for example, single and many objective optimisation or dynamic and
stationary optimisation problems. An emergent and exciting area that has
received little attention from the SAEAs community is in neuroevolution. This
refers to the use of evolutionary algorithms in the automatic configuration of
artificial neural network (ANN) architectures, hyper-parameters and/or the
training of ANNs. However, ANNs suffer from two major issues: (a) the use of
highly-intense computational power for their correct training, and (b) the
highly specialised human expertise required to correctly configure ANNs
necessary to get a well-performing network. This work aims to fill this
important research gap in SAEAs in neuroevolution by addressing these two
issues. We demonstrate how one can use a Kriging Partial Least Squares method
in place of the well-known Kriging method, which normally cannot be used in
neuroevolution due to the high dimensionality of the data.

Neuroevolution, Neural Architecture Search, Surrogate-assisted Evolutionary
Algorithms, Kriging, Partial Least Squares

††ccs: Computing methodologies Bio-inspired approaches††ccs: Computing
methodologies Neural networks††ccs: Mathematics of computing Probability and
statistics

##  1\. Introduction

Artificial Neural Networks (ANNs) are a branch of ML algorithms originally
designed to mimic how the human brain works. They have revolutionised the way
researchers and society alike tackle complex and challenging problems, for
example, they are used extensively in the emerging field of autonomous driving
(neurotraj2022fergal, ). However, there are a number of issues associated with
using these ANNs algorithms. Firstly, is in the training of these networks
which often requires heavy computation resources to train ANNs effectively.
Secondly, is the difficulty in designing ANNs’ network architectures a priori,
often leading practitioners to rely on intuition and random search in the
design process.

The first of these issues, regarding the extensive computational time to train
these ANNs, can be addressed using surrogate-assisted evolutionary algorithms
(jin2011surrogate, ). The aim of these models is to sufficiently estimate the
correct fitness value of a potential solution, while reducing the run time of
the overall process, when compared to using the fitness function in the entire
set of potential solutions. This requires that the surrogate model is well-
informed and has a robust model management strategy. The model may converge to
a false optimum (jin2011surrogate, ), otherwise. The second issue, related to
finding a correct architecture to have well-performing ANNs, can be addressed
using neuroevolution (Galvn2021NeuroevolutionID, ). The latter refers to a
branch of evolutionary algorithms (EAs) techniques (eiben_2015_from, )
designed to search and evolve ANNs’ hyperparameters, weights and/or
architectures.

A major challenge remains, however, at the intersection of the aforementioned
two issues, in that it can be difficult to generate input data for a surrogate
model based on the architecture information alone. The evolvable structure, or
genotypic representation, can often result in mismatches when comparing
networks, since network sizes, weights, and parameters may vary. A novel
workaround to this limitation is the work by Stork et al. (pheno_dist_kernel,
), which makes use of a phenotypic distance vector that instead relies on the
behaviour of the network. Stork et al., work demonstrated that
neuroevolutionary approaches need not be tied directly to the genotypic
architecture. It is fair to say, however, that one limitation of their work is
the costly implementation of the Kriging approach used. For example, the
computational cost increases by $O(m^{3})$ where $m$ is the surrogate model
sample size. In order to counteract this issue they subset the surrogate model
samples from a larger archive. Their results were promising for subset sizes
of 25 to 200. However, it is likely for more complex networks, with larger
search spaces, the surrogate may require larger sample sizes in order to be
well-informed.

The main contributions of this work are, firstly, to vastly reduce the
computational requirement normally used in surrogate models based on the well-
known Kriging approach (cressie1990origins, ) by using a Kriging Partial Least
Square (KPLS) approach (bouhlel2016improving, ) and, secondly, to demonstrate
the validity of using the phenotype distance vector on larger network sizes
(pheno_dist_kernel, ).

Table 1. Details of the classification data sets used in our research as well
as preliminary results for an initial surrogate model of 100 networks. A ’-’
indicates the experiment did not complete within 48 hrs.

Dataset | Brief description | Pheno. dist. | KPLS | Kriging
---|---|---|---|---
|  | vector |  |
Iris | 3 classes (Type of Iris plant) | 336 | 00:00:25 | 01:52:38
Yeast | 2 classes (Protein sequence) | 1338 | 00:02:24 | -
Ecoli | 8 classes (Localization site) | 2016 | 00:01:09 | -
Abalone | 2 classes (Number of rings) | 6264 | 00:10:38 | -

##  2\. Methodology

A major impediment in the use of SAEAs in neuroevolution is in the
computational power still required when using well-known SAEAs approaches,
such as the use of the Kriging method. To correctly address this, we use and
store ANNs’ behaviours in vectors, employ a population-based surrogate model
strategy (jin2011surrogate, ), and apply Kriging Partial Least Squares (KPLS)
using the stored vectors (pheno_dist_kernel, ). For full details of the
Kriging and KPLS method see (bouhlel2016improving, ). More specifically, we
use the behaviours of the ANNs encoded in our EAs candidates solutions. These
behaviours refer to the outputs obtained when these ANNs are executed over an
input data set. A similar notion of behaviour (semantics) of individuals
(Uy2011, ) has been widely adopted by the Genetic Programming community
(9308386, ; 10.1145/3520304.3534073, ; galvan2022semantics, ; Uy2011, ;
stapleton2021semantic, ; galvan2020promoting, ). In our work, we design a
surrogate-assisted evolutionary algorithm to approximate the fitness function
of our evolutionary computation system. This is then used in a subset of the
population at each generation. To this end, we use a population-based
surrogate model strategy (jin2011surrogate, ).

It is well-known that stochastic gradient descent (SGD) is the most effective
method to train ANNs (Galvn2021NeuroevolutionID, ). To account for this key
element, we adopt dCGPANN (izzo2017differentiable, ; martens2019neural, ), a
variation of Cartesian Genetic Programming (CGP) (Miller2011, ). dCGPANN
allows us to differentiate weights, and so, we can use SGD instead of evolving
the ANNs’ weights. Each dCGPANN individual representing an ANN is then trained
to a limited number of epochs. This results in using the behaviours of the
networks. That is, we generate a phenotypic distance vector, similar to the
work by Stork et al. (pheno_dist_kernel, ), where the vector contains the
output of the nodes at the final layer for all data instances. We use the KPLS
method, by employing the SMT toolbox (SMT2019, ), to estimate the fitness
values of the subset of the population by feeding these vectors.

##  3\. Experimental setup and results

To test the efficiency of our methods, we use data sets from the UCI machine
learning repository (Dua:2019, ). The third column, in Table 1, denotes the
phenotype vector length. Larger phenotype vectors result in higher dimensional
data for the surrogate model to train on. All experiments were run on Intel
Xeon Gold 6148. Table 1 summarises the preliminary results, showing the time
taken to construct an initial population of 100 fully trained networks that
are then used to construct the surrogate model. A dash symbol (-) indicates
the experiment did not complete within 48 hours. We can see that based on the
last two columns it is infeasible to use the Kriging approach for these
datasets while for the KPLS approach, the times would be more than sufficient.

##  4\. Conclusions

Surrogate-assisted evolutionary algorithms (SAEAs) are an active area of
research, with limited research carried out in the area of neuroevolution, a
method highly popular in the autonomous configuration of ANNs’
hyperparameters, weights and architectures. The use of common SAEAs techniques
such as Kriging cannot be used in neuroevolution due to the high dimensional
data normally used in neuroevolution. We have taken the initial steps to show
that this can be addressed by adopting a correct representation and using the
Kriging Partial Least Square method. Further studies are required to
generalise our findings.

##  5\. Acknowledgments

This publication has emanated from research conducted with the financial
support of Science Foundation Ireland under Grant number 18/CRT/6049. For the
purpose of Open Access, the author has applied a CC BY public copyright
licence to any Author Accepted Manuscript version arising from this
submission.

## References

  * [1] M. A. Bouhlel, N. Bartoli, A. Otsmane, and J. Morlier.  Improving kriging surrogates of high-dimensional design models by partial least squares dimension reduction.  Structural and Multidisciplinary Optimization, 53:935–952, 2016\.
  * [2] M. A. Bouhlel, J. T. Hwang, N. Bartoli, R. Lafage, J. Morlier, and J. R. R. A. Martins.  A python surrogate modeling framework with derivatives.  Advances in Engineering Software, page 102662, 2019.
  * [3] N. Cressie.  The origins of kriging.  Mathematical geology, 22:239–252, 1990.
  * [4] D. Dua and C. Graff.  UCI machine learning repository, 2017.
  * [5] A. E. Eiben and J. Smith.  From evolutionary computation to the evolution of things.  Nature, 521:476–482, 05 2015.
  * [6] E. Galván and P. Mooney.  Neuroevolution in deep neural networks: Current trends and future challenges.  IEEE Transactions on Artificial Intelligence, 2:476–493, 2021.
  * [7] E. Galván and F. Stapleton.  Promoting semantics in multi-objective genetic programming based on decomposition.  arXiv preprint arXiv:2012.04717, 2020.
  * [8] E. Galván and F. Stapleton.  Semantic-based distance approaches in multi-objective genetic programming.  In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pages 149–156. IEEE, 2020.
  * [9] E. Galván, L. Trujillo, and F. Stapleton.  Highlights of semantics in multi-objective genetic programming.  In Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO ’22, page 19–20, New York, NY, USA, 2022. Association for Computing Machinery.
  * [10] E. Galván, L. Trujillo, and F. Stapleton.  Semantics in multi-objective genetic programming.  Applied Soft Computing, 115:108143, 2022.
  * [11] D. Izzo, F. Biscani, and A. Mereta.  Differentiable genetic programming.  In Genetic Programming: 20th European Conference, EuroGP 2017, Amsterdam, The Netherlands, April 19-21, 2017, Proceedings 20, pages 35–51. Springer, 2017.
  * [12] Y. Jin.  Surrogate-assisted evolutionary computation: Recent advances and future challenges.  Swarm and Evolutionary Computation, 1(2):61–70, 2011.
  * [13] M. Märtens and D. Izzo.  Neural network architecture search with differentiable cartesian genetic programming for regression.  In Proceedings of the genetic and evolutionary computation conference companion, pages 181–182, 2019.
  * [14] J. F. Miller.  Cartesian Genetic Programming, pages 17–34.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2011.
  * [15] F. Stapleton and E. Galván.  Semantic neighborhood ordering in multi-objective genetic programming based on decomposition.  In 2021 IEEE Congress on Evolutionary Computation (CEC), pages 580–587. IEEE, 2021.
  * [16] F. Stapleton, E. Galván, G. Sistu, and S. Yogamani.  Neuroevolutionary multi-objective approaches to trajectory prediction in autonomous vehicles.  In Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO ’22, page 675–678, New York, NY, USA, 2022. Association for Computing Machinery.
  * [17] J. Stork, M. Zaefferer, and T. Bartz-Beielstein.  Improving neuroevolution efficiency by surrogate model-based optimization with phenotypic distance kernels.  In International Conference on the Applications of Evolutionary Computation (Part of EvoStar), pages 504–519. Springer, 2019.
  * [18] N. Q. Uy, N. X. Hoai, M. O’Neill, R. I. McKay, and E. Galván-López.  Semantically-based crossover in genetic programming: application to real-valued symbolic regression.  Genetic Programming and Evolvable Machines, 12(2):91–119, 2011\.


#  DiffI2I: Efficient Diffusion Model for Image-to-Image Translation

Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian,
Wenming Yang, Radu Timotfe, Luc Van Gool  Bin Xia and Wenming Yang are with
the Shenzhen International Graduate School, Tsinghua University, Shenzhen
518055, China. Email<EMAIL_ADDRESS><EMAIL_ADDRESS>Yulun Zhang and Luc Van Gool are with the Computer Vision Lab, ETH Zürich,
Zürich 8092, Switzerland. Email<EMAIL_ADDRESS><EMAIL_ADDRESS>Yapeng Tian is with the Department of Computer
Science, The University of Texas at Dallas, Richardson, TX 75080, USA. E-Mail:
<EMAIL_ADDRESS>Yitong Wang, Shiyin Wang, and Xinglong Wu are with
ByteDance Inc, Shenzhen 518055, China. E-Mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Radu Timofte is with
Computer Vision Lab, IFI $\&$ CAIDAS, University of Würzburg, Germany. E-Mail:
<EMAIL_ADDRESS>
###### Abstract

The Diffusion Model (DM) has emerged as the SOTA approach for image synthesis.
However, the existing DM cannot perform well on some image-to-image
translation (I2I) tasks. Different from image synthesis, some I2I tasks, such
as super-resolution, require generating results in accordance with GT images.
Traditional DMs for image synthesis require extensive iterations and large
denoising models to estimate entire images, which gives their strong
generative ability but also leads to artifacts and inefficiency for I2I. To
tackle this challenge, we propose a simple, efficient, and powerful DM
framework for I2I, called DiffI2I. Specifically, DiffI2I comprises three key
components: a compact I2I prior extraction network (CPEN), a dynamic I2I
transformer (DI2Iformer), and a denoising network. We train DiffI2I in two
stages: pretraining and DM training. For pretraining, GT and input images are
fed into CPENS1 to capture a compact I2I prior representation (IPR) guiding
DI2Iformer. In the second stage, the DM is trained to only use the input
images to estimate the same IRP as CPENS1. Compared to traditional DMs, the
compact IPR enables DiffI2I to obtain more accurate outcomes and employ a
lighter denoising network and fewer iterations. Through extensive experiments
on various I2I tasks, we demonstrate that DiffI2I achieves SOTA performance
while significantly reducing computational burdens.

###### Index Terms:

Diffusion model, image-to-image translation, image restoration, inpainting,
super-resolution, motion deblurring, dense prediction.

##  1 Introduction

Image-to image translation (I2I) aims to learn a mapping function between two
distinct image domains. This field has garnered significant attention due to
its versatile application potential and has been implemented in various
domains, including inpainting [1, 2], super-resolution [3, 4], and semantic
segmentation[5, 6]. Currently, deep-learning-based I2I methods have achieved
astonishing results by assimilating robust priors from extensive datasets.

Recently, Diffusion Models (DMs) [7] have demonstrated competitive performance
in image synthesis [8, 9, 10, 11] and I2I, including inpainting [12, 13] and
super-resolution [14, 15, 16]. In particular, DMs are trained by iteratively
denoising the image in a reverse diffusion process. They have demonstrated the
efficacy of principled probabilistic diffusion modeling in achieving high-
quality mapping from randomly sampled Gaussian noise to complex target
distributions, such as realistic images or latent distributions. It is notable
that DMs do not encounter mode-collapse or training instabilities commonly
observed in GANs [13].

However, traditional DMs designed for image synthesis perform diffusion and
reverse processes on the whole images [7] or feature maps [13]. This dense
estimation framework can bring them a strong generation ability while also
requiring a large number of iteration steps (approximately $50-1000$ steps) on
large denoising models, which consumes massive computational resources.
However, some I2I tasks, such as super-resolution and deblurring, are
different from image synthesis tasks, which provide rich guidance information
and need to generate results in accordance with ground-truth images (GT).
Therefore, the traditional DM framework is not suitable for directly being
applied to some I2I tasks, which not only tends to generate unpleasant
artifacts but also leads to low efficiency and slow convergence.

In this paper, we aim to design a simple, effective, and extendable diffusion
framework for image-to-image translation. To this end, we propose DiffI2I,
which performs the diffusion and the reverse process on the compressed I2I
prior representation (IPR) to guide I2I rather than on whole images. This can
not only adequately use the mapping ability of DM but also improve efficiency
and convergence speed for I2I. Specifically, since the transformer can model
long-range pixel dependencies, we adopt the transformer structure to form
Dynamic I2Iformer (DI2Iformer). We train our DiffI2I in two stages: (1) In the
first stage (Fig. 2 (a)), we focus on developing a compact I2I prior
extraction network (CPEN), which can extract compressed IPR from the ground-
truth image. This compact IPR serves as guidance information for the
DI2Iformer. Additionally, we introduce two crucial components for the
DI2Iformer: the Dynamic Feed-Forward Network (DFFN) and Dynamic Attention
(DA). Both DFFN and DA enable the DI2Iformer to effectively utilize the
extracted IPR. Notably, CPEN and DI2Iformer are jointly optimized during this
stage. (2) In the second stage (Fig. 2 (b)), we train the DM to directly
estimate the accurate IPR from the input images. As the IPR is compact and
primarily contributes details for guiding the I2I, our DM can estimate
accurate and effective results with only several iterations on a lightweight
denoising network.

Furthermore, we develop a joint optimization of DM and decoder (i.e.,
DI2Iformer) scheme for DiffI2I. Specifically, traditional DM-based methods
commonly separate the training of DM and decoder. This is because that
traditional DM [13] consumes excessive computational costs and cannot run all
iterations to optimize with the latter decoder jointly. However, a minor
estimated error of DM would cause a performance drop of the decoder.
Therefore, LDM [13] has to use the quantization dictionary to alleviate this
problem. Fortunately, since the DM of our DiffI2I needs a few computational
costs, we can conveniently execute all iterations and derive the estimated IPR
to perform joint optimization with DI2Iformer. As illustrated in Fig. 1,
DiffI2I surpasses the SOTA performance while significantly reducing runtime
compared to other DM-based methods, such as RePaint [12], SRDiff [16], and LDM
[13]. Notably, DiffI2I achieves a remarkable efficiency improvement of
3500$\times$ over RePaint. Our contributions can be summarized into fourfold:

  * •

We introduce DiffI2I, a strong, simple, and efficient DM-based baseline for
I2I. By leveraging powerful mapping capabilities, DiffI2I can estimate a
compact IPR to guide the I2I process. This not only improves the performance
of DM while enhancing efficiency, stability, and convergence speed on I2I.

  * •

To fully harness the IPR for I2I, we propose Dynamic I2Iformer with DFFN and
DA as essential components in the network structure.

  * •

Unlike previous latent DMs that individually optimize denoising networks, we
present joint optimization of the denoising network and DI2Iformer to further
enhance the robustness of error estimation.

  * •

Through extensive experiments, we demonstrate that DiffI2I achieves state-of-
the-art (SOTA) performance in I2I tasks while significantly reducing runtime
compared to other DM-based methods.

A preliminary version of this work [15] has been accepted by ICCV 2023. In the
current work, we introduce additional content in significant ways:

  * •

We explore an efficient and effective DiffI2I framework, deploying it across
various I2I tasks. These applications not only demonstrate the generality of
our approach but also extend the potential breadth.

  * •

We delve deeper into the particulars and enhance the initial version with
extensive analyses, including assessments of resolution robustness, a
comparison with the accelerated DM method, and examinations of the runtime.

  * •

We extend our method for real-world super-resolution, semantic segmentation,
and depth estimation. Comprehensive benchmark experiments demonstrate that our
method maintains its superiority over existing approaches in these I2I tasks.

|  |
---|---|---
(a) Inpainting (Tab. I) |  (b) Real-world super-resolution (Tab. II) |  (c) Single image super-resolution (Tab. III)
|  |
(d) Motion deblurring (Tab. IV) |  (e) Semantic segmentation (Tab. V) |  (f) Depth estimation (Tab. VI)

Figure 1:  Our DiffI2I achieves exceptional state-of-the-art performance in
I2I tasks, all while maintaining an impressive level of efficiency. It is
worth highlighting that existing methods like LDM [13], RePaint [12], SRDiff
[16], and DDP [17] are all grounded in Differential Morphism (DM). Remarkably,
DiffI2I stands out by achieving superior performance while boasting an
astonishing efficiency improvement of 3500 times over RePaint.

##  2 Related Work

###  2.1 Image-to-image Translation

Image-to-image translation (I2I) has been widely studied in computer vision
and has applications in inpainting [1, 18], image super-resolution [3, 19],
image deblurring [20, 21], semantic segmentation [22], and depth estimation
[23]. Up to now, numerous CNN-based methods have emerged and achieved
tremendous witnessed progress. Many of these advances focus on elaborate
network architecture designs and learning schemes, such as residual block [24,
25, 26], GAN [27, 28, 2], attention [29, 30, 31, 32, 33, 34, 35], knowledge
distillation [4], and multi-scale feature aggregation [36, 37, 38].

Recently, transformers, originally designed for natural language processing,
have garnered significant attention in the computer vision community. In
comparison to convolutional neural networks (CNN), transformers excel at
capturing global interactions among various image regions, leading to state-
of-the-art performance. As a result, transformers have found widespread
adoption in numerous vision tasks, including image recognition [39, 40],
segmentation [41, 5, 42, 6, 43], object detection [44, 45], depth estimation
[46, 47], and image restoration [48, 49, 50, 51, 52].

###  2.2 Diffusion Model

Diffusion Models (DMs) [8, 53, 10] have emerged as a powerful approach in the
domain of density estimation and sample quality, achieving impressive results
across different modalities, such as images [13, 15], video [54, 55], audio
[56], and biomedical [57, 58]. In contrast to other generative models like
GANs, DMs leverage parameterized Markov chains to optimize the lower
variational bound on the likelihood function, empowering them to generate
highly accurate target outputs. Considering the recent remarkable achievements
of DMs in their respective domains, harnessing these models to develop I2I
models presents a highly promising pathway to propel the boundaries of I2I
tasks to new heights.

Recently, DMs have gained significant prominence in I2I tasks such as super-
resolution [14], inpainting [59, 12, 60, 13], semantic segmentation [61, 62],
and depth estimation [17, 63]. For example, SR3 [14] and SRDiff [16]
introduced a DM for image super-resolution and demonstrated superior
performance compared to traditional GAN-based methods. Palette [64] took
inspiration from conditional generation models [65] and proposed a conditional
diffusion model for image restoration. Similarly, LDM [13] proposed a novel
approach by applying DM on the latent space to enhance I2I efficiency, while
RePaint designed an improved denoising strategy by resampling iterations in DM
for inpainting. Moreover, Wolleb et al. [62] exploits the DM for medical
segmentation, and DDP [17] extends the denoising diffusion process into the
dense prediction.

Nonetheless, all the aforementioned DMs follow the framework of traditional DM
in image synthesis, which diffuse on the whole images or feature maps. Some
I2I tasks are different from image synthesis, which pose a strong constraint
and need to produce results in accordance with GT. Thus, the strong generative
ability of the traditional DM not only tends to generate artifacts but also be
inefficient and unstable. In this paper, we develop a simple, effective, and
efficient DM framework for I2I, which extends DM on a compact IPR to improve
performance and efficiency.

##  3 Preliminaries: Diffusion Models

A diffusion model (DM) [8] comprises two essential processes: the forward
process (known as the diffusion process), and the reverse inference process.

During the training phase, DM methods define a diffusion process with a fixed
Markov chain that converts an input image $x_{0}$ into Gaussian noise
$x_{T}\sim\mathcal{N}(0,1)$ by $T$ iterations. Each iteration of the diffusion
process can be formulated as follows:

$q\left(x_{t}\mid
x_{t-1}\right)=\mathcal{N}\left(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathbf{I}\right),$
(1)

where $x_{t}$ is the noised image at time-step $t$, $\beta_{t}$ is the
predefined scale factor, and $\mathcal{N}$ represents the Gaussian
distribution. The Eq. (1) can be further simplified as follows:

$q\left(\mathbf{x}_{t}\mid\mathbf{x}_{0}\right)=\mathcal{N}\left(\mathbf{x}_{t};\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0},\left(1-\bar{\alpha}_{t}\right)\mathbf{I}\right),$
(2)

where $\alpha_{t}=1-\beta_{t}$, $\bar{\alpha}_{t}=\prod_{i=0}^{t}\alpha_{i}$.

In the inference stage (reverse process), DM methods sample a Gaussian random
noise map $x_{T}$ and then iteratively and progressively denoise $x_{T}$ until
it converges to a high-quality output $x_{0}$.

$p\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{x}_{0}\right)=\mathcal{N}\left(\mathbf{x}_{t-1};\boldsymbol{\mu}_{t}\left(\mathbf{x}_{t},\mathbf{x}_{0}\right),\sigma_{t}^{2}\mathbf{I}\right),$
(3)

where the mean
$\boldsymbol{\mu}_{t}\left(\mathbf{x}_{t},\mathbf{x}_{0}\right)$ is defined as
$\boldsymbol{\mu}_{t}\left(\mathbf{x}_{t},\mathbf{x}_{0}\right)=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\epsilon\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\right)$
and the $\epsilon$ represents the noise in $x_{t}$, which is the only
uncertain variable in the reverse process. The variance $\sigma_{t}$ is
defined as
$\sigma_{t}^{2}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$. To
estimate $\epsilon$, DMs employ a denoising network
$\epsilon_{\theta}(x_{t},t)$. To train $\epsilon_{\theta}(x_{t},t)$, given a
clean image $x_{0}$, DMs randomly sample a time step $t$ and a noise
$\epsilon\sim\mathcal{N}(0,\mathbf{I})$ to generate noisy images $x_{t}$ using
Eq. (2). Subsequently, DMs optimize the network parameters $\theta$ of
$\epsilon_{\theta}$ following [8]:

$\nabla_{\boldsymbol{\theta}}\left\|\epsilon-\epsilon_{\boldsymbol{\theta}}\left(\sqrt{\bar{\alpha}_{t}}\mathrm{x}_{0}+\epsilon\sqrt{1-\bar{\alpha}_{t}},t\right)\right\|_{2}^{2}.$
(4)

Figure 2: The overview of the proposed DiffI2I, which consists of three main
components: DI2Iformer, CPEN, and a denoising network. DiffI2I has two
training stages: (a) In the first stage, CPENS1 takes the GT image as input
and produces an IPR $\mathbf{Z}$. This IPR guides DI2Iformer to finish I2I.
The training of CPENS1 is optimized jointly with DI2Iformer to enable
DI2Iformer to effectively utilize the extracted IPR. (b) Moving on to the
second stage, we use the strong data estimation abilities of the DM to
estimate the IPR extracted by pretrained CPENS1. Notably, we do not provide
the GT image as input to CPENS2 and the denoising network. During the
inference stage, we employ the reverse process of DM for I2I.  Algorithm 1
DiffI2IS2 Training

Input: Trained DiffI2IS1 (including CPENS1 and DI2Iformer),
$\beta_{t}(t\in[1,T])$.
Output: Trained DiffI2IS2.

1: Init: $\alpha_{t}=1-\beta_{t}$,
$\bar{\alpha}_{T}=\prod_{i=0}^{T}\alpha_{i}$.

2: Init: The DI2Iformer of DiffI2IS2 copies the parameters of trained
DiffI2IS1.

3: for $I_{Input}$, $I_{GT}$ do

4:
$\mathbf{Z}=\operatorname{CPEN_{S1}}(\operatorname{PixelUnshuffle}(\operatorname{Concat}(I_{GT},I_{Input}))).$
( Eq. (5))

5: Diffusion Process:

6: We sample $\mathbf{Z}_{T}$ by
$q\left(\mathbf{Z}_{T}\mid\mathbf{Z}\right)=\mathcal{N}\left(\mathbf{Z}_{T};\sqrt{\bar{\alpha}_{T}}\mathbf{Z},\left(1-\bar{\alpha}_{T}\right)\mathbf{I}\right)$
(i.e., diffusion process. Eq. (10))

7: Reverse Process:

8: $\mathbf{\hat{Z}}_{T}=\mathbf{Z}_{T}$

9:
$\mathbf{D}=\operatorname{CPEN_{S2}}(\operatorname{PixelUnshuffle}(I_{Input}))$
(Eq. (12))

10: for $t=T$ to $1$ do

11:
$\mathbf{\hat{Z}}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{\hat{Z}}_{t}-\epsilon_{\theta}(\operatorname{Concat}(\mathbf{\hat{Z}}_{t},t,\mathbf{D}))\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\right)$
( Eq. (11))

12: end for

13: $\mathbf{\hat{Z}}=\mathbf{\hat{Z}}_{0}$

14: $I_{Output}=\operatorname{DI2Iformer}(I_{Input},\mathbf{\hat{Z}})$

15: Calculate $\mathcal{L}_{diff}$ loss ( Eq. (13)).

16: end for

17: Output the trained model DiffI2IS2.

Algorithm 2 DiffI2IS2 Inference

Input: Trained DiffI2IS2 (including CPENS2 and DI2Iformer),
$\beta_{t}(t\in[1,T])$, input images $I_{Input}$.
Output: Output images $\hat{I}_{Output}$.

1: Init: $\alpha_{t}=1-\beta_{t}$,
$\bar{\alpha}_{T}=\prod_{i=0}^{T}\alpha_{i}$.

2: Reverse Process:

3: Sample $\mathbf{\hat{Z}}_{T}\sim\mathcal{N}(0,1)$

4:
$\mathbf{D}=\operatorname{CPEN_{S2}}(\operatorname{PixelUnshuffle}(I_{Input}))$
( Eq. (12))

5: for $t=T$ to $1$ do

6:
$\mathbf{\hat{Z}}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{\hat{Z}}_{t}-\epsilon_{\theta}(\operatorname{Concat}(\mathbf{\hat{Z}}_{t},t,\mathbf{D}))\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\right)$
( Eq. (11))

7: end for

8: $\mathbf{\hat{Z}}=\mathbf{\hat{Z}}_{0}$

9: $I_{Output}=\operatorname{DI2Iformer}(I_{Input},\mathbf{\hat{Z}})$

10: Output images $I_{Output}$.

##  4 Methodology

Recently, some DM-based methods [16, 14, 12, 17] follow the DM framework of
image synthesis. However, Some I2I tasks have strong constraints to generate
results in accordance with GT, while image synthesis does not. Considering the
difference between I2I and image synthesis, it is necessary to design a DM
framework for I2I in particular. Traditional DMs perform the diffusion process
on entire images or feature maps, which tend to generate artifacts and require
a large denoising network and quite a few iterations. To alleviate the issue,
we introduce an effective and efficient approach for I2I, called DiffI2I,
which utilizes DM to estimate a compressed IPR. The IPR serves as guidance
information for the DI2Iformer to generate outputs. This can not only
effectively leverage the mapping ability of DM to produce precise outputs but
also necessitate a less complex denoising network and fewer iterations.

In this section, we take the image restoration task as an example to introduce
DiffI2I, as illustrated in Fig. 2. For other tasks, such as depth estimation,
we incorporate IPR into the SWIN Transformer [66] in a similar manner. DiffI2I
is primarily composed of three components: a compact IR prior extraction
network (CPEN), a dynamic I2Iformer (DI2Iformer), and a denoising network. The
training of DiffI2I involves two stages: pretraining DiffI2I and training the
diffusion model. In the following sections, we will first discuss the
pretraining of DiffI2I in Sec. 4.1, followed by the details of efficiently
training the diffusion model for DiffI2I in Sec. 4.2.

###  4.1 Pretrain DiffI2I

Before introducing pretraining DiffI2I, we would like to introduce two
networks in the first stage: the compact I2I prior extraction network (CPEN)
and the dynamic I2Iformer (DI2Iformer). The architecture of CPEN is depicted
in Fig. 2 (yellow box), and it primarily consists of stacked residual blocks
and linear layers to extract the compact I2I Prior Representation (IPR).
Subsequently, the extracted IPR is utilized by DI2Iformer to translate images.
The structure of DI2Iformer is represented in Fig. 2 (pink box), which is
composed of dynamic transformer blocks arranged in a Unet shape. These dynamic
transformer blocks consist of two main components: dynamic attention (DA)
shown in Fig. 2 (green box) and dynamic feed-forward network (DFFN)
illustrated in Fig. 2 (nattier blue box). These components leverage the
extracted IPR as dynamic modulation parameters to incorporate I2I guidance
information into feature maps effectively.

In order to solve the problem of artifacts and inefficiency caused by
traditional DM performing diffusion on the entire images, we conducted
diffusion on a compressed I2I prior representation (IPR). In the pretraining
stage, we train CPENS1 to learn an effective and compact IPR to guide
DI2Iformer. To this end, as shown in Fig. 2 (a), we simultaneously train
CPENS1 and DI2Iformer together. To begin, we concatenate the ground-truth (GT)
and input images and apply the PixelUnshuffle operation to downsample them,
resulting in the input for CPENS1. Next, CPENS1 extracts the IPR
$\mathbf{Z}\in\mathbb{R}^{4C^{\prime}}$ using the following process:

$\mathbf{Z}=\operatorname{CPEN_{S1}}(\operatorname{PixelUnshuffle}(\operatorname{Concat}(I_{GT},I_{Input}))).$
(5)

By concatenating GT and input images as input to CPENS1, and then training
CPENS1 and DI2Iformer collaboratively, CPENS1 becomes proficient in deriving
an effective IPR. This learned IPR effectively steers the DI2Iformer in
producing accurate outputs.

For the DA block of DI2Iformer (Fig. 2 (a)), we use channel attention to
efficiently aggregate global information. Specifically, given feature maps
$\mathbf{F}\in\mathbb{R}^{\hat{H}\times\hat{W}\times\hat{C}}$, it is projected
into feature maps
$\mathbf{F^{\prime}}=W_{d}W_{c}\operatorname{Norm}(\mathbf{F})$, where $W_{c}$
is the $1\times 1$ point-wise convolution, $W_{d}$ is the $3\times 3$ depth-
wise convolution, and $\operatorname{Norm}$ denotes layer normalization [67].
Then, we use the SimpleGate ($\operatorname{SG}(.)$) as the activation
function, which is the simplified version of GELU [68] and can be described as
Eq. (6). After that, we use a global average pooling operation to aggregate
the spatial information into channels and multiply global average pooling
results with the input to weight the feature map (Eq. (7)).

$\mathbf{F^{\prime}}_{SG}=\operatorname{SG}(\mathbf{F^{\prime}}_{1},\mathbf{F}^{\prime}_{2})=\mathbf{F}^{\prime}_{1}\odot\mathbf{F^{\prime}}_{2},$
(6)
$\mathbf{F}^{\prime}_{CA}=\mathbf{F^{\prime}}_{SG}\odot\phi(\mathbf{F^{\prime}}_{SG}),$
(7)

where
$\mathbf{F^{\prime}_{1}},\mathbf{F^{\prime}_{2}}\in\mathbb{R}^{\hat{H}\times\hat{W}\times\hat{C}}$
are obtained by equally splitting $\mathbf{F^{\prime}}$. $\phi$ indicates the
global average pooling operation. $\odot$ indicates element-wise
multiplication.
$\mathbf{F^{\prime}}_{SG},\mathbf{F^{\prime}}_{CA}\in\mathbb{R}^{\hat{H}\times\hat{W}\times\hat{C}}$
and $\phi(\mathbf{F^{\prime}}_{SG})\in\mathbb{R}^{\hat{C}}$.

Then, we use the extracted IPR $\mathbf{Z}$ as dynamic modulation parameters
for the DA of the DI2Iformer to guide I2I. The overall process of DA can be
described as follows:

$\hat{\mathbf{F}}=W_{d}\mathbf{F}^{\prime}_{CA}+W_{l}\mathbf{Z}+\mathbf{F},$
(8)

where $W_{l}$ represents linear layer, $\mathbf{F}$ and
$\hat{\mathbf{F}}\in\mathbb{R}^{\hat{H}\times\hat{W}\times\hat{C}}$ are input
and output feature maps respectively, and
$W_{l}\mathbf{Z}\in\mathbb{R}^{\hat{C}}$.

Next, in DFFN, we aggregate local features. To achieve this, we utilize a
$1\times 1$ Conv layer to aggregate information from different channels and a
$3\times 3$ depth-wise Conv layer to aggregate information from spatially
neighboring pixels. Additionally, we also use the SimpleGate (Eq. (6)) as the
activation function as DA does. Similar to DA, we incorporate IPR into DFFN to
guide I2I. The overall process of DFFN can be defined as:

$\mathbf{\hat{F}}=W_{d}^{2}\operatorname{SG}\left(W_{d}^{1}W_{c}\operatorname{Norm}(\mathbf{F})\right)+W_{l}\mathbf{Z}+\mathbf{F}.$
(9)

We jointly train CPENS1 and DI2Iformer to enable integration of IPR extracted
by CPENS1 into the I2I process performed by DI2Iformer. This empowers
DI2Iformer to leverage the guidance information from CPENS1 for enhancing the
quality and accuracy of the I2I.

###  4.2 Diffusion Models for I2I

The training and inference procedures of DiffI2IS2 (Fig. 2 (b)) are
illustrated in Algs. 1 and 2, respectively. Detailed explanations of these
procedures are given in the subsequent parts.

In the second stage (Fig. 2 (b)), we capitalize on the powerful data
estimation capabilities of the DM to estimate IPR without inputting GT images.
To achieve this, we leverage the pre-trained CPENS1 to capture the IPR
$\mathbf{Z}\in\mathbb{R}^{4C^{\prime}}$. Subsequently, we subject $\mathbf{Z}$
to a diffusion process to obtain a sample
$\mathbf{Z}_{T}\in\mathbb{R}^{4C^{\prime}}$. This diffusion process can be
described as follows:

$q\left(\mathbf{Z}_{T}\mid\mathbf{Z}\right)=\mathcal{N}\left(\mathbf{Z}_{T};\sqrt{\bar{\alpha}_{T}}\mathbf{Z},\left(1-\bar{\alpha}_{T}\right)\mathbf{I}\right),$
(10)

where $T$ represents the total number of iterations, $\bar{\alpha}$ and
$\alpha$ are defined in Eqs. (1) and (2) (i.e.,
$\bar{\alpha}_{T}=\prod_{i=0}^{T}\alpha_{i}$).

In the reverse process, DiffI2IS2 can achieve quite accurate estimations using
much fewer iterations and a smaller model size compared to traditional DMs
[13, 12], mainly because DiffI2IS2 diffuses on compact IPR.

Moreover, we introduce joint optimization of DMs and decoders in DiffI2IS2,
which can further improve performance. Traditional DMs incur significant
computational costs due to the large denoising network and the number of
iterations they require. Consequently, they are forced to randomly sample a
time-step $t\in[1,T]$ and optimize the denoising network solely at that time
step (Eqs. (1), (2), (3), and (4)). Unfortunately, this lack of joint training
between the denoising network and the decoder (i.e., DI2Iformer) means that
minor errors in estimations from the denoising network would make the
performance drop of DI2Iformer. In contrast, the lightweight DM framework
employed in DiffI2IS2 offers a notable advantage. It initiates its iterations
from the $T$-th time step (Eq. (10)), facilitating the execution of all
denoising iterations (Eq. (11)) to derive $\mathbf{\hat{Z}}$. After that,
$\mathbf{\hat{Z}}$ can be seamlessly integrated into the joint optimization
process with DI2Iformer.

$\mathbf{\hat{Z}}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{\hat{Z}}_{t}-\epsilon\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\right),$
(11)

where $\epsilon$ indicates the same noise settings in Eq. (3). We employ
CPENS2 and the denoising network to predict the noise using Eq. (3). Notably,
different from traditional DMs in Eq. (3), our DiffI2IS2 eliminates the
variance estimation. This modification proves beneficial, leading to more
accurate IPR estimation and improved overall performance (as discussed in Sec.
6).

Specifically, in the reverse process of DM, we begin by using CPENS2 to derive
a conditional vector $\mathbf{D}\in\mathbb{R}^{4C^{\prime}}$ from input
images:

$\mathbf{D}=\operatorname{CPEN_{S2}}(\operatorname{PixelUnshuffle}(I_{input})),$
(12)

where CPENS2 shares the same structure as CPENS1, with the only difference
being the input dimension of the first convolution. Subsequently, we utilize
the denoising network $\epsilon_{\theta}$ to estimate the noise for each time
step $t$, which is computed as
$\epsilon_{\theta}(\operatorname{Concat}(\mathbf{\hat{Z}}_{t},t,\mathbf{D}))$.
The estimated noise is then utilized in Eq. (11) to calculate
$\mathbf{\hat{Z}}_{t-1}$, which serves as the starting point for the next
iteration.

TABLE I: Quantitative evaluation (FID/LPIPS) for inpainting on benchmark
datasets. The best and second-best performances are highlighted in bold and
underlined, respectively. Notably, the last four methods, shaded in gray,
utilize the diffusion model.

Method | #Params (M) | Places [69] (512$\times$512) | CelebA-HQ [70] (256$\times$256)
---|---|---|---
Narrow Masks | Wide Masks | Narrow Masks | Wide Masks
FID $\downarrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | LPIPS $\downarrow$
EdgeConnect [18] | 22 | 1.3421 | 0.1106 | 8.4866 | 0.1594 | 6.9566 | 0.0922 | 7.8346 | 0.1149
ICT [71] | 150 | - | - | - | - | 8.4977 | 0.0982 | 9.8794 | 0.1196
LaMa [1] | 27 | 0.6340 | 0.0898 | 2.2494 | 0.1339 | 5.3889 | 0.0806 | 5.7023 | 0.0951
LDM [13] | 215 | - | - | 2.1500 | 0.1440 | - | - | - | -
RePaint [12] | 607 | - | - | - | - | 4.7395 | 0.0890 | 5.4881 | 0.1094
DiffIRS2 [15] (Ours) | 26 | 0.4913 | 0.0758 | 1.9788 | 0.1306 | 4.5967 | 0.0769 | 5.1440 | 0.0918
DiffI2IS2 (Ours) | 27 | 0.4452 | 0.0706 | 1.7835 | 0.1261 | 3.8640 | 0.0635 | 4.7013 | 0.0859

GT Input LaMa [1] RePaint [12] DiffIRS2 [15] (Ours) DiffI2IS2 (Ours)

Figure 3: Visual comparison of  inpainting methods. Zoom-in for better
details.

After performing $T$ iterations, we obtain the final estimated IPR
$\mathbf{\hat{Z}}\in\mathbb{R}^{4C^{\prime}}$. We jointly train CPENS2, the
denoising network, and DI2Iformer using the combined loss function
$\mathcal{L}{all}$:

$\mathcal{L}_{diff}=\frac{1}{4C^{\prime}}\sum_{i=1}^{4C^{\prime}}\left|\hat{\mathbf{Z}}(i)-\mathbf{Z}(i)\right|,\mathcal{L}_{all}=\mathcal{L}_{task}+\mathcal{L}_{diff},$
(13)

where $\mathcal{L}_{task}$ is the common task-specific loss. For example, we
use $L_{rec}=\left\|I_{GT}-I_{Output}\right\|_{1}$ as $\mathcal{L}_{task}$ for
deblurring. Moreover, we use the cross-entropy loss as $\mathcal{L}_{task}$
for semantic segmentation.

During the inference stage, only the reverse diffusion process is employed, as
depicted in the lower part of Fig. 2 (b). CPENS2 is responsible for extracting
a conditional vector $\mathbf{D}$ from the input images, and a Gaussian noise
$\mathbf{\hat{Z}}_{T}$ is randomly sampled. The denoising network utilizes
both $\mathbf{\hat{Z}}_{T}$ and $\mathbf{D}$ to estimate the IPR
$\mathbf{\hat{Z}}$ after $T$ iterations. Following this, DI2Iformer employs
the IPR to complete I2I.

##  5 Experiments

###  5.1 Experiment Settings

We apply our method to six I2I tasks separately: (a) inpainting, (b) real-
world super-resolution (Real-World SR), (c) single-image super-resolution, (d)
single-image motion deblurring, (e) semantic segmentation, and (f) depth
estimation. Regarding the diffusion models (DMs) settings in our DiffI2I for
these tasks, we set the total timesteps $T$ to be 4, and the parameter
$\beta_{t}$ in Eq. (11) (where $\alpha_{t}=1-\beta_{t}$) is linearly increased
from $\beta_{1}=0.1$ to $\beta_{T}=0.99$ during the diffusion and reverse
process. The task-specific settings will be explained in detail in the
subsequent sections.

###  5.2 Evaluation on Inpainting

We conduct training and validation of our DiffI2I for inpainting using the
identical settings employed in the LaMa [1]. Specifically, we opt for the Adam
optimizer and set the learning rates to 0.0002 and 0.0001 for training the
DiffI2I and the discriminator networks (used for GAN loss [27]), respectively.
In the training process, all models are iterated over 1M times with a batch
size of 30. Moreover, we train DiffI2I by employing random crops of size
256$\times$256 sourced from the Places-Standard [69] (Places) and CelebA-HQ
[70] datasets, respectively. During testing, we utilize fixed datasets
comprising 2,000 validation samples and 30,000 testing samples from the
CelebA-HQ and Places datasets separately. Furthermore, the validation of
DiffI2IS2 is conducted using crop sizes of 512$\times$512 for the Places
dataset and 256$\times$256 for the CelebA-HQ dataset.

We evaluate our DiffI2IS2 against SOTA inpainting methods, including ICT [71],
LaMa [1], RePaint [12], using metrics such as LPIPS [72] and FID [73] on
validation datasets. Moreover, we compare DiffI2IS2 with our baseline DiffIR
[15]. The quantitative results are presented in Tab. I and Fig. 1 (a). Our
DiffI2IS2 exhibits remarkable superiority over the other methods. In
particular, DiffI2IS2 outperforms the competitive LaMa by a substantial margin
in terms of FID, achieving up to 0.3665 and 1.001 improvements with wide masks
on the Places and CelebA-HQ datasets, respectively. Notably, this improvement
is achieved while maintaining comparable levels of total parameters and
runtime. Moreover, in comparison to the diffusion model (DM) based method
RePaint [13], DiffI2IS2 demonstrates superior performance while utilizing
significantly fewer resources—merely 4.5$\%$ of the parameters and 0.03$\%$ of
the runtime. These results underscore the ability of DiffI2I to effectively
leverage the data estimation capabilities of DM for impressive and efficient
I2I outcomes. Furthermore, our DiffI2I surpasses our baseline model, DiffIR
[15]. This substantial improvement validates the efficacy of our modified
DI2Iformer architecture.

The qualitative results are shown in Fig. 3. Notably, our DiffI2IS2
demonstrates its capacity to generate remarkably realistic and coherent
structures, along with intricate details, outperforming other competitive
inpainting methods.

###  5.3 Evaluation on Real-world Super-Resolution

We train and validate our DiffI2I and baseline DiffIR [15] on real-world SR
using the same configuration as Real-ESRGAN [74]. This involves employing
identical loss functions, as proposed by Real-ESRGAN [28], which incorporate
perceptual and adversarial losses in addition to the fundamental
$\mathcal{L}{1}$ loss. The learning rate for DiffI2IS2 is set to $2\times
10^{-4}$. During optimization, the Adam optimizer is employed with
$\beta{1}=0.9$ and $\beta_{2}=0.99$. For both stages of training, we set the
batch size to $64$, with the input patch size being $64$.

The evaluation of all methods is conducted on the dataset presented in the
Real-World Super-Resolution challenge: NTIRE2020 Track1 and Track2 [75] and
AIM2019 Track2 [76]. Moreover, we assess the performance of our DiffI2I model
on the RealSRSet [77]. As the NTIRE2020 Track1, AIM2019 Track2, and RealSRSet
datasets offer paired validation sets, we employ LPIPS [72], DISTS [78], and
PSNR as evaluation metrics. We compare our DiffI2I and DiffIR [15] with
competitive real-world SR methods, such as BSRGAN [79], Real-ESRGAN [74],
KDSR-GAN [4], and LDM [13].

The quantitative results are shown in Tab. II and Fig. 1 (b). We can see that
DiffI2IS2 outperforms SOTA real-world SR method KDSRS-GAN on LPIPS, DISTS, and
PSNR, consuming similar runtime. In addition, we can see that DiffI2IS2
outperforms classic real-world SR method Real-ESRGAN on LPIPS, DISTS, and
PSNR, only consuming fewer runtime. Furthermore, compared with DM-based LDM
[13], DiffI2IS2 achieve significant 1.84 dB PSNR and 0.182 LPIPS improvement
on NTIRE2020 Track1 dataset consuming only $3.6\%$ runtime.

We have additionally provided visualizations of our results on NTIRE2020
Track2, a dataset captured using smartphones. These qualitative outcomes are
illustrated in Fig. 4. It is apparent that DiffI2IS2 excels in delivering the
most impressive performance among all compared methods.

TABLE II: Quantitative Comparison on 4$\times$ real-world super-resolution
benchmarks. The best and second-best performances are highlighted in bold and
underlined, respectively. Notably, the lower trio of methods, shaded in gray,
incorporate the diffusion model.

Methods | RealSRSet [77] | NTIRE2020 Track1 [75] | AIM2019 Track2 [76]
---|---|---|---
LPIPS$\downarrow$ | DISTS$\downarrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ | DISTS$\downarrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ | DISTS$\downarrow$ | PSNR$\uparrow$
BSRGAN [79] | 0.3648 | 0.1676 | 26.90 | 0.3691 | 0.1368 | 26.75 | 0.4048 | 0.1811 | 24.20
Real-ESRGAN [74] | 0.3629 | 0.1609 | 26.07 | 0.3471 | 0.1326 | 26.40 | 0.3956 | 0.1719 | 23.89
KDSR-GAN [4] | 0.3610 | 0.1627 | 27.18 | 0.3198 | 0.1252 | 27.12 | 0.3758 | 0.1684 | 24.22
LDM [13] | 0.4369 | 0.1982 | 26.37 | 0.4763 | 0.1844 | 25.68 | 0.5082 | 0.1972 | 22.63
DiffIRS2 [15] (Ours) | 0.3527 | 0.1588 | 27.65 | 0.3088 | 0.1131 | 27.31 | 0.3650 | 0.1718 | 23.88
DiffI2IS2 (Ours) | 0.3457 | 0.1448 | 27.77 | 0.2943 | 0.1038 | 27.52 | 0.3476 | 0.1533 | 24.12

Input KDSRS-GAN [4] LDM [13] Real-ESRGAN [74] DiffIRS2 (Ours) DiffI2IS2 (Ours)
Input KDSRS-GAN [4] LDM [13] Real-ESRGAN [74] DiffIRS2 (Ours) DiffI2IS2 (Ours)
---
Input KDSRS-GAN [4] LDM [13] Real-ESRGAN [74] DiffIRS2 (Ours) DiffI2IS2 (Ours)
Input KDSRS-GAN [4] LDM [13] Real-ESRGAN [74] DiffIRS2 (Ours) DiffI2IS2 (Ours)

Figure 4:  Visual comparison of 4$\times$ real-world super-resolution methods.
Zoom-in for better details. TABLE III: Quantitative evaluation ( LPIPS/DISTS).
for single image super-resolution on benchmark datasets. The best and second-
best performances are highlighted in bold and underlined, respectively. The
bottom four methods marked in gray adopt the diffusion model.

Method |  Set14 [80] |  Urban100 [81] |  Manga109 [82] |  General100 [83] |  DIV2K100 [84]
---|---|---|---|---|---
PSNR $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | LPIPS $\downarrow$
SFTGAN [85] | 26.74 | 0.1313 | 24.34 | 0.1343 | 28.17 | 0.0716 | 29.16 | 0.0947 | 28.09 | 0.1331
SRGAN [86] | 26.84 | 0.1327 | 24.41 | 0.1439 | 28.11 | 0.0707 | 29.33 | 0.0964 | 28.17 | 0.1257
ESRGAN [28] | 26.59 | 0.1241 | 24.37 | 0.1229 | 28.41 | 0.0649 | 29.43 | 0.0879 | 28.18 | 0.1154
USRGAN [87] | 27.41 | 0.1347 | 24.89 | 0.1330 | 28.75 | 0.0630 | 30.00 | 0.0937 | 28.79 | 0.1325
SPSR [88] | 26.86 | 0.1207 | 24.80 | 0.1184 | 28.56 | 0.0672 | 29.42 | 0.0862 | 28.18 | 0.1099
BebyGAN [89] | 27.09 | 0.1157 | 25.23 | 0.1096 | 29.19 | 0.0529 | 29.95 | 0.0778 | 28.62 | 0.1022
LDM [13] | 25.62 | 0.2034 | 23.36 | 0.1816 | 25.87 | 0.1321 | 27.17 | 0.1655 | 26.66 | 0.1939
SRDiff [16] | 27.14 | 0.1450 | 25.12 | 0.1379 | 28.67 | 0.0665 | 29.83 | 0.1009 | 28.58 | 0.1293
DiffIRS2 [15] (Ours) | 27.73 | 0.1117 | 26.05 | 0.1007 | 30.32 | 0.0463 | 30.58 | 0.0717 | 29.13 | 0.0871
DiffI2IS2 (Ours) | 27.46 | 0.1094 | 25.99 | 0.0925 | 30.57 | 0.0462 | 30.39 | 0.0666 | 29.26 | 0.0929

GT BebyGAN SRDiff Input DiffIRS2 (Ours) DiffI2IS2 (Ours) GT BebyGAN SRDiff
Input DiffIRS2 (Ours) DiffI2IS2 (Ours)
---
GT BebyGAN SRDiff Input DiffIRS2 (Ours) DiffI2IS2 (Ours) GT BebyGAN SRDiff
Input DiffIRS2 (Ours) DiffI2IS2 (Ours)

Figure 5:  Visual comparison of 4$\times$ image super-resolution methods.
Zoom-in for better details.

###  5.4 Evaluation on Image Super-Resolution

We train and validate our DiffI2I model for image super-resolution.
Specifically, we train DiffI2I on DIV2K dataset [84] comprising 800 images, as
well as the Flickr2K dataset [90] consisting of 2650 images, all aimed at
achieving a $4\times$ super-resolution. During training, we employ a batch
size of 64, utilize input patches of size 64$\times$64, and set the number of
iterations to 1M. To optimize our DiffI2I, we utilize the Adam optimizer with
distinct learning rates: $0.0002$ for the DiffI2I network and $0.0001$ for the
discriminator network (used in adversarial loss).

Subsequently, we subject our DiffI2IS2 model, along with several other state-
of-the-art GAN-based SR methods, to evaluation across five benchmark datasets:
Set5 [91], Set14 [80], General100 [83], Urban100 [81], and DIV2K100 [84]. This
evaluation employs two metrics, LPIPS [72] and PSNR, to gauge the performance
of the models. Tab. III and Fig. 1 (c) show the performance and runtime
comparsion of DiffI2IS2 and DiffIR [15] with SOTA GAN-based SR methods: SFTGAN
[85], SRGAN [86], ESRGAN [28], USRGAN [87], SPSR [88], and BebyGAN [89].
Compared with the competitive SR method BebyGAN, our DiffI2IS2 surpasses it by
LPIPS margin of up to 0.0112 and 0.0093 on General100 and DIV2K100 while
merely consuming $82\%$ runtime. Moreover, it is notable that DiffI2IS2
significantly outperforms the traditional DM-based method LDM while consuming
$3.6\%$ runtime. Additionally, DiffI2IS2 demonstrates a substantial
performance improvement over SRDiff, all while utilizing only $1.3\%$ of the
runtime. Moreover, comparing DiffIRS2, DiffI2IS2 achieves similar performance
with fewer runtime.

The qualitative results are displayed in Fig. 5. Among them, DiffI2IS2 stands
out for its exceptional visual quality, encompassing a plethora of realistic
details. These visual contrasts harmonize with the corresponding quantitative
findings, thus affirming the superiority of DiffI2I. Evidently, DiffI2I
adequately harnesses the potency of DM, facilitating image super-resolution.

TABLE IV: Quantitative evaluation for single image motion deblurring on
benchmark datasets. The best and second-best performances are marked in bold
and underlined, respectively. ∗ represents using the TLC [92].

Method |  GoPro [93] |  HIDE [94]
---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
Xu et al. [95] | 21.00 | 0.741 | - | -
DeblurGAN [96] | 28.70 | 0.858 | 24.51 | 0.871
Nah et al. [93] | 29.08 | 0.914 | 25.73 | 0.874
Zhang et al. [97] | 29.19 | 0.931 | - | -
DeblurGAN-v2 [98] | 29.55 | 0.934 | 26.61 | 0.875
SRN [99] | 30.26 | 0.934 | 28.36 | 0.915
Gao et al. [100] | 30.90 | 0.935 | 29.11 | 0.913
DBGAN [21] | 31.10 | 0.942 | 28.94 | 0.915
MT-RNN [101] | 31.15 | 0.945 | 29.15 | 0.918
DMPHN [102] | 31.20 | 0.940 | 29.09 | 0.924
Suin et al. [103] | 31.85 | 0.948 | 29.98 | 0.930
MIMO-Unet+ [20] | 32.45 | 0.957 | 29.99 | 0.930
IPT [48] | 32.52 | - | - | -
MPRNet [104] | 32.66 | 0.959 | 30.96 | 0.939
Restormer [50] | 32.92 | 0.961 | 31.22 | 0.942
NAFNet [52] | 33.08 | 0.965 | 31.22 | 0.946
NAFNet∗ [52] | 33.71 | 0.967 | 31.32 | 0.941
DiffIRS2 [15] (Ours) | 33.20 | 0.963 | 31.55 | 0.947
DiffI2IS2 (Ours) | 33.25 | 0.964 | 31.56 | 0.947
DiffI2I${}_{S2}^{*}$ (Ours) | 33.85 | 0.968 | 31.42 | 0.943

###  5.5 Evaluation on Image Motion Deblurring

Following previous works in single-image motion deblurring [20, 104, 50], we
train our DiffI2I utilizing only the $\mathcal{L}_{1}$ loss function to ensure
fair comparisons. We train DiffI2I for 300,000 iterations, commencing with an
initial learning rate of $2\times 10^{-4}$, which gradually decreases to
$1\times 10^{-6}$ through the application of the cosine annealing [105]. In
harmony with previous research [50], we progressively the batch size during
training. Specifically, our training process initiates with a patch size of
$128\times 128$ and a batch size of $64$. The patch size and batch size
combination dynamically evolves to $[(160\times 160,40),(192\times
192,32),(256\times 256,16),(320\times 320,8),(384\times 384,8)]$ at the
corresponding iterations of $[92K,156K,204K,240K,276K]$. We train DiffI2I on
GoPro [93] dataset for image motion deblurring and evaluate DiffI2I on two
classic benchmarks (GoPro, HIDE [94]).

We present a comprehensive comparison of DiffI2IS2 with state-of-the-art image
motion deblurring methods, including Restormer [50], MPRNet [104], IPT [48],
NAFNet [52], and our baseline, DiffIR [15]. The quantitative evaluation,
measured in terms of PSNR and SSIM, is detailed in Tab. IV, while the runtime
comparisons are illustrated in Fig. 1 (d). The results underscore the
superiority of DiffI2IS2 over alternative motion deblurring methods.
Specifically, DiffI2IS2 exhibits a remarkable performance advantage,
surpassing IPT and MIMI-Unet+ by a substantial margin of 0.73dB and 0.8dB on
the GoPro dataset, respectively. Furthermore, on both the GoPro and HIDE
datasets, DiffI2IS2 demonstrates a clear advantage over Restormer, achieving
improvements of 0.33dB and 0.34dB, respectively, while only consuming 63$\%$
runtime. Additionally, regardless of whether utilizing TLC [92], our DiffI2IS2
consistently outperforms NAFNet. Moreover, when compared to our own DiffIRS2,
DiffI2IS2 proves its enhanced efficacy by delivering superior results with a
mere 68$\%$ of the runtime. This demonstrates the effectiveness of our
DiffI2I.

###  5.6 Evaluation on Semantic Segmentation

Following the pretraining details of seaformer [6], we adopt the network pre-
trained on ImageNet-1K [106] as the backbone. For semantic segmentation, the
standard BatchNorm [107] layer is replaced by synchronized BatchNorm. Our
model undergoes comprehensive training and evaluation on the ADE20K [108]
dataset, which encompasses 20K/2K/3K images allocated for training,
validation, and testing, encompassing 150 categories. Our method aligns with
the training settings elucidated in segformer [5]. Specifically, we adopt a
160K scheduler and maintain a batch size of 16 throughout the training
process. We employ the AdamW optimizer and regulate the learning rate at
2.5$\times 10^{-4}$ while using a ”poly” learning rate schedule with a factor
of 1.0. To better accommodate the semantic segmentation, we use axial
attention [6] to replace the channel attention in the DA block of DI2Iformer
to obatin the encoder.

We present a comparison of our DiffI2IS2 against previous methods on the
ADE20K validation set, as shown in Tab. V. Additionally, we offer a runtime
comparison with state-of-the-art semantic segmentation methods, illustrated in
Fig. 1 (e). Notably, our DiffI2IS2 outperforms the other methods in terms of
mIOU while demonstrating comparable or lower runtime consumption.

TABLE V: Quantitative evaluation for semantic segmentation on ADE20K [108] val
set. The best and second-best performances are marked in bold and underlined,
respectively.

Backbone | Decoder | Params (M) | mIOU$\uparrow$
---|---|---|---
ConvMLP-S [109] | Semantic FPN [110] | 12.8 | 35.8
EfficientNet [111] | DeepLabV3+ [112] | 17.1 | 36.2
MobileNetV2 [113] | Lite-ASPP [112] | 2.9 | 36.6
MiT-B0 [5] | SegFormer | 3.8 | 37.4
TopFormer-S [5] | Simple Head | 3.1 | 36.5
SeaFormer-S [6] | Light Head | 4.0 | 38.1
DiffI2IS2-S | Light Head | 4.4 | 38.8
ResNet18 [114] | Lite-ASPP [112] | 12.5 | 37.5
ShuffleNetV2-1.5x [115] | DeepLabV3+ [112] | 16.9 | 37.6
MobileNetV2 [113] | DeepLabV3+ [112] | 15.4 | 38.1
MiT-B1 [5] | SegFormer | 13.7 | 41.6
Seaformer-L [6] | Light Head | 14.0 | 42.7
DiffI2IS2-L | Light Head | 12.9 | 43.5

###  5.7 Evaluation on Depth Estimation

We train the DiffI2I using a batch size of 16 and a learning rate of $1\times
10^{-4}$ for a total of $38.4K$ iterations. We employ the AdamW optimizer with
$(\beta_{1},\beta_{2})=(0.9,0.999)$. Moreover, we implement a linear learning
rate warm-up strategy during the initial 30$\%$ of iterations. Additionally,
we incorporate a cosine annealing learning rate decay strategy to further
refine the learning process. The effectiveness of our approach is validated
through training and evaluation on the NYU-Depth-v2 dataset [116]. To better
accommodate the depth estimation and follow binsformer does, we use a pre-
trained Swin transformer as the encoder of DI2Iformer.

We assess the performance of our DiffI2IS2 by conducting a comparison with
existing methods on the NYU-Depth-v2 validation dataset in Tab. VI.
Additionally, we present a runtime comparison with SOTA depth estimation
methods, as depicted in Fig. 1 (f). It is evident from our findings that
DiffI2IS2 not only outperforms the other methods in terms of performance but
also exhibits lower or similar runtime, further establishing its superiority
in depth estimation.

TABLE VI: Quantitative evaluation for depth estimation on NYU-Depth-v2 [116].
The best and second-best performances are marked in bold and underlined,
respectively. The last two methods, shaded in gray, utilize the diffusion
model.

Method | Rel.$\downarrow$ | RMSE$\downarrow$ | log${}_{10}\downarrow$ | $\delta^{1}\uparrow$ | $\delta^{2}\uparrow$ | $\delta^{3}\uparrow$
---|---|---|---|---|---|---
VNL [117] | 0.108 | 0.416 | 0.048 | 0.875 | 0.976 | 0.994
BTS [37] | 0.113 | 0.407 | 0.049 | 0.871 | 0.977 | 0.995
DAV [118] | 0.108 | 0.412 | - | 0.882 | 0.980 | 0.996
PWA [119] | 0.105 | 0.374 | 0.045 | 0.892 | 0.985 | 0.997
TransDepth [120] | 0.106 | 0.365 | 0.045 | 0.900 | 0.983 | 0.996
Adabins [121] | 0.103 | 0.364 | 0.044 | 0.903 | 0.984 | 0.997
P3Depth [122] | 0.104 | 0.356 | 0.043 | 0.898 | 0.981 | 0.996
DepthFormer [46] | 0.096 | 0.339 | 0.041 | 0.921 | 0.989 | 0.998
NeWCRFs [122] | 0.095 | 0.334 | 0.041 | 0.922 | 0.992 | 0.998
BinsFormer [47] | 0.094 | 0.330 | 0.040 | 0.925 | 0.989 | 0.997
DDP [17] | 0.101 | 0.336 | 0.042 | 0.923 | 0.990 | 0.998
DiffI2IS2 | 0.091 | 0.315 | 0.038 | 0.933 | 0.991 | 0.998

##  6 Ablation Study

Efficient diffusion model for I2I. In this section, we undertake a
comprehensive validation of the efficacy of various components within DiffI2I.
These components include the DM, distinct training schemes for DM, as well as
the impact of introducing variance noise into the DM (refer to Tab. VII).

(1) DiffI2IS2-V3 is actually the DiffI2IS2 model used in Tab. I, and DiffI2IS1
represents the first stage pretraining network that utilizes ground-truth
images as inputs. A comparative analysis between DiffI2IS1 and DiffI2IS2-V3
reveals a noteworthy similarity in their LPIPS scores. This observation
underscores the powerful data modeling capabilities of the DM in accurately
predicting IPR.

(2) To provide additional evidence of the DM’s efficacy, we intentionally omit
its usage in DiffI2IS2-V3, obtaining DiffI2IS2-V1. A comparison between
DiffI2IS2-V1 and DiffI2IS2-V3 reveals a marked performance disparity, with
DiffI2IS2-V3 (leveraging DM) notably surpassing DiffI2IS2-V1. This outcome
underscores the pivotal role of the IPR learned by the DM in effectively
guiding DI2Iformer for I2I.

(3) To investigate optimal training schemes for DM, we contrast two distinct
approaches: traditional DM optimization and our proposed joint optimization.
The traditional DM optimization, as utilized in previous works such as [13,
7], necessitates numerous iterations for estimating large images or feature
maps. Consequently, it resorts to random timestep sampling for optimizing the
denoising network, rendering it unable to synchronize optimization with
subsequent components, specifically the decoder (i.e., the DI2Iformer in our
study). In contrast, DiffI2I relies on DM solely to derive a compact one-
dimensional vector IPR. This permits fewer iterations and obtains notably
accurate outcomes. Hence, we adopt a joint optimization scheme that runs all
iterations of the denoising network to obtain IPR, facilitating mutual
optimization with DI2Iformer. A comparative analysis between DiffI2IS2-V2 and
DiffI2IS2-V3 reveals a significant performance disparity, with DiffI2IS2-V3
outperforming DiffI2IS2-V2. This significant improvement underscores the
efficacy of our proposed joint optimization for training DM. This joint
optimization is especially valuable as even minor inaccuracies in DM’s IPR
estimation can detrimentally affect DI2Iformer’s performance. By training DM
and DI2Iformer jointly, we mitigate this challenge.

(4) In traditional DM methods, a common practice involves introducing variance
noise during the reverse DM process (as shown in Eq. (3)) to enhance the
realism of generated images. Diverging from these traditional DMs, our DiffI2I
takes a departure by eliminating the incorporation of noise into the DM
process. We undertake experiments to validate the efficacy of this approach.
Specifically, within the DiffI2IS2-V4, we introduce noise during the reverse
DM process. As shown in Tab. VII, our noise-free DiffI2IS2-V3 outperforms the
noise-inclusive DiffI2IS2-V4 model in terms of performance. This observation
suggests that abstaining from the addition of noise leads to heightened
accuracy in estimating IPR.

TABLE VII:  FID results evaluated on the CelebA-HQ dataset for inpainting. The
evaluation encompassed both performance and runtime, all conducted using
inputs with the resolution of 256x256

Method |  Runtime (ms) | GT | DM | Training Schemes |  Inserting Noise | CelebA-HQ (FID$\downarrow$)
---|---|---|---|---|---|---
Traditional DM Optimization |  Joint Optimization
DiffI2IS1 | 28.1 | ✓ | ✗ | ✗ | ✗ | ✗ | 4.4826
DiffI2IS2-V1 | 28.6 | ✗ | ✗ | ✗ | ✗ | ✗ | 5.2452
DiffI2IS2-V2 | 28.6 | ✗ | ✓ | ✓ | ✗ | ✗ | 5.5221
DiffI2IS2-V3 (Ours) | 28.6 | ✗ | ✓ | ✗ | ✓ | ✗ | 4.7013
DiffI2IS2-V4 | 28.6 | ✗ | ✓ | ✗ | ✓ | ✓ | 4.7796

TABLE VIII:  Various DM loss functions comparison (FID) in inpainting.

Loss functions |  $\mathcal{L}_{diff}$ (Eq. (13)) |  $\mathcal{L}_{2}$ (Eq. (14)) |  $\mathcal{L}_{kl}$ (Eq. (15))
---|---|---|---
CelebA-HQ (FID$\downarrow$) | 4.7013 | 4.7518 | 4.7963

Figure 6: Ablation study of the number of iterations in DM.

The loss functions for DM. We investigate the optimal loss function for
guiding both the denoising network and CPENS2 in accurately estimating the IPR
from input images. In this context, we introduce three distinct loss
functions: (1) We define $\mathcal{L}_{diff}$ and outlined in Eq. (13) to
serve as our optimization criterion. (2) To quantify the extent of estimation
error, we employ the $\mathcal{L}_{2}$ loss (Eq. (14)). (3) We utilizes the
Kullback-Leibler (KL) divergence to gauge the similarity between
distributions, denoted as $\mathcal{L}_{kl}$ (Eq. (15)).

$\mathcal{L}_{2}=\frac{1}{4C^{\prime}}\sum_{i=1}^{4C^{\prime}}\left(\hat{\mathbf{Z}}(i)-\mathbf{Z}(i)\right)^{2},$
(14) $\displaystyle\mathcal{L}_{kl}$
$\displaystyle=\sum_{i=1}^{4C^{\prime}}\mathbf{Z}_{norm}(i)\log\left(\frac{\mathbf{Z}_{norm}(i)}{\hat{\mathbf{Z}}_{norm}(i)}\right),$
(15)

where $\mathbf{Z}$ and $\hat{\mathbf{Z}}\in\mathbb{R}^{4C^{\prime}}$ represent
IPRs extracted by DiffI2IS1 and DiffI2IS2, respectively.
$\hat{\mathbf{Z}}_{norm}$ and $\mathbf{Z}_{norm}\in\mathbb{R}^{4C^{\prime}}$
denote the results of applying the softmax operation to normalize
$\hat{\mathbf{Z}}$ and $\mathbf{Z}$ independently. We employ these three loss
functions on DiffI2IS2 individually, aiming to directly predict accurate IPR
values from input images. Subsequently, we assess their performance in the
inpainting task using the CelebA-HQ dataset. The outcome of these evaluations
is presented in Tab. VIII. Notably, our results demonstrate that
$\mathcal{L}_{diff}$ outperforms both $\mathcal{L}_{2}$ and
$\mathcal{L}_{kl}$.

TABLE IX: Comparison on 4$\times$ real-world SR efficient DM methods

Method | Runtime (ms) | LPIPS$\downarrow$ | DISTS$\downarrow$ | PSNR$\uparrow$
---|---|---|---|---
LDM [13] | 4706.18 | 0.4763 | 0.1844 | 25.68
LDM+DPM-Sovler [123] | 3411.59 | 0.5003 | 0.1871 | 24.31
DiffI2IS2 (Ours) | 170.4 | 0.2943 | 0.1038 | 27.52

Impact of the number of iterations. In this part, we delve into the impact of
the number of iterations in the DiffI2IS2 on its performance. We employ
varying iteration counts within DiffI2IS2 and fine-tune the parameter
$\beta_{t}$ ($\alpha_{t}=1-\beta_{t}$) as depicted in Eq. (10), ensuring that
$\mathbf{Z}$ evolves into Gaussian noise $\mathbf{Z}_{T}\sim\mathcal{N}(0,1)$
through the diffusion process (i.e., as $\bar{\alpha}_{T}\rightarrow 0$). The
results are illustrated in Fig. 6. Remarkably, as the number of iterations
reaches 3, the performance of DiffI2IS2 experiences a substantial enhancement.
Beyond 4 iterations, DiffI2IS2 stabilizes, effectively reaching its upper-
performance limit. Furthermore, it is evident that our DiffI2IS2 exhibits a
swifter convergence rate compared to traditional DMs, which typically
necessitate over 200 iterations. This advantageous acceleration arises from
our focused application of DM on the IPR, a compressed one-dimensional vector.

Comparison with efficient DM methods. We conduct a comparative analysis
involving LDM [13], which is an efficient DM method and is accelerated by DDIM
[124]. Additionally, we leverage the DPM-Solver [123] to optimize the
efficiency of LDM. It is officially adopted for accelerating both LDM and
Stable Diffusion. As illustrated in Tab. IX, our proposed DiffI2I framework,
tailored for I2I, demonstrates remarkable superiority over DPM-Solver,
showcasing a performance enhancement while being 20 times more efficient.

Whether concatenate input and GT images. Our experimental evaluations are
conducted on the NTIRE2020-track1 dataset, which is a real-world SR dataset.
As indicated in Tab. X, two distinct inputs are considered for the CPENS1:
solely the ground truth (GT) images, and the concatenation of both GT and
input images. It is evident that employing the concatenation of GT and input
images as input to CPENS1 yields superior performance. This approach
facilitates the DM in comprehending the disparity between the input and GT
images, thereby enhancing its ability to provide more effective guidance for
I2I.

The effect of resolutions. Our experimental evaluations are conducted on the
Single Image Super-Resolution (SISR) dataset, precisely the DIV2K100 dataset,
which is resized to a variety of resolutions. The results, displayed in Tab.
XI, unequivocally showcase DiffI2IS2’s superior performance across different
resolutions, surpassing the ByteGAN and SRDiff. These results underscore the
exceptional adaptability of our DiffI2IS2 to diverse resolution settings.

TABLE X: Whether concatenate GT and input images? Method | LPIPS$\downarrow$ | DISTS$\downarrow$ | PSNR$\uparrow$
---|---|---|---
DiffI2IS2 (concat GT and input, Ours) | 0.2943 | 0.1038 | 27.52
DiffI2IS2 (only GT) | 0.2951 | 0.1042 | 27.44
TABLE XI: $4\times$ SR results on different resolutions (LPIPS$\downarrow$). Method | 256$\times$256 | 512$\times$512 | 1024$\times$1024
---|---|---|---
BybeGAN [89] | 0.1210 | 0.1129 | 0.1088
SRDiff [16] | 0.1469 | 0.1433 | 0.1412
DiffI2IS2 (Ours) | 0.1088 | 0.1032 | 0.0992

##  7 Conclusion

Traditional DMs have demonstrated impressive capabilities in image synthesis.
However, when it comes to certain I2I tasks that demand strict adherence to GT
constraints, a deviation from the traditional approach is necessary. The
direct application of the DM framework, designed primarily for starting pixel-
level generation from scratch, is inefficient and unsuitable for such I2I
tasks. In this paper, we introduce a novel and efficient DM tailored
specifically for I2I, named DiffI2I. This approach comprises three key
components: CPEN, DI2Iformer, and a denoising network. The process begins with
feeding the ground-truth image into CPENS1, resulting in the generation of a
compact IPR. This IPR plays a crucial role in guiding the subsequent
operations of the DI2Iformer. After that, the DM is trained to accurately
estimate the compact IPR, as derived from CPENS1. Remarkably distinguishing
itself from traditional DMs, our DiffI2I demonstrates the remarkable ability
to achieve significantly more precise estimations and effectively diminish
artifacts in the generated results, all while employing far fewer iterations
and a lighter denoising network. Furthermore, this reduction in the
computation of DM facilitates the adoption of a joint optimization strategy
encompassing CPENS2, DI2Iformer, and the denoising network. This approach is
designed to mitigate the potential influence of estimation errors. Our
extensive experimental evaluations showcase DiffI2I’s achievement of a new
SOTA benchmark performance in the realm of I2I.

## References

  * [1] R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, K. Park, and V. Lempitsky, “Resolution-robust large mask inpainting with fourier convolutions,” in _WACV_ , 2022.
  * [2] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, “Context encoders: Feature learning by inpainting,” in _CVPR_ , 2016.
  * [3] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _TPAMI_ , 2015.
  * [4] B. Xia, Y. Zhang, Y. Wang, Y. Tian, W. Yang, R. Timofte, and L. Van Gool, “Knowledge distillation based degradation estimation for blind super-resolution,” _ICLR_ , 2023.
  * [5] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” _NeurIP_ , 2021.
  * [6] Q. Wan, Z. Huang, J. Lu, G. Yu, and L. Zhang, “Seaformer: Squeeze-enhanced axial transformer for mobile semantic segmentation,” _ICLR_ , 2023.
  * [7] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in _ICML_ , 2015\.
  * [8] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _NeurIPS_ , 2020.
  * [9] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” _ICLR_ , 2021.
  * [10] P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” _NeurIPS_ , 2021.
  * [11] J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans, “Cascaded diffusion models for high fidelity image generation.” _JMLR_ , 2022.
  * [12] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in _CVPR_ , 2022.
  * [13] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _CVPR_ , 2022.
  * [14] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” _TPAMI_ , 2022.
  * [15] B. Xia, Y. Zhang, S. Wang, Y. Wang, X. Wu, Y. Tian, W. Yang, and L. Van Gool, “Diffir: Efficient diffusion model for image restoration,” _ICCV_ , 2023\.
  * [16] H. Li, Y. Yang, M. Chang, S. Chen, H. Feng, Z. Xu, Q. Li, and Y. Chen, “Srdiff: Single image super-resolution with diffusion probabilistic models,” _Neurocomputing_ , 2022.
  * [17] Y. Ji, Z. Chen, E. Xie, L. Hong, X. Liu, Z. Liu, T. Lu, Z. Li, and P. Luo, “Ddp: Diffusion model for dense visual prediction,” _ICCV_ , 2023.
  * [18] K. Nazeri, E. Ng, T. Joseph, F. Z. Qureshi, and M. Ebrahimi, “Edgeconnect: Generative image inpainting with adversarial edge learning,” _arXiv preprint arXiv:1901.00212_ , 2019.
  * [19] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in _CVPRW_ , 2017.
  * [20] S.-J. Cho, S.-W. Ji, J.-P. Hong, S.-W. Jung, and S.-J. Ko, “Rethinking coarse-to-fine approach in single image deblurring,” in _ICCV_ , 2021.
  * [21] K. Zhang, W. Luo, Y. Zhong, L. Ma, B. Stenger, W. Liu, and H. Li, “Deblurring by realistic blurring,” in _CVPR_ , 2020.
  * [22] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _CVPR_ , 2015.
  * [23] C. Liu, S. Kumar, S. Gu, R. Timofte, and L. Van Gool, “Va-depthnet: A variational approach to single image depth prediction,” _ICLR_ , 2023.
  * [24] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in _CVPR_ , 2016.
  * [25] K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte, “Plug-and-play image restoration with deep denoiser prior,” _TPAMI_ , 2021.
  * [26] L. Cavigelli, P. Hager, and L. Benini, “Cas-cnn: A deep convolutional neural network for image compression artifact suppression,” in _IJCNN_ , 2017.
  * [27] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, “Improved training of wasserstein gans,” _arXiv preprint arXiv:1704.00028_ , 2017.
  * [28] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in _ECCVW_ , 2018, pp. 0–0.
  * [29] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in _ECCV_ , 2018.
  * [30] B. Xia, Y. Hang, Y. Tian, W. Yang, Q. Liao, and J. Zhou, “Efficient non-local contrastive attention for image super-resolution,” _AAAI_ , 2022.
  * [31] T. Dai, J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in _CVPR_ , 2019.
  * [32] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, “Generative image inpainting with contextual attention,” in _CVPR_ , 2018.
  * [33] Z. Yi, Q. Tang, S. Azizi, D. Jang, and Z. Xu, “Contextual residual aggregation for ultra high-resolution image inpainting,” in _CVPR_ , 2020.
  * [34] C. Xie, S. Liu, C. Li, M.-M. Cheng, W. Zuo, X. Liu, S. Wen, and E. Ding, “Image inpainting with learnable bidirectional attention maps,” in _ICCV_ , 2019.
  * [35] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, “Free-form image inpainting with gated convolution,” in _ICCV_ , 2019.
  * [36] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _TPAMI_ , 2017.
  * [37] J. H. Lee, M.-K. Han, D. W. Ko, and I. H. Suh, “From big to small: Multi-scale local planar guidance for monocular depth estimation,” _arXiv preprint arXiv:1907.10326_ , 2019.
  * [38] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _ECCV_ , 2018.
  * [39] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _ICLR_ , 2021.
  * [40] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in _ICML_ , 2021.
  * [41] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in _ICCV_ , 2021.
  * [42] W. Zhang, Z. Huang, G. Luo, T. Chen, X. Wang, W. Liu, G. Yu, and C. Shen, “Topformer: Token pyramid transformer for mobile semantic segmentation,” in _CVPR_ , 2022.
  * [43] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr _et al._ , “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in _CVPR_ , 2021.
  * [44] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _ECCV_ , 2020.
  * [45] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” _arXiv preprint arXiv:2010.04159_ , 2020.
  * [46] Z. Li, Z. Chen, X. Liu, and J. Jiang, “Depthformer: Exploiting long-range correlation and local information for accurate monocular depth estimation,” _arXiv preprint arXiv:2203.14211_ , 2022.
  * [47] Z. Li, X. Wang, X. Liu, and J. Jiang, “Binsformer: Revisiting adaptive bins for monocular depth estimation,” _arXiv preprint arXiv:2204.00987_ , 2022\.
  * [48] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, “Pre-trained image processing transformer,” in _CVPR_ , 2021.
  * [49] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in _ICCVW_ , 2021.
  * [50] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in _CVPR_ , 2022.
  * [51] W. Li, Z. Lin, K. Zhou, L. Qi, Y. Wang, and J. Jia, “Mat: Mask-aware transformer for large hole image inpainting,” in _CVPR_ , 2022.
  * [52] L. Chen, X. Chu, X. Zhang, and J. Sun, “Simple baselines for image restoration,” in _ECCV_. Springer, 2022.
  * [53] D. Kingma, T. Salimans, B. Poole, and J. Ho, “Variational diffusion models,” _NeurIPS_ , 2021.
  * [54] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet _et al._ , “Imagen video: High definition video generation with diffusion models,” _arXiv preprint arXiv:2210.02303_ , 2022.
  * [55] W. Harvey, S. Naderiparizi, V. Masrani, C. Weilbach, and F. Wood, “Flexible diffusion modeling of long videos,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 27 953–27 965, 2022.
  * [56] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “Diffwave: A versatile diffusion model for audio synthesis,” _arXiv preprint arXiv:2009.09761_ , 2020.
  * [57] B. L. Trippe, J. Yim, D. Tischer, D. Baker, T. Broderick, R. Barzilay, and T. Jaakkola, “Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem,” _arXiv preprint arXiv:2206.04119_ , 2022\.
  * [58] N. Anand and T. Achim, “Protein structure and sequence generation with equivariant denoising diffusion probabilistic models,” _arXiv preprint arXiv:2205.15019_ , 2022.
  * [59] B. Kawar, M. Elad, S. Ermon, and J. Song, “Denoising diffusion restoration models,” _arXiv preprint arXiv:2201.11793_ , 2022.
  * [60] H. Chung, B. Sim, and J. C. Ye, “Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction,” in _CVPR_ , 2022.
  * [61] J. Wu, H. Fang, Y. Zhang, Y. Yang, and Y. Xu, “Medsegdiff: Medical image segmentation with diffusion probabilistic model,” _arXiv preprint arXiv:2211.00611_ , 2022.
  * [62] J. Wolleb, R. Sandkühler, F. Bieder, P. Valmaggia, and P. C. Cattin, “Diffusion models for implicit image segmentation ensembles,” in _International Conference on Medical Imaging with Deep Learning_ , 2022.
  * [63] S. Saxena, A. Kar, M. Norouzi, and D. J. Fleet, “Monocular depth estimation using diffusion models,” _arXiv preprint arXiv:2302.14816_ , 2023.
  * [64] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi, “Palette: Image-to-image diffusion models,” in _ACM SIGGRAPH_ , 2022.
  * [65] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _arXiv preprint arXiv:1411.1784_ , 2014.
  * [66] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _ICCV_ , 2021.
  * [67] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [68] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” _arXiv preprint arXiv:1606.08415_ , 2016.
  * [69] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” _TPAMI_ , 2017.
  * [70] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” _arXiv preprint arXiv:1710.10196_ , 2017.
  * [71] Z. Wan, J. Zhang, D. Chen, and J. Liao, “High-fidelity pluralistic image completion with transformers,” in _ICCV_ , 2021.
  * [72] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _CVPR_ , 2018\.
  * [73] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” _NeurIPS_ , 2017.
  * [74] X. Wang, L. Xie, C. Dong, and Y. Shan, “Real-esrgan: Training real-world blind super-resolution with pure synthetic data,” in _ICCV_ , 2021.
  * [75] A. Lugmayr, M. Danelljan, and R. Timofte, “Ntire 2020 challenge on real-world image super-resolution: Methods and results,” in _CVPRW_ , 2020.
  * [76] A. Lugmayr, M. Danelljan, R. Timofte, M. Fritsche, S. Gu, K. Purohit, P. Kandula, M. Suin, A. Rajagoapalan, N. H. Joon _et al._ , “Aim 2019 challenge on real-world image super-resolution: Methods and results,” in _ICCVW_ , 2019.
  * [77] J. Cai, H. Zeng, H. Yong, Z. Cao, and L. Zhang, “Toward real-world single image super-resolution: A new benchmark and a new model,” in _ICCV_ , 2019\.
  * [78] K. Ding, K. Ma, S. Wang, and E. P. Simoncelli, “Image quality assessment: Unifying structure and texture similarity,” _TPAMI_ , 2020.
  * [79] K. Zhang, J. Liang, L. Van Gool, and R. Timofte, “Designing a practical degradation model for deep blind image super-resolution,” _arXiv preprint arXiv:2103.14006_ , 2021.
  * [80] R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in _International conference on curves and surfaces_ , 2010.
  * [81] J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in _CVPR_ , 2015.
  * [82] Y. Matsui, K. Ito, Y. Aramaki, A. Fujimoto, T. Ogawa, T. Yamasaki, and K. Aizawa, “Sketch-based manga retrieval using manga109 dataset,” _Multimedia Tools and Applications_ , 2017.
  * [83] C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in _ECCV_ , 2016.
  * [84] E. Agustsson and R. Timofte, “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in _CVPRW_ , 2017.
  * [85] X. Wang, K. Yu, C. Dong, and C. C. Loy, “Recovering realistic texture in image super-resolution by deep spatial feature transform,” in _CVPR_ , 2018.
  * [86] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang _et al._ , “Photo-realistic single image super-resolution using a generative adversarial network,” in _CVPR_ , 2017.
  * [87] K. Zhang, L. V. Gool, and R. Timofte, “Deep unfolding network for image super-resolution,” in _CVPR_ , 2020.
  * [88] C. Ma, Y. Rao, Y. Cheng, C. Chen, J. Lu, and J. Zhou, “Structure-preserving super resolution with gradient guidance,” in _CVPR_ , 2020.
  * [89] W. Li, K. Zhou, L. Qi, L. Lu, and J. Lu, “Best-buddy gans for highly detailed image super-resolution,” in _AAAI_ , 2022.
  * [90] R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, and L. Zhang, “Ntire 2017 challenge on single image super-resolution: Methods and results,” in _CVPRW_ , 2017.
  * [91] M. Bevilacqua, A. Roumy, C. Guillemot, and M. line Alberi Morel, “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” in _BMVC_ , 2012.
  * [92] X. Chu, L. Chen, , C. Chen, and X. Lu, “Improving image restoration by revisiting global information aggregation,” _ECCV_ , 2022.
  * [93] S. Nah, T. Hyun Kim, and K. Mu Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” in _CVPR_ , 2017.
  * [94] Z. Shen, W. Wang, X. Lu, J. Shen, H. Ling, T. Xu, and L. Shao, “Human-aware motion deblurring,” in _ICCV_ , 2019.
  * [95] L. Xu, S. Zheng, and J. Jia, “Unnatural l0 sparse representation for natural image deblurring,” in _CVPR_ , 2013.
  * [96] O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, “Deblurgan: Blind motion deblurring using conditional adversarial networks,” in _CVPR_ , 2018.
  * [97] J. Zhang, J. Pan, J. Ren, Y. Song, L. Bao, R. W. Lau, and M.-H. Yang, “Dynamic scene deblurring using spatially variant recurrent neural networks,” in _CVPR_ , 2018.
  * [98] O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang, “Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better,” in _ICCV_ , 2019.
  * [99] X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia, “Scale-recurrent network for deep image deblurring,” in _CVPR_ , 2018.
  * [100] H. Gao, X. Tao, X. Shen, and J. Jia, “Dynamic scene deblurring with parameter selective sharing and nested skip connections,” in _CVPR_ , 2019.
  * [101] D. Park, D. U. Kang, J. Kim, and S. Y. Chun, “Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training,” in _ECCV_ , 2020.
  * [102] H. Zhang, Y. Dai, H. Li, and P. Koniusz, “Deep stacked hierarchical multi-patch network for image deblurring,” in _CVPR_ , 2019.
  * [103] M. Suin, K. Purohit, and A. Rajagopalan, “Spatially-attentive patch-hierarchical network for adaptive motion deblurring,” in _CVPR_ , 2020\.
  * [104] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in _CVPR_ , 2021.
  * [105] I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” _ICLR_ , 2017.
  * [106] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _CVPR_ , 2009.
  * [107] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _ICML_ , 2015.
  * [108] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in _CVPR_ , 2017.
  * [109] J. Li, A. Hassani, S. Walton, and H. Shi, “Convmlp: Hierarchical convolutional mlps for vision,” in _CVPR_ , 2023.
  * [110] A. Kirillov, R. Girshick, K. He, and P. Dollár, “Panoptic feature pyramid networks,” in _CVPR_ , 2019.
  * [111] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _ICML_ , 2019.
  * [112] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _ECCV_ , 2018.
  * [113] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _CVPR_ , 2018.
  * [114] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [115] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, “Shufflenet v2: Practical guidelines for efficient cnn architecture design,” in _ECCV_ , 2018.
  * [116] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _CVPR_ , 2016.
  * [117] W. Yin, Y. Liu, C. Shen, and Y. Yan, “Enforcing geometric constraints of virtual normal for depth prediction,” in _ICCV_ , 2019.
  * [118] L. Huynh, P. Nguyen-Ha, J. Matas, E. Rahtu, and J. Heikkilä, “Guiding monocular depth estimation using depth-attention volume,” in _ECCV_ , 2020\.
  * [119] S. Lee, J. Lee, B. Kim, E. Yi, and J. Kim, “Patch-wise attention network for monocular depth estimation,” in _AAAI_ , 2021.
  * [120] G. Yang, H. Tang, M. Ding, N. Sebe, and E. Ricci, “Transformer-based attention networks for continuous pixel-wise prediction,” in _ICCV_ , 2021.
  * [121] S. F. Bhat, I. Alhashim, and P. Wonka, “Adabins: Depth estimation using adaptive bins,” in _CVPR_ , 2021.
  * [122] V. Patil, C. Sakaridis, A. Liniger, and L. Van Gool, “P3depth: Monocular depth estimation with a piecewise planarity prior,” in _CVPR_ , 2022.
  * [123] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, “Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps,” _NeurIPS_ , 2022.
  * [124] J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” _arXiv preprint arXiv:2010.02502_ , 2020.