# ReNoise: Real Image Inversion Through Iterative Noising

Daniel Garibi1 Or Patashnik1 Andrey Voynov2 Hadar Averbuch-Elor1 Daniel Cohen-
Or1
1Tel Aviv University 2Google Research
https://garibida.github.io/ReNoise-Inversion/

###### Abstract

Recent advancements in text-guided diffusion models have unlocked powerful
image manipulation capabilities. However, applying these methods to real
images necessitates the inversion of the images into the domain of the
pretrained diffusion model. Achieving faithful inversion remains a challenge,
particularly for more recent models trained to generate images with a small
number of denoising steps. In this work, we introduce an inversion method with
a high quality-to-operation ratio, enhancing reconstruction accuracy without
increasing the number of operations. Building on reversing the diffusion
sampling process, our method employs an iterative renoising mechanism at each
inversion sampling step. This mechanism refines the approximation of a
predicted point along the forward diffusion trajectory, by iteratively
applying the pretrained diffusion model, and averaging these predictions. We
evaluate the performance of our ReNoise technique using various sampling
algorithms and models, including recent accelerated diffusion models. Through
comprehensive evaluations and comparisons, we show its effectiveness in terms
of both accuracy and speed. Furthermore, we confirm that our method preserves
editability by demonstrating text-driven image editing on real images.

Figure 1:  Our ReNoise inversion technique can be applied to various diffusion
models, including recent few-step ones. This figure illustrates the
performance of our method with SDXL Turbo and LCM models, showing its
effectiveness compared to DDIM inversion. Additionally, we demonstrate that
the quality of our inversions allows prompt-driven editing. As illustrated on
the right, our approach also allows for prompt-driven image edits.

##  1 Introduction

Large-scale text-to-image diffusion models have revolutionized the field of
image synthesis [40, 41, 38, 20]. In particular, many works have shown that
these models can be employed for various types of image manipulation [17, 35,
48, 30, 9, 8, 7, 23, 15, 36, 6]. To edit _real_ images, many of these
techniques often require the inversion of the image into the domain of the
diffusion model. That is, given a real image $z_{0}$, one has to find a
Gaussian noise $z_{T}$, such that denoising $z_{T}$ with the pretrained
diffusion model reconstructs the given real image $z_{0}$. The importance of
this task for real image manipulation has prompted many efforts aimed at
achieving accurate reconstruction [32, 31, 16, 21].

Figure 2:  The diffusion process samples a Gaussian noise and iteratively
denoises it until reaching the data distribution. At each point along the
denoising trajectory, the model predicts a direction,
$\epsilon_{\theta}(z_{t})$, to step to the next point along the trajectory. To
invert a given image from the distribution, the direction from $z_{t}$ to
$z_{t+1}$ is approximated with the inverse of the direction from $z_{t}$ to
$z_{t-1}$ denoted by a dotted blue line.

The diffusion process consists of a series of denoising steps
$\\{\epsilon_{\theta}(z_{t},t)\\}_{t=T}^{1}$, which form a trajectory from the
Gaussian noise to the model distribution (see Figure 2). Each denoising step
is computed by a trained network, typically implemented as a UNet, which
predicts $z_{t-1}$ from $z_{t}$ [20]. The output of the model at each step
forms a _direction_ from $z_{t}$ to $z_{t-1}$ [45]. These steps are not
invertible, in the sense that the model was not trained to predict $z_{t}$
from $z_{t-1}$. Thus, the problem of inverting a given image is a challenge,
and particularly for real images, as they are not necessarily in the model
distribution (see Figure 3).

In this paper, we present an inversion method with a high quality-to-operation
ratio, which achieves superior reconstruction accuracy for the same number of
UNet operations. We build upon the commonly used approach of reversing the
diffusion sampling process, which is based on the linearity assumption that
the direction from $z_{t}$ to $z_{t+1}$ can be approximated by the negation of
the direction from $z_{t}$ to $z_{t-1}$ [12, 44] (see Figure 2). To enhance
this approximation, we employ the fixed-point iteration methodology [10].
Specifically, given $z_{t}$, we begin by using the common approximation to get
an initial estimate for $z_{t+1}$, denoted by $z_{t+1}^{(0)}$. Then, we
iteratively renoise $z_{t}$, following the direction implied by
$z_{t+1}^{(k)}$ to obtain $z_{t+1}^{(k+1)}$. After repeating this renoising
process several times, we apply an averaging on $z_{t+1}^{(k)}$ to form a more
accurate direction from $z_{t}$ to $z_{t+1}$.

We show that this approach enables longer strides along the inversion
trajectory while improving image reconstruction. Therefore, our method can
also be effective with diffusion models trained to generate images using a
small number of denoising steps [28, 43]. Furthermore, despite the need to
repeatedly renoise in each step of the inversion process, the longer strides
lead to a more favorable tradeoff of UNet operations for reconstruction
quality.

Through extensive experiments, we demonstrate the effectiveness of our method
in both image reconstruction and inversion speed. We validate the versatility
of our approach across different samplers and models, including recent time-
distilled diffusion models (e.g., SDXL-Turbo [43]). Importantly, we
demonstrate that the editability of the inversion achieved by our method
allows a wide range of text-driven image manipulations (see Figure 1).

Original Image | DDIM Inversion | \+ 1 ReNoise Step
---|---|---
|  |
|  |
Figure 3:  Comparing reconstruction results of plain DDIM inversion (middle
column) on SDXL to DDIM inversion with one ReNoise iteration (rightmost
column).  Figure 4:  Method overview. Given an image $z_{0}$, we iteratively
compute $z_{1},...,z_{T}$, where each $z_{t}$ is calculated from $z_{t-1}$. At
each time step, we apply the UNet ($\epsilon_{\theta}$) $\mathcal{K}+1$ times,
each using a better approximation of $z_{t}$ as the input. The initial
approximation is $z_{t-1}$. The next one, $z_{t}^{(1)}$, is the result of the
reversed sampler step (i.e., DDIM). The reversed step begins at $z_{t-1}$ and
follows the direction of $\epsilon_{\theta}(z_{t-1},t)$. At the $k$ renoising
iteration, $z_{t}^{(k)}$ is the input to the UNet, and we obtain a better
$z_{t}$ approximation. For the lasts iterations, we optimize
$\epsilon_{\theta}(z_{t}^{(k)},t)$ to increase editability. As the final
denoising direction, we use the average of the UNet predictions of the last
few iterations.

##  2 Related Work

#### Image Editing via Diffusion Models

Recent advancements in diffusion models [20, 12] have resulted in
unprecedented diversity and fidelity in visual content creation guided by
free-form text prompts [40, 37, 38, 41]. Text-to-image models do not directly
support text-guided image editing. Therefore, harnessing the power of these
models for image editing is a significant research area and many methods have
utilized these models for different types of image editing [30, 23, 17, 48, 9,
36, 35, 15, 6, 18, 14, 49, 8, 7, 11]. A common approach among these methods
requires _inversion_ [44, 32, 21, 50] to edit real images, i.e., obtaining a
latent code $z_{T}$ such that denoising it with the pretrained diffusion model
returns the original image. Specifically, in this approach two backward
processes are done simultaneously using $z_{T}$. One of the processes
reconstructs the image using the original prompt, while the second one injects
features from the first process (e.g., attention maps) to preserve some
properties of the original image while manipulating other aspects of it.

#### Inversion in Diffusion Models

Initial efforts in image inversion for real image editing focused on GANs [54,
55, 53, 2, 3, 39, 4, 47, 5, 13, 34]. The advancements in diffusion models, and
in diffusion-based image editing in particular have recently prompted works
studying the inversion of a diffusion-based denoising process. This inversion
depends on the sampler algorithm used during inference, which can be
deterministic [44] or non-deterministic [20, 22]. Inversion methods can be
accordingly categorized into two: methods that are suitable for deterministic
sampling, and methods suitable for non-deterministic sampling.

Methods that approach the deterministic inversion commonly rely on the DDIM
sampling method [44], and build upon DDIM inversion [12, 44]. Mokady et al.
[32] observed that the use of classifier-free guidance during inference
magnifies the accumulated error of DDIM inversion and therefore leads to poor
reconstruction. Following this observation, several works [32, 31, 16] focused
on solving this issue by replacing the null text token with a different
embedding, which is found after an optimization process or by a closed
solution. However, excluding [32] which requires a lengthy optimization, these
methods are limited by the reconstruction accuracy of DDIM inversion, which
can be poor, especially when a small number of denoising steps is done. In our
work, we present a method that improves the reconstruction quality of DDIM
inversion and therefore can be integrated with methods that build on it.

Another line of work [21, 51] tackles the inversion of DDPM sampler [20]. In
these works [51, 21], instead of inverting the image into an initial noise
$z_{T}$, a series of noises $\\{z_{T},\epsilon_{T},...,\epsilon_{1}\\}$ is
obtained. The definition of this noises series ensures that generating an
image with it returns the original input image. However, these methods require
a large number of inversion and denoising steps to allow image editing.
Applying these methods with an insufficient number of steps leads to too much
information encoded in $\\{\epsilon_{T},...,\epsilon_{1}\\}$ which limits the
ability to edit the generated image. As shall be shown, The editability issue
of these methods is particularly evident in few-steps models [28, 27, 43].

Most relevant to our work, two recent inversion methods [33, 29] also use the
fixed-point iteration technique. Specifically, they improve the reconstruction
accuracy of DDIM inversion [44] with Stable Diffusion [40] without introducing
a significant computational overhead. In our work, we focus on the problem of
real image inversion for recently introduced few-step diffusion models, where
the difficulties encountered by previous methods are amplified. Furthermore,
we show that our inversion method successfully works with various models and
different samplers.

#### Few Steps Models

Recently, new methods [43, 27, 28, 46, 42] that fine-tune text-to-image
diffusion models enabled a significant reduction of the number of steps needed
for high-quality image generation. While standard diffusion models typically
require 50 denoising steps to generate high-quality images, recent accelerated
models achieve high-quality synthesis with 1-4 steps only. These new methods
pave the way for interactive editing workflows. However, as we show in this
paper, using current methods for the inversion of an image with a small number
of steps degrades the reconstruction quality in terms of accuracy [44, 12] or
editability [21, 51].

##  3 Method

###  3.1 ReNoise Inversion

Figure 5:  Geometric intuition for ReNoise. At each inversion step, we are
trying to estimate $z_{t}$ (marked with a red star) based on $z_{t-1}$. The
straightforward approach is to use the reverse direction of the denoising step
from $z_{t-1}$, assuming the trajectory is approximately linear. However, this
assumption is inaccurate, especially in few-step models, where the size of the
steps is not small. We use the linearity assumption only as an initial
estimation and keep improving the estimation. We recalculate the denoising
step from the previous estimation (which is closer to $z_{t}$) and then
proceed with its opposite direction from $z_{t-1}$ (see the orange vectors).

#### Reversing the Sampler

Samplers play a critical role in the diffusion-based image synthesis process.
They define the noising and denoising diffusion processes and influence the
processes’ trajectories and quality of the generated images. While different
samplers share the same pre-trained UNet model (denoted by
$\epsilon_{\theta}$) as their backbone, their sampling approaches diverge,
leading to nuanced differences in output. The goal of the denoising sampler is
to predict the latent code at the previous noise level, $z_{t-1}$, based on
the current noisy data $z_{t}$, the pretrained UNet model, and a sampled
noise, $\epsilon_{t}$. Various denoising sampling algorithms adhere to the
form:

$z_{t-1}=\phi_{t}z_{t}+\psi_{t}\epsilon_{\theta}(z_{t},t,c)+\rho_{t}\epsilon_{t},$
(1)

where $c$ represents a text embedding condition, and $\phi_{t}$, $\psi_{t}$,
and $\rho_{t}$ denote sampler parameters. At each step, these parameters
control the extent to which the previous noise is removed ($\phi_{t}$), the
significance assigned to the predicted noise from the UNet ($\psi_{t}$), and
the given weight to the additional noise introduced ($\rho_{t}$).

A given image $z_{0}$ can be inverted by reformulating Equation 1 and applying
it iteratively:

$z_{t}=\frac{z_{t-1}-\psi_{t}\epsilon_{\theta}(z_{t},t,c)-\rho_{t}\epsilon_{t}}{\phi_{t}},$
(2)

where for non-deterministic samplers, a series of random noises
$\\{\epsilon_{t}\\}_{t=1}^{T}$ is sampled and used during both inversion and
image generation processes. However, directly computing $z_{t}$ from Equation
2 is infeasible since it relies on $\epsilon_{\theta}(z_{t},t,c)$, which, in
turn, depends on $z_{t}$, creating a circular dependency. To solve this
implicit function, Dhariwal et al. [12] propose using the approximation
$\epsilon_{\theta}(z_{t},t,c)\approx\epsilon_{\theta}(z_{t-1},t,c)$:

$z^{(1)}_{t}=\frac{z_{t-1}-\psi_{t}\epsilon_{\theta}(z_{t-1},t,c)-\rho_{t}\epsilon_{t}}{\phi_{t}}.$
(3)

This method has several limitations. First, the assumption underlying the
approximation used in [12] is that the number of inversion steps is large
enough, implying a trajectory close to linear. This assumption restricts the
applicability of this inversion method in interactive image editing with
recent few-step diffusion models [46, 27, 28, 43], as the inversion process
would take significantly longer than inference. Second, this method struggles
to produce accurate reconstructions in certain cases, such as highly detailed
images or images with large smooth regions, see Figure 3. Moreover, we observe
that this inversion method is sensitive to the prompt $c$ and may yield poor
results for certain prompts.

#### ReNoise

In a successful inversion trajectory, the direction from $z_{t-1}$ to $z_{t}$
aligns with the direction from $z_{t}$ to $z_{t-1}$ in the denoising
trajectory. To achieve this, we aim to improve the approximation of
$\epsilon_{\theta}(z_{t},t,c)$ in Eq. 2 compared to the one used in [12].
Building on the fixed-point iteration technique [10], our approach better
estimates the instance of $z_{t}$ that is inputted to the UNet, rather than
relying on $z_{t-1}$.

Intuitively, we utilize the observation that $z_{t}^{(1)}$ (from Eq. 3) offers
a more precise estimate of $z_{t}$ compared to $z_{t-1}$. Therefore, employing
$z_{t}^{(1)}$ as the input to the UNet is likely to yield a more accurate
direction, thus contributing to reducing the overall error in the inversion
step. We illustrate this observation in Figure 5. Iterating this process
generates a series of estimations for $z_{t}$, denoted by
$\\{z_{t}^{(k)}\\}_{k=1}^{\mathcal{K}+1}$. While the fixed-point iteration
technique [10] does not guarantee convergence of this series in the general
case, in Section 4, we empirically show that convergence holds in our setting.
However, as the convergence is not monotonic, we refine our prediction of
$z_{t}$ by averaging several $\\{z_{t}^{(k)}\\}$, thus considering more than a
single estimation of $z_{t}$. See Figure 6 for an intuitive illustration.

In more detail, our method iteratively computes estimations of $z_{t}$ during
each inversion step $t$ by renoising the noisy latent $z_{t-1}$ multiple
times, each with a different noise prediction (see Figure 4). Beginning with
$z_{t}^{(1)}$, in the $k$-th renoising iteration, the input to the UNet is the
result of the previous iteration, $z_{t}^{(k)}$. Then, $z_{t}^{(k+1)}$ is
calculated using the inverted sampler while maintaining $z_{t-1}$ as the
starting point of the step. After $\mathcal{K}$ renoising iterations, we
obtain a set of estimations $\\{z_{t}^{(k)}\\}_{k=1}^{\mathcal{K}+1}$. The
next point on the inversion trajectory, $z_{t}$, is then defined as their
weighted average, where $w_{k}$ is the weight assigned to $z_{t}^{(k)}$. For a
detailed description of our method, refer to Algorithm 1.

Figure 6:  Schematic illustration of the ReNoise convergence process to the
true inversion of $z_{t-1}$. While estimates may converge non-monotonically to
the unknown target $z_{t}$, we found that averaging them improves true value
estimation. Typically, the initial iteration exhibits an exponential decrease
in the norm between consecutive elements.

###  3.2 Reconstruction-Editability Tradeoff

#### Enhance Editability

The goal of inversion is to facilitate the editing of real images using a
pretrained image generation model. While the the renoising approach attains
highly accurate reconstruction results, we observe that the resulting $z_{T}$
lacks editability. This phenomenon can be attributed to the reconstruction-
editability tradeoff in image generative models [47]. To address this
limitation, we incorporate a technique to enhance the editability of our
method.

It has been shown [35] that the noise maps predicted during the inversion
process often diverge from the statistical properties of uncorrelated Gaussian
white noise, thereby affecting editability. To tackle this challenge, we
follow pix2pix-zero [35] and regularize the predicted noise at each step,
$\epsilon_{\theta}(z_{t},t,c)$, using the following loss terms.

First, we encourage $\epsilon_{\theta}(z_{t},t,c)$ to follow the same
distribution as $\epsilon_{\theta}(z_{t}^{\prime},t,c)$, where
$z_{t}^{\prime}$ represents the input image $z_{0}$ with added random noise
corresponding to the noise level at timestep $t$. We do so by dividing
$\epsilon_{\theta}(z_{t},t,c)$ and $\epsilon_{\theta}(z_{t}^{\prime},t,c)$
into small patches (e.g., 4$\times$4), and computing the KL-divergence between
corresponding patches. We denote this loss term by $\mathcal{L}_{\text{patch-
KL}}$. Second, we utilize $\mathcal{L}_{\text{pair}}$ proposed in pix2pix-zero
[35], which penalizes correlations between pairs of pixels. We leverage these
losses to enhance the editability of our method, and denote the combination of
them as $\mathcal{L}_{\text{edit}}$. For any renoising iteration $k$ where
$w_{k}>0$, we regularize the UNet’s prediction
$\epsilon_{\theta}(z_{t}^{(k)},t,c)$ using $\mathcal{L}_{\text{edit}}$ before
computing $z_{t}^{(k+1)}$. See line 9 in Algorithm 1.

1 Input: An image $z_{0}$, number of renoising steps $\mathcal{K}$, number of
inversion steps $T$, a series of renoising weights
$\\{w_{k}\\}_{k=1}^{\mathcal{K}}$.

2 Output: A noisy latent $z_{T}$ and set of noises
$\\{\epsilon_{t}\\}_{t=1}^{T}$.

3

4for  _$t=1,2,\ldots,T$_ do

5 sample $\epsilon_{t}\sim\mathcal{N}(0,I)$

6 $z_{t}^{(0)}\leftarrow z_{t-1}$

7 $z_{t}^{(\text{avg})}\leftarrow 0$

8 for  _$k=1,\ldots,\mathcal{K}$_ do

9 $\delta_{t}^{k}\leftarrow\epsilon_{\theta}(z_{t}^{(k-1)},t)$

10 $\delta_{t}^{k}\leftarrow\texttt{Enhance-
editability}(\delta_{t}^{k},w_{k})$

11 $z_{t}^{(k)}\leftarrow\texttt{Inverse-Step}(z_{t-1},\delta_{t}^{k})$

12

13 end for

// Average ReNoised predictions

14 $z_{t}^{(\text{avg})}\leftarrow\sum_{k=1}^{\mathcal{K}}{w_{k}\cdot
z_{t}^{(k)}}$

15 $\epsilon_{t}\leftarrow\texttt{Noise-
Correction}(z_{t}^{(\text{avg})},t,\epsilon_{t},z_{t-1})$

16

17 end for

18return _$(z_{T},\\{\epsilon_{t}\\}_{t=1}^{T})$_

19

20 Function  _Inverse-Step(_$z_{t-1}$ , $\delta_{t}$, $t$_)_:

21 return
${\frac{1}{\phi_{t}}z_{t-1}-\frac{\psi_{t}}{\phi_{t}}\delta_{t}-\frac{\rho_{t}}{\phi_{t}}\epsilon_{t}}$

22

23

24 Function  _Enhance-editability(_$\delta_{t}^{(k)}$ , $w_{k}$_)_:

25 if  _$w_{k} >0$_ then

26
$\delta_{t}^{k}\leftarrow\delta_{t}^{k}-\nabla_{\delta_{t}^{k}}\mathcal{L}_{\text{edit}}(\delta_{t}^{k})$

27

28 end if

29 return $\delta_{t}^{k}$

30

31

32 Function  _Noise-Correction(_$z_{t}$ , $t$, $\epsilon_{t}$, $z_{t-1}$_)_:

33 $\delta_{t}\leftarrow\epsilon_{\theta}(z_{t},t)$

34
$\epsilon_{t}\leftarrow\epsilon_{t}-\nabla_{\epsilon_{t}}\frac{1}{\rho_{t}}(z_{t-1}-\phi_{t}z_{t}-\psi_{t}\delta_{t})$

35 return $\epsilon_{t}$

36

Algorithm 1 ReNoise Inversion

#### Noise Correction in Non-deterministic Samplers

Non-deterministic samplers, in which $\rho_{t}>0$, introduce noise
($\epsilon_{t}$) at each denoising step. Previous methods [51, 21] suggested
using $\epsilon_{t}$ to bridge the gap between the inversion and denoising
trajectories in DDPM inversion. Specifically, given a pair of points
$z_{t-1},z_{t}$ on the inversion trajectory, we denote by $\hat{z}_{t-1}$ the
point obtained by denoising $z_{t}$. Ideally, $z_{t-1}$ and $\hat{z}_{t-1}$
should be identical. We define:

$\epsilon_{t}=\frac{1}{\rho_{t}}(z_{t-1}-\phi_{t}z_{t}-\psi_{t}\epsilon_{\theta}(z_{t},t,c)).$
(4)

Integrating this definition into Eq. 1 yields $\hat{z}_{t-1}=z_{t-1}$.
However, we found that replacing $\epsilon_{t}$ with the above definition
affects editability. Instead, we suggest a more tender approach, optimizing
$\epsilon_{t}$ based on Eq. 4 as our guiding objective:

$\epsilon_{t}=\epsilon_{t}-\nabla_{\epsilon_{t}}\frac{1}{\rho_{t}}(z_{t-1}-\phi_{t}z_{t}-\psi_{t}\epsilon_{\theta}(z_{t},t,c)).$
(5)

This optimization improves the reconstruction fidelity while preserving the
distribution of the noisy-latents.

##  4 Convergence Discussion

In this section, we first express the inversion process as a backward Euler
process and our renoising iterations as fixed-point iterations. While these
iterations do not converge in the general case, we present a toy example where
they yield accurate inversions. Then, we analyze the convergence of the
renoising iterations in our real-image inversion scenario and empirically
verify our method’s convergence.

#### Inversion Process as Backward Euler

The denoising process of diffusion models can be mathematically described as
solving an ordinary differential equation (ODE). A common method for solving
such equations is the Euler method, which takes small steps to approximate the
solution. For ODE in the form of $y^{\prime}(t)=f(t,y(t))$, Euler solution is
defined as:

$y_{n+1}=y_{n}+h\cdot f(t_{n},y_{n}),$

where $h$ is the step size. The inversion process can be described as solving
ODE using the backward Euler method (or implicit Euler method) [1]. This
method is similar to forward Euler, with the difference that $y_{n+1}$ appears
on both sides of the equation:

$y_{n+1}=y_{n}+h\cdot f(t_{n+1},y_{n+1}).$

For equations lacking an algebraic solution, several techniques estimate
$y_{n+1}$ iteratively. As we described in Section 3.1, the inversion process
lacks a closed-form solution, as shown in Equation 2. To address this, the
ReNoise method leverages fixed-point iterations, which we refer to as
_reonising iterations,_ to progressively refine the estimate of $y_{n+1}$:

$y_{n+1}^{(0)}=y_{n},\quad y_{n+1}^{(k+1)}=y_{n}+h\cdot
f(t_{n+1},y_{n+1}^{(k)}).$

In our ReNoise method, we average these renoising iterations to mitigate
convergence errors, leading to improvement in the reconstruction quality.

#### Renoising Toy Example

We begin with the simple toy example, the diffusion of a shifted Gaussian.
Given the initial distribution $\mu_{0}\sim\mathcal{N}(a,I)$, where $a$ is a
non-zero shift value and $I$ is the identity matrix. The diffusion process
defines the family of distributions $\mu_{t}\sim\mathcal{N}(ae^{-t},I)$, and
the probability flow ODE takes the form $\frac{dz}{dt}=-ae^{-t}$ (see [24] for
details). The Euler solver step at a state $(z_{t},t)$, and timestep $\Delta
t$ moves it to $(z_{t+\Delta t}^{(1)},t+\Delta t)=(z_{t}-ae^{-t}\cdot\Delta
t,t+\Delta t)$. Notably, the backward Euler step at this point does not lead
to $z_{t}$. After applying the first renoising iteration, we get
$(z^{(2)}_{t+\Delta t},t+\Delta t)=(z_{t}-ae^{-(t+\Delta t)}\cdot\Delta
t,t+\Delta t)$ and the backward Euler step at this point leads exactly to
$(z_{t},t)$. Thus, in this simple example, we successfully estimates the exact
pre-image after a single step. While this convergence cannot be guaranteed in
the general case, in the following, we discuss some sufficient conditions for
the algorithm’s convergence and empirically verify them for the image
diffusion model.

Figure 7:  Average distance between consequent estimations $z_{t}^{(k)}$, and
$z_{t}^{(k+1)}$. Vertical bars indicate the standard deviation. The averages
are computed over 32 images and 10 different timesteps.

#### ReNoise Convergence

During the inversion process, we aim to find the next noise level inversion,
denoted by $\hat{z}_{t}$, such that applying the denoising step to
$\hat{z}_{t}$ recovers the previous state, $z_{t-1}$. Given the noise
estimation $\epsilon_{\theta}(z_{t},t)$ and a given $z_{t-1}$, the ReNoise
mapping defined in Section 3.1 can be written as
$\mathcal{G}:z_{t}\to\mathrm{InverseStep}(z_{t-1},\epsilon_{\theta}(z_{t},t))$.
For example, in the case of using DDIM sampler the mapping is
$\mathcal{G}(z_{t})=\frac{1}{\phi_{t}}(z_{t-1}-\psi_{t}\epsilon_{\theta}(z_{t},t))$.
The point $\hat{z}_{t}$, which is mapped to $z_{t-1}$ after the denoising
step, is a stationary point of this mapping. Given $z_{t}^{(1)}$, the first
approximation of the next noise level $z_{t}$, our goal is to show that the
sequence $z_{t}^{(k)}=\mathcal{G}^{k-1}(z_{t}^{(1)}),k\to\infty$ converges. As
the mapping $\mathcal{G}$ is continuous, the limit point would be its
stationary point. The definition of $\mathcal{G}$ gives:

$\|z_{t}^{(k+1)}-z_{t}^{(k)}\|=\|\mathcal{G}(z_{t}^{(k)})-\mathcal{G}(z_{t}^{(k-1)})\|,$

where the norm is always assumed as the $l_{2}$-norm. For the ease of the
notations, we define $\Delta^{(k)}=z_{t}^{(k)}-z_{t}^{(k-1)}$. For convergence
proof, it is sufficient to show that the sum of norms of these differences
converges, which will imply that $z_{t}^{(k)}$ is the Cauchy sequence. Below
we check that in practice $\|\Delta^{(k)}\|$ decreases exponentially as
$k\to\infty$ and thus has finite sum. In the assumption that $\mathcal{G}$ is
$\mathcal{C}^{2}$-smooth, the Taylor series conducts:

$\|\Delta^{(k+1)}\|=\|\mathcal{G}(z_{t}^{(k)})-\mathcal{G}(z_{t}^{(k-1)})\|=\\\
\|\mathcal{G}(z_{t}^{(k-1)})+\frac{\partial\mathcal{G}}{\partial
z}|_{z_{t}^{(k-1)}}\cdot\Delta^{(k)}+O(\|\Delta^{(k)}\|^{2})-\mathcal{G}(z_{t}^{(k-1)})\|=\\\
\|\frac{\partial\mathcal{G}}{\partial
z}|_{z_{t}^{(k-1)}}\cdot\Delta^{(k)}+O(\|\Delta^{(k)}\|^{2})\|\leq\\\
\|\frac{\partial\mathcal{G}}{\partial
z}|_{z_{t}^{(k-1)}}\|\cdot\|\Delta^{(k)}\|+O(\|\Delta^{(k)}\|^{2})=\\\
\frac{\psi_{t}}{\phi_{t}}\cdot\|\frac{\partial\epsilon_{\theta}}{\partial
z}|_{z_{t}^{(k-1)}}\|\cdot\|\Delta^{(k)}\|+O(\|\Delta^{(k)}\|^{2})$

Thus, in a sufficiently small neighborhood, the convergence dynamics is
defined by the scaled Jacobian norm
$\frac{\psi_{t}}{\phi_{t}}~{}\cdot~{}\|\frac{\partial\epsilon_{\theta}}{\partial
z}|_{z_{t}^{(k-1)}}\|$. In the Appendix A, we show this scaled norm estimation
for the SDXL diffusion model for various steps and ReNoise iterations indices
$(k)$. Remarkably, the ReNoise indices minimally impact the scale factor,
consistently remaining below 1. This confirms in practice the convergence of
the proposed algorithm. Notably, the highest scaled norm values occur at
smaller $t$ (excluding the first step) and during the initial renoising
iteration. This validates the strategy of not applying ReNoise in early steps,
where convergence tends to be slower compared to other noise levels.
Additionally, the scaled norm value for the initial $t$ approaches 0, which
induces almost immediate convergence.

Figure 7 illustrates the exponential decrease in distances between consecutive
elements $z_{t}^{(k)}$ and $z_{t}^{(k+1)}$, which confirms the algorithm’s
convergence towards the stationary point of the operator $\mathcal{G}$.

The proposed averaging strategy is aligned with the conclusions described
above, and also converges to the desired stationary point. In The Appendix A,
we present a validation for this claim.

SDXL | SDXL Turbo | LCM
---|---|---
|  |
Figure 8:  Image reconstruction results comparing sampler reversing inversion
techniques across different samplers (e.g., vanilla DDIM inversion) with our
ReNoise method using the same sampler. The number of denoising steps remains
constant. However, the number of UNet passes varies, with the sampler
reversing approach increasing the number of inversion steps, while our method
increases the number of renoising iterations. We present various configuration
options for our method, including options with or without edit enhancement
loss and Noise Correction (NC).

##  5 Experiments

In this section, we conduct extensive experiments to validate the
effectiveness of our method. We evaluate both the reconstruction quality of
our inversion and its editability. To demonstrate the versatility of our
approach, we apply it to four models, SD [40], SDXL [37], SDXL Turbo [43], and
LCM-LoRA [28], with SDXL Turbo and LCM-LoRA being few-step models.
Additionally, we use various sampling algorithms including both deterministic
and non-deterministic ones. Implementation details for each model are provided
in Appendix B. Following previous works [32, 4], we quantitatively evaluate
our method with three metrics: $L_{2}$, LPIPS [52], and PSNR. Unless stated
otherwise, for both inversion and generation we use the prompt obtained from
BLIP2 [25].

###  5.1 Reconstruction and Speed

We begin by evaluating the reconstruction-speed tradeoff. The main
computational cost of both the inversion and denoising processes is the
forward pass through the UNet. In each renoising iteration, we perform one
forward pass, which makes it computationally equal to a standard inversion
step (as done in DDIM Inversion for example). In the following experiments, we
compare the results of a sampler reversing with our method, where we match the
number of UNet passes between the methods. For example, 8 steps of sampler
reversing are compared against 4 steps with one renoising iteration at each
step.

Original Image |  |  |  |
---|---|---|---|---
4 UNet Operations |  |  |  |
8 UNet Operations |  |  |  |
20 UNet Operations |  |  |  |
40 UNet Operations |  |  |  |
|  | DDIM
---
Inversion
| w/
---
ReNoise
| DDIM
---
Inversion
| w/
---
ReNoise
Figure 9:  Qualitative comparison between DDIM Inversion to our ReNoise method
using the DDIM sampler on SDXL Turbo. The first row presents the input images.
In each subsequent row, we present the reconstruction results of both
approaches, each utilizing the same number of UNet operations in the inversion
process. To generate the images, we use 4 denoising steps in all cases. DDIM
Inversion performs more inversion steps (i.e., smaller strides), while our
method performs more renoising steps.  Table 1:  Image reconstruction results
with a fixed number of 100 UNet operations. Each row showcases the results
obtained using different combinations of inversion steps, denoising steps, and
renoising iterations, totaling 100 operations. As observed, allocating some of
the operations to renoising iterations improves the reconstruction quality
while maintaining the same execution time.  Image Reconstruction With a Fixed
Number of UNet Operations
---
Inversion | Inference | ReNoise |  L2 $\downarrow$ |  PSNR $\uparrow$ |  LPIPS $\downarrow$
Steps | Steps | Steps |  |  |
50 | 50 | 0 | 0.00364 | 26.023 | 0.06273
75 | 25 | 0 | 0.00382 | 25.466 | 0.06605
80 | 20 | 0 | 0.00408 | 25.045 | 0.07099
90 | 10 | 0 | 0.01023 | 20.249 | 0.10305
25 | 25 | 2 | 0.00182 | 29.569 | 0.03637
20 | 20 | 3 | 0.00167 | 29.884 | 0.03633
10 | 10 | 8 | 0.00230 | 28.156 | 0.04678

#### Qualitative Results

In Figure 9 we show qualitative results of image reconstruction on SDXL Turbo
[43]. Here, we utilize DDIM as the sampler, and apply four denoising steps for
all configurations. Each row exhibits results obtained using a different
amount of UNet operations. In our method, we apply four inversion steps, and a
varying number of renoising iterations. As can be seen, the addition of
renoising iterations gradually improves the reconstruction results.
Conversely, employing more inversion steps proves insufficient for capturing
all details in the image, as evident by the background of the car, or even
detrimental to the reconstruction, as observed in the Uluro example.

#### Quantitative Results

For the quantitative evaluation, we use the MS-COCO 2017 [26] validation
dataset. Specifically, we retain images with a resolution greater than
$420\times 420$, resulting in a dataset containing 3,865 images.

We begin by evaluating both the sampler reversing approach and our ReNoise
method, while varying the number of UNet operations during the inversion
process and keeping the number of denoising steps fixed. This experiment is
conducted using various models (SDXL, SDXL Turbo, LCM) and samplers. For all
models, we utilize the DDIM [44] sampler. In addition, we employ the
Ancestral-Euler scheduler for SDXL Turbo, and the default LCM sampler for LCM-
LoRA. We set the number of denoising steps to 50 for SDXL, and to 4 for SDXL
Turbo and LCM-LoRA. Quantitative results, using PSNR as the metric, are
presented in Figure 8. We evaluate our method using different configurations.
The x-axis refers to the number of UNet operations in the inversion process.
LPIPS metrics results are provided in Appendix C.

As depicted in the graphs, incorporating additional renoising iterations
proves to be more beneficial for image reconstruction compared to adding more
inversion steps. Note that the performance of the Ancestral-Euler and LCM
samplers noticeably degrades when the number of inversion steps exceeds the
number of denoising steps. Unlike DDIM, these samplers have $\Phi_{t}\approx
1$, resulting in an increase in the latent vector’s norm beyond what can be
effectively denoised in fewer steps. In this experiment, we maintain the same
number of UNet operations for both ReNoise and the sampler reversing approach.
However, in ReNoise, the number of inversion steps remains fixed, and the
additional operations are utilized for renoising iterations, refining each
point on the inversion trajectory. Consequently, our method facilitates
improved reconstruction when using these noise samplers.

We continue by evaluating both the sampler reversing approach and our method
while maintaining a fixed total number of UNet operations for the inversion
and denoising processes combined. The results for SDXL with DDIM are presented
in Table 1. The table displays various combinations of inversion, denoising,
and renoising steps, totaling 100 UNet operations. Despite employing longer
strides along the inversion and denoising trajectories, our ReNoise method
yields improved reconstruction accuracy, as evident in the table. Furthermore,
a reduced number of denoising steps facilitates faster image editing,
especially since it commonly involves reusing the same inversion for multiple
edits.

Original Image |  $\longleftarrow$ Editing Results $\longrightarrow$
---|---
|  |  |
“cat” | “koala” | “cat statue” | “bear”
|  |  |
“person” | “panda mask” | “purple shirt” | “astronaut”
Figure 10:  LCM Editing Results. Each row showcases one image. The leftmost
image is the original, followed by three edited versions. The text below each
edited image indicates the specific word or phrase replaced or added to the
original prompt for that specific edit.

###  5.2 Reconstruction and Editability

In Figure 10, we illustrate editing results generated by our method with LCM
LoRA [28]. These results were obtained by inverting the image using a source
prompt and denoising it with a target prompt. Each row exhibits an image
followed by three edits accomplished by modifying the original prompt. These
edits entail either replacing the object word or adding descriptive adjectives
to it. As can be seen, the edited images retain the details present in the
original image. For instance, when replacing the cat with a koala, the details
in the background are adequately preserved.

Original Image |  |  |  |  |
---|---|---|---|---|---
Euler Inversion |  |  |  |  |
With ReNoise |  |  |  |  |
With Averaging |  |  |  |  |
With Edit Losses |  |  |  |  |
With Noise Correction |  |  |  |  |
Figure 11:  Ablation study on SDXL Turbo. The first row presents the input
image. In each subsequent row, we show the reconstruction results using an
additional component of our inversion method. The images in the bottom row
represent the results obtained by our full method.

###  5.3 Ablation Studies

#### Image Reconstruction

Figure 11 qualitatively demonstrates the effects of each component in our
method, highlighting their contribution to the final outcome. Here, we use
SDXL Turbo model [43], with the Ancestral-Euler sampler, which is non-
deterministic. As our baseline, we simply reverse the sampler process. The
reconstruction, while semantically capturing the main object, fails to
reproduce the image’s unique details. For example, in the middle column, the
image contains a bird standing on a branch, but the branch is in a different
pose and the bird is completely different. Using 9 ReNoise iterations
significantly improves the reconstruction, recovering finer details like the
bird’s original pose and branch texture. However, some subtle details, such as
the bird’s colors or the color in Brad Pitt’s image, remain incomplete.
Averaging the final iterations effectively incorporates information from
multiple predictions, leading to a more robust reconstruction that captures
finer details. Regularize the UNet’s noise prediction with
$\mathcal{L}_{\text{edit}}$ can introduce minor artifacts to the
reconstruction, evident in the smoother appearance of the hair of the two
people on the left, or in the cake example. Finally, we present our full
method by adding the noise correction technique.

Table 2 quantitatively showcases the effect each component has on
reconstruction results. As can be seen, the best results were obtained by our
full method or by averaging the last estimations of $z_{t}$. Our final method
also offers the distinct advantage of getting an editable latent
representation.

In Appendix C, we present an ablation study to justify our editability
enhancement and noise correction components.

Table 2:  Quantitative ablation study on SDXL Turbo. We demonstrate the impact
of each component of our inversion method on reconstruction results. The
results improve with additional renoising iterations and significant
enhancements occur through averaging final estimations. Additionally, we
observe a reconstruction-editability trade-off, with edit losses causing
degradation that is effectively mitigated by Noise Correction.  Ablation -
Image Reconstruction
---
|  L2 $\downarrow$ |  PSNR $\uparrow$ |  LPIPS$\downarrow$
Euler Inversion | 0.0700 | 11.784 | 0.20337
\+ 1 ReNoise | 0.0552 | 12.796 | 0.20254
\+ 4 ReNoise | 0.0249 | 16.521 | 0.14821
\+ 9 ReNoise | 0.0126 | 19.702 | 0.10850
\+ Averaging ReNoise | 0.0087 | 21.491 | 0.08832
\+ Edit Losses | 0.0276 | 18.432 | 0.12616
\+ Noise Correction | 0.0196 | 22.077 | 0.08469
Ours |  |  |  |
---|---|---|---|---
Edit Friendly |  |  |  |
| Original | Reconstruction | “ginger cat” |  “wood” $\rightarrow$ “metal”
Ours |  |  |  |
Edit Friendly |  |  |  |
| Original | Reconstruction |  “cat” $\rightarrow$ “dog” | “wooden cat”
Figure 12:  Comparison with edit-friendly DDPM Inversion with SDXL Turbo. We
invert two images with the prompts: “a cat laying in a bed made out of wood”
(left) and “a cat is sitting in front of a mirror” (right) and apply two edits
to each image.

###  5.4 Comparisons

#### Inversion for Non-deterministic Samplers.

In Figure 12 we show a qualitative comparison with “an edit-friendly DDPM”
[21] where we utilize SDXL Turbo [43]. Specifically, we assess the performance
of the edit-friendly DDPM method alongside our ReNoise method in terms of both
reconstruction and editing.

We observe that in non-deterministic samplers like DDPM, the parameter
$\rho_{0}$ in Equation 1 equals zero. This means that in the final denoising
step, the random noise addition is skipped to obtain a clean image. In long
diffusion processes (e.g., 50-100 steps), the final denoising step often has
minimal impact as the majority of image details have already been determined.
Conversely, shorter diffusion processes rely on the final denoising step to
determine fine details of the image. Due to focusing solely on noise
correction to preserve the original image in the inversion process, the edit-
friendly DDPM struggles to reconstruct fine details of the image, such as the
shower behind the cat in the right example. However, our ReNoise method finds
an inversion trajectory that faithfully reconstructs the image and does not
rely solely on noise corrections. This allows us to better reconstruct fine
details such as the shower.

Additionally, encoding a significant amount of information within only a few
external noise vectors, $\epsilon_{t}$, limits editability in certain
scenarios, such as the ginger cat example. It is evident that the edit-
friendly DDPM method struggles to deviate significantly from the original
image while also failing to faithfully preserve it. For instance, it
encounters difficulty in transforming the cat into a ginger cat while omitting
the preservation of the decoration in the top left corner.

|  |  |
---|---|---|---
|  |  |
| Original
---
Image
| Null Text
---
Inversion
| Negative
---
Prompt Inv
| DDIM inv
---
w/ ReNoise
Figure 13:  Image reconstruction comparisons with Stable Diffusion. We present
the results of Null-Text Inversion (NTI), Negative-Prompt Inversion (NPI), and
our method. While NTI and our method achieve comparable results, ours
demonstrates significant speed improvement.

#### Null-prompt Inversion Methods

In Figure 13, we present a qualitative comparison between our method and null-
text based inversion methods. For this comparison, we utilize Stable Diffusion
[40] since these methods rely on a CFG [19] mechanism, which is not employed
in SDXL Turbo [43]. Specifically, we compare DDIM Inversion [44] with one
renoising iteration to Null-Text Inversion (NTI)[32] and Negative-Prompt
Inversion (NPI)[31]. Both NTI and NPI enhance the inversion process by
replacing the null-text token embedding when applying CFG. Our method achieves
results comparable to NTI, while NPI highlights the limitations of plain DDIM
inversion. This is because NPI sets the original prompt as the negative
prompt, essentially resulting in an inversion process identical to plain DDIM
inversion. Regarding running time, our ReNoise inversion process takes 13
seconds, significantly faster than NTI’s 3 minutes. For comparison, plain DDIM
inversion and NPI each take 9 seconds.

##  6 Conclusion

In this work, we have introduced ReNoise, a universal approach that enhances
various inversion algorithms of diffusion models. ReNoise gently guides the
inversion curve of a real image towards the source noise from which a
denoising process reconstructs the image. ReNoise can be considered as a meta-
algorithm that warps the trajectory of any iterative diffusion inversion
process. Our experiments demonstrate that averaging the last few renoising
iterations significantly enhances reconstruction quality. For a fixed amount
of computation, ReNoise shows remarkably higher reconstruction quality and
editability. The method is theoretically supported and our experiments
reconfirm its effectiveness on a variety of diffusion models and sampling
algorithms. Moreover, the method is numerically stable, and always converges
to some inversion trajectory that eases hyperparameters adjustment.

Beyond the net introduction of an effective inversion, the paper presents a
twofold important contribution: an effective inversion for few-steps diffusion
models, which facilitates effective editing on these models.

A limitation of ReNoise is the model-specific hyperparameter tuning required
for Edit Enhancement and Noise Correction. While these hyperparameters remain
stable for a given model, they may vary across models, and tuning them is
necessary to achieve high reconstruction quality while maintaining
editability.

While ReNoise demonstrates the potential for editing few-step diffusion
models, more extensive testing with advanced editing methods is needed. It is
worth noting that no such editing has been demonstrated for the few-step
diffusion models. We believe and hope that our ReNoise method will pave the
way for fast and effective editing methods based on the few-steps models. We
also believe that ReNoise can be adapted to the challenging problem of
inverting video-diffusion models.

## References

  * doi [2003] _Numerical Differential Equation Methods_ , chapter 2, pages 45–121.  John Wiley and Sons, Ltd, 2003.
  * Abdal et al. [2019] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE, 2019.
  * Abdal et al. [2020] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2020.
  * Alaluf et al. [2021a] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or.  Restyle: A residual-based stylegan encoder via iterative refinement.  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE, 2021a.
  * Alaluf et al. [2021b] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit H. Bermano.  Hyperstyle: Stylegan inversion with hypernetworks for real image editing, 2021b.
  * Alaluf et al. [2023] Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch-Elor, and Daniel Cohen-Or.  Cross-image attention for zero-shot appearance transfer, 2023.
  * Avrahami et al. [2022a] Omri Avrahami, Ohad Fried, and Dani Lischinski.  Blended latent diffusion.  _arXiv preprint arXiv:2206.02779_ , 2022a.
  * Avrahami et al. [2022b] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18208–18218, 2022b.
  * Brooks et al. [2023] Tim Brooks, Aleksander Holynski, and Alexei A. Efros.  Instructpix2pix: Learning to follow image editing instructions, 2023.
  * Burden et al. [2015] R.L. Burden, J.D. Faires, and A.M. Burden.  _Numerical Analysis_.  Cengage Learning, 2015.
  * Couairon et al. [2022] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Diffedit: Diffusion-based semantic image editing with mask guidance.  _ArXiv_ , abs/2210.11427, 2022.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis, 2021.
  * Dinh et al. [2022] Tan M. Dinh, Anh Tuan Tran, Rang Nguyen, and Binh-Son Hua.  Hyperinverter: Improving stylegan inversion via hypernetwork.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Epstein et al. [2023] Dave Epstein, Allan Jabri, Ben Poole, Alexei A. Efros, and Aleksander Holynski.  Diffusion self-guidance for controllable image generation.  2023\.
  * Ge et al. [2023] Songwei Ge, Taesung Park, Jun-Yan Zhu, and Jia-Bin Huang.  Expressive text-to-image generation with rich text.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * Han et al. [2023] Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Anastasis Stathopoulos, Xiaoxiao He, Yuxiao Chen, Di Liu, Qilong Zhangli, Jindong Jiang, Zhaoyang Xia, Akash Srivastava, and Dimitris Metaxas.  Improving tuning-free real image editing with proximal guidance, 2023.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control, 2022.
  * Hertz et al. [2023] Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or.  Style aligned image generation via shared attention.  2023\.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance, 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models, 2020.
  * Huberman-Spiegelglas et al. [2023] Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli.  An edit friendly ddpm noise space: Inversion and manipulations, 2023.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _Advances in Neural Information Processing Systems_ , 35:26565–26577, 2022.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _Conference on Computer Vision and Pattern Recognition 2023_ , 2023.
  * Khrulkov et al. [2022] Valentin Khrulkov, Gleb Ryzhakov, Andrei Chertkov, and Ivan Oseledets.  Understanding ddpm latent codes through optimal transport.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll’a r, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  _CoRR_ , abs/1405.0312, 2014.
  * Luo et al. [2023a] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao.  Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023a.
  * Luo et al. [2023b] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao.  Lcm-lora: A universal stable-diffusion acceleration module.  _arXiv preprint arXiv:2311.05556_ , 2023b.
  * Meiri et al. [2023] Barak Meiri, Dvir Samuel, Nir Darshan, Gal Chechik, Shai Avidan, and Rami Ben-Ari.  Fixed-point inversion for text-to-image diffusion models, 2023.
  * Meng et al. [2022] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations, 2022.
  * Miyake et al. [2023] Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka.  Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models, 2023.
  * Mokady et al. [2022] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models, 2022.
  * Pan et al. [2023] Zhihong Pan, Riccardo Gherardi, Xiufeng Xie, and Stephen Huang.  Effective real image editing with accelerated iterative diffusion inversion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 15912–15921, 2023.
  * Parmar et al. [2022] Gaurav Parmar, Yijun Li, Jingwan Lu, Richard Zhang, Jun-Yan Zhu, and Krishna Kumar Singh.  Spatially-adaptive multilayer selection for gan inversion and editing.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2022.
  * Parmar et al. [2023] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.  Zero-shot image-to-image translation.  In _Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings_. ACM, 2023.
  * Patashnik et al. [2023] Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or.  Localizing object-level shape variations with text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022.
  * Richardson et al. [2021] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding, 2022.
  * Salimans and Ho [2021] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In _International Conference on Learning Representations_ , 2021.
  * Sauer et al. [2023] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.  Adversarial diffusion distillation, 2023.
  * Song et al. [2022] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models, 2022.
  * Song et al. [2020] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _International Conference on Learning Representations_ , 2020.
  * Song et al. [2023] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models, 2023.
  * Tov et al. [2021] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  _ACM Trans. Graph._ , 40(4), 2021.
  * Tumanyan et al. [2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  pages 1921–1930, 2023.
  * Voynov et al. [2023] Andrey Voynov, Amir Hertz, Moab Arar, Shlomi Fruchter, and Daniel Cohen-Or.  Anylens: A generative diffusion model with any rendering lens.  2023\.
  * Wallace et al. [2022] Bram Wallace, Akash Gokul, and Nikhil Naik.  Edict: Exact diffusion inversion via coupled transformations.  _arXiv preprint arXiv:2211.12446_ , 2022.
  * Wu and De la Torre [2022] Chen Henry Wu and Fernando De la Torre.  Unifying diffusion models’ latent space, with applications to cyclediffusion and guidance.  _arXiv preprint arXiv:2210.05559_ , 2022.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _CVPR_ , 2018.
  * Zhu et al. [2020a] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou.  In-domain gan inversion for real image editing.  In _Proceedings of European Conference on Computer Vision (ECCV)_ , 2020a.
  * Zhu et al. [2016] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros.  Generative visual manipulation on the natural image manifold.  In _Proceedings of European Conference on Computer Vision (ECCV)_ , 2016.
  * Zhu et al. [2020b] Peihao Zhu, Rameen Abdal, Yipeng Qin, John Femiani, and Peter Wonka.  Improved stylegan embedding: Where are the good latents?, 2020b.

##  Appendix A Convergence Discussion

###  A.1 ReNoise As Contraction mapping

In this section, we continue the convergence discussion of our method.

#### Scaled Jacobian Norm of ReNoise Mapping

Figure 14 shows this scaled norm estimation for the SDXL diffusion model for
various steps and ReNoise iterations indices $(k)$. Remarkably, the ReNoise
indices minimally impact the scale factor, consistently remaining below 1.
This confirms in practice the convergence of the proposed algorithm. Notably,
the highest scaled norm values occur at smaller $t$ (excluding the first step)
and during the initial renoising iteration. This validates the strategy of not
applying ReNoise in early steps, where convergence tends to be slower compared
to other noise levels. Additionally, the scaled norm value for the initial $t$
approaches 0, which induces almost immediate convergence, making ReNoise an
almost identical operation.

Table 3:  Implementation details of ReNoise with Stable Diffusion [40], SDXL
[37], SDXL Turbo [43] and LCM LoRA [28].  Implementation details
---
Model Name | Stable Diffusion | SDXL | SDXL Turbo | LCM LoRA
Noise Sampler | DDIM | DDIM | Ancestral-Euler | DDIM
Number of denoising steps | 50 | 50 | 4 | 4
Number of renoising iterations | 1 | 1 | 9 | 7
Weights for $t<250$ | $w_{1},w_{2}=0.5$ | $w_{1},w_{2}=0.5$ | $w_{1},...,w_{4}=0.25$ | $w_{1},...,w_{4}=0.25$
Weights for $t>250$ | $w_{2}=1.0$ | $w_{2}=1.0$ | $w_{8},...,w_{10}=0.33$ | $w_{6},...,w_{8}=0.33$
$\lambda_{\text{pair}}$ | 10 | 10 | 10 | 20
$\lambda_{\text{patch-KL}}$ | 0.05 | 0.055 | 0.055 | 0.075

#### Validation for the Averaging Strategy

Notably, the proposed averaging strategy is aligned with the conclusions
described in the main paper and also converges to the desired stationary
point. To verify this claim we will show that if a sequence $z_{t+1}^{(k)}$
converges to some point $z_{t+1}$, then the averages
$\bar{z}_{t+1}^{(k)}=\frac{1}{k}\sum_{i=1}^{k}z_{t+1}^{(i)}$ converges to the
same point. That happens to be the stationary point of the operator
$\mathcal{G}$. We demonstrate it with the basic and standard calculus. Assume
that $z_{t+1}^{(k)}=\varepsilon_{t+1}^{(k)}+z_{t+1}$ with
$\|\varepsilon_{t+1}^{(k)}\|\to 0$ as $k\to\infty$. For a fixed $\varepsilon$
we need to show that there exists $K$ so that
$\|\bar{z}_{t+1}^{(k)}-z_{t+1}\|<\epsilon$ for any $k>K$. One has

$\bar{z}_{t+1}^{(k)}-z_{t+1}=\sum_{i=1}^{k}\frac{\varepsilon_{t+1}^{(i)}}{k}$

There exists $m$ such that $\|\varepsilon_{t+1}^{(k)}\|<0.5\cdot\varepsilon$
once $k>m$. Then we have

$\|\sum_{i=1}^{k}\frac{\varepsilon_{t+1}^{(i)}}{k}\|\leq\frac{\|\sum_{i=1}^{m}\varepsilon_{t+1}^{(i)}\|}{k}+\frac{\|\sum_{i=m+1}^{k}\varepsilon_{t+1}^{(i)}\|}{k}\leq\\\
\frac{\|\sum_{i=1}^{m}\varepsilon_{t+1}^{(i)}\|}{k}+\frac{\varepsilon}{2}\frac{\cdot(k-m)}{k}\leq\frac{\|\sum_{i=1}^{m}\varepsilon_{t+1}^{(i)}\|}{k}+\frac{\varepsilon}{2}$

given that $m$ is fixed, we can always take $k$ sufficiently large such that

$\frac{\|\sum_{i=1}^{m}\varepsilon_{t+1}^{(i)}\|}{k}<\frac{\varepsilon}{2}$

this ends the proof. The very same computation conducts a similar result if
the elements’ weights $w_{k}$ are non-equal.

Figure 14: Scaled Jacobian norm of the mapping $z^{(k)}_{t}\to z^{(k)}_{t}$ calculated for various noise levels $t$, iterations $k$. We report the average and the standard deviation calculated over 32 images. Values below 1 indicate the exponential convergence of the ReNoise algorithm. | Original
---
Image
$\longleftarrow$ Editing Results $\longrightarrow$
|  |  |
“kitten” | “broccoli” | “made of lego” | “plush toy”
|  |  |
“…in the woods” | “…in the beach” | “…in the desert” | “…in the snow”
|  |  |
“lion” | “tiger” | “monkey” | “panda”
|  |  |
“monkey” | “beaver” | “raccoon” | “…in the snow”
Figure 15:  SDXL Turbo editing results. Each row showcases one image. The leftmost image is the original, followed by three edited versions. The text below each edited image indicates the specific word or phrase replaced or added to the original prompt for that specific edit.  SDXL | SDXL Turbo | LCM
---|---|---
|  |
Figure 16:  Image reconstruction results comparing sampler reversing inversion
techniques across different samplers (e.g., vanilla DDIM inversion) with our
ReNoise method using the same sampler. The number of denoising steps remains
fixed. However, the number of UNet passes varies, with the number of inversion
steps increasing in the sampler reversing approach, and the number of
renoising iterations increasing in our method. We present various
configuration options for our method, including options with or without edit
enhancement loss and Noise Correction (NC).

##  Appendix B Implementation Details

We used BLIP-2 [25] to generate captions for the input images, which were then
used as prompts for the diffusion models. In Table 3 , we provide all
hyperparameters of ReNoise inversion per model, optimized for the best
reconstruction-editability trade-off:

Where $\\{w_{i}\\}$ are the renoising estimations averaging weights, and
$\lambda_{\text{pair}}$ and $\lambda_{\text{patch-KL}}$ are the weights we
assign to each component of the edit enhancement loss:

$\mathcal{L}_{\text{edit}}=\lambda_{\text{pair}}\cdot\mathcal{L}_{\text{pair}}+\lambda_{\text{patch-
KL}}\cdot\mathcal{L}_{\text{patch-KL}}$

##  Appendix C Additional Experiments

#### Editing Results With SDXL Turbo

Figure 15 showcases additional image editing examples achieved using our
ReNoise inversion method. These edits are accomplished by inverting the image
with a source prompt, and then incorporating a target prompt that differs by
only a few words during the denoising process.

| Original
---
Image
| Pix2Pix zero
---
| Pix2Pix zero
---
w/ ReNoise
|  |
|  “Cat” $\longrightarrow$ “Dog”
|  |
|  “Dog” $\longrightarrow$ “Cat”
Figure 17:  Zero-Shot Image-to-Image Translation editing results. This figure
compares editing results with Stable Diffusion [40] achieved using two
inversion methods: pix2pix-zero [35] and our proposed ReNoise inversion. As
observed, ReNoise inversion preserves image details while effectively
incorporating the desired edits.

#### Reconstruction and Speed

We continue our evaluation of the reconstruction-speed tradeoff from Section
5.1 in the main paper. Figure 16 presents quantitative LPIPS results for the
same configuration described in the main paper. As expected, LPIPS scores
exhibit similar trends to the PSNR metric shown previously.

|  | Original
---
Image
| Our w/o
---
Edit Losses
| Our w/
---
$\mathcal{L}_{\text{KL}}$ and $\mathcal{L}_{\text{pair}}$
| Our w/
---
$\mathcal{L}_{\text{patch-KL}}$ and $\mathcal{L}_{\text{pair}}$
| Our w/
---
Noise Correction
Override
| Our w/
---
Noise Correction
Optimize
Rec. |  |  |  |  |  |
Edit |  |  |  |  |  |
Figure 18:  Ablation study on SDXL Turbo for image editing. The first row displays reconstructed images, while the second row showcases edits replacing “cat” with “raccoon”. Each column represents a different inversion configuration. From left to right, the second column demonstrates results with renoising iterations and estimations averaging. In the following two columns we employ different edit enhancement losses. Finally, the last two columns present results using both noise correction (NC) and $\mathcal{L}_{\text{edit}}$. In the fifth column, NC overrides the noise $\epsilon_{t}$, while in the sixth column, we present the results of our full method, where NC optimizes $\epsilon_{t}$.  Ours |  |  |  |
---|---|---|---|---
Edit Friendly |  |  |  |
| Original | Reconstruction | “ginger cat” |  “wood” $\rightarrow$ “metal”
Ours |  |  |  |
Edit Friendly |  |  |  |
| Original | Reconstruction | “black&white dog” | “…with glasses”
Figure 19:  Comparison with edit-friendly DDPM Inversion with SDXL Turbo. We
invert two images with the prompts: “a cat laying in a bed made out of wood”
(top) and “a dog sitting on the beach with its tongue out” (bottom) and apply
two edits to each image.

#### Image Editing Ablation

Figure 18 visually illustrates the impact of edit enhancement losses and noise
correction when editing inverted images using SDXL Turbo [43]. While achieving
good reconstructions without the $\mathcal{L}_{\text{edit}}$ regularization,
the method struggles with editing capabilities (second column). Although the
$\mathcal{L}_{\text{edit}}$ regularization enhances editing capabilities, it
comes at the cost of reduced reconstruction accuracy of the original image, as
evident in the two middle columns. In the third column, we use
$\mathcal{L}_{\text{KL}}$ as defined in pix2pix-zero [35]. While
$\mathcal{L}_{\text{patch-KL}}$ surpasses $\mathcal{L}_{\text{KL}}$ in
original image preservation, further improvements are necessary. These
improvements are achieved by using the noise correction technique. To correct
the noise, we can either override the noise $\epsilon_{t}$ in Equation 1 in
the main paper (fifth column), or optimize it (sixth column). As observed,
overriding the noise $\epsilon_{t}$ affects editability, while optimizing it
achieves good results in terms of both reconstruction and editability.
Therefore, in our full method, we use $\mathcal{L}_{\text{patch-KL}}$,
$\mathcal{L}_{\text{pair}}$, and optimization-based noise correction.

#### Editing With ReNoise

The ReNoise technique provides a drop-in improvement for methods (e.g.,
editing methods) that rely on inversion methods like DDIM [12], negative
prompt inversion [31] and more. It seamlessly integrates with these existing
approaches, boosting their performance without requiring extensive
modifications. Figure 17 showcases image editing examples using Zero-Shot
Image-to-Image Translation (pix2pix-zero) [35]. We compared inversions with
both the pix2pix-zero inversion method and our ReNoise method. Our method
demonstrably preserves finer details from the original image while improving
editability, as exemplified by the dog-to-cat translation.

#### Inversion for Non-deterministic Samplers

In Figure 19 we show more qualitative comparisons with “an edit-friendly DDPM”
[21] where we utilize SDXL Turbo [43]. As can be seen, encoding a significant
amount of information within only a few external noise vectors,
$\epsilon_{t}$, limits editability in certain scenarios, such as the ginger
cat example. It is evident that the edit-friendly DDPM method struggles to
deviate significantly from the original image in certain aspects while also
failing to faithfully preserve it in others. For instance, it encounters
difficulty in transforming the cat into a ginger cat while omitting the
preservation of the decoration in the top left corner. In addition, the image
quality of edits produced by edit-friendly DDPM is lower, as demonstrated in
the dog example.

#### Improving DDIM Instabilities

As mentioned in Section 3.1 of the main paper, DDIM inversion [12] can exhibit
instabilities depending on the prompt. Figure 20 demonstrates this with image
reconstruction on SDXL [37] using an empty prompt. Notably, incorporating even
a single renoising iteration significantly improves inversion stability.

Original Image | DDIM Inversion | +1 ReNoise Step | Original Image | DDIM Inversion | +1 ReNoise Step
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Figure 20:  Comparing reconstruction results with an empty prompt of plain
DDIM inversion on SDXL to DDIM inversion with one ReNoise iteration.