# Delta Denoising Score

Amir Hertz*** Google Research  Tel Aviv University  Kfir Aberman  Google
Research  Daniel Cohen-Or* Google Research  Tel Aviv University

###### Abstract

We introduce Delta Denoising Score (DDS), a novel scoring function for text-
based image editing that guides minimal modifications of an input image
towards the content described in a target prompt. DDS leverages the rich
generative prior of text-to-image diffusion models and can be used as a loss
term in an optimization problem to steer an image towards a desired direction
dictated by a text. DDS utilizes the Score Distillation Sampling (SDS)
mechanism for the purpose of image editing. We show that using only SDS often
produces non-detailed and blurry outputs due to noisy gradients. To address
this issue, DDS uses a prompt that matches the input image to identify and
remove undesired erroneous directions of SDS. Our key premise is that SDS
should be zero when calculated on pairs of matched prompts and images, meaning
that if the score is non-zero, its gradients can be attributed to the
erroneous component of SDS. Our analysis demonstrates the competence of DDS
for text based image-to-image translation. We further show that DDS can be
used to train an effective zero-shot image translation model. Experimental
results indicate that DDS outperforms existing methods in terms of stability
and quality, highlighting its potential for real-world applications in text-
based image editing. For code and additional results, please visit our project
page: https://delta-denoising-score.github.io/.

\begin{overpic}[width=216.81pt,tics=10,trim=0.0pt 0.0pt 0.0pt
0.0pt,clip]{figures/dds_diagram_rev.pdf} \put(10.5,55.5){$\mathbf{\hat{z}}$}
\put(10.5,24.0){$\mathbf{z}$} \put(10.5,93.0){$\mathbf{z}$}
\par\put(38.0,55.8){$\mathbf{\hat{z}_{t}}$} \put(38.0,24.3){$\mathbf{z_{t}}$}
\put(38.0,93.3){$\mathbf{z_{t}}$}
\par\par\put(78.2,56.3){$\hat{\epsilon}_{\phi}$}
\put(78.2,24.9){$\epsilon_{\phi}$} \put(78.2,93.8){$\epsilon_{\phi}$}
\par\put(20.5,29.5){$\begin{subarray}{c}t\sim\mathcal{U}(0,1)\\\
\epsilon\sim\mathcal{N}(0,\mathbf{I})\end{subarray}$}
\put(20.5,66.0){$\begin{subarray}{c}t\sim\mathcal{U}(0,1)\\\
\epsilon\sim\mathcal{N}(0,\mathbf{I})\end{subarray}$}
\par\par\put(87.0,28.0){$\nabla_{\theta}\mathcal{L}_{\textrm{DDS}}$}
\put(54.0,66.1){$\nabla_{\theta}\mathcal{L}_{\textrm{SDS}}$} \end{overpic}
Figure 1: Score Distillation Sampling (SDS) vs. Delta Denoising Score (DDS).
Top: SDS mechanism optimizes a given image by querying the denoising model on
the noisy version of the image and a target text prompt. The resulting image
can often be blurry and unfaithful to the target prompt. Bottom: DDS queries
an additional reference branch with a matched text-prompt, and generates delta
scores that represent the difference between the outputs of the two queries.
DDS provides cleaner gradient directions that modify the edited portions of
the optimized image, while leaving the other parts unchanged. 11footnotetext:
Performed this work while working at Google.

##  1 Introduction

Large-scale language-vision models have revolutionized the way images and
visual content, in general, can be generated and edited. Recently, we have
witnessed a surge in the development of text-to-image generative models, which
utilize textual input to condition the generation of images. A promising
avenue in this field is Score Distillation Sampling (SDS) [27] – a sampling
mechanism that utilizes probability density distillation to optimize a
parametric image generator using a 2D diffusion model as a prior.

The effectiveness of the SDS stems from rich generative prior of the diffusion
model it samples from. This is in contrast to the direct use of a language-
vision model, like CLIP, which was trained using contrastive loss [28]. The
prior of large generative diffusion models, like Stable Diffusion [31],
DALLE-2 [30] and Imagen [35] is particularly rich and expressive and has been
demonstrated to be highly effective in generating visually stunning assets
across various domains, including images and 3D models, among others.

Despite its usefulness, one of the primary issues associated with SDS is its
tendency to converge towards specific modes, which often leads to the
production of blurry outputs that only capture the elements explicitly
described in the prompt.

In particular, using SDS to _edit an existing image_ by initializing the
optimization procedure from that image, may result in significant blurring of
the image beyond the edited elements.

In this paper, we introduce a new diffusion-based scoring technique for
optimizing a parametric model for the task of editing. Unlike SDS, which
queries the generative model with a pair of image and text, our method
utilizes an additional query of a reference image-text pair, where the text
matches the content of the image. Then, the output score is the difference, or
delta, between the results of the two queries (see Figure 1). We refer to this
scoring technique as Delta Denoising Score (DDS).

In its basic form, DDS is applied on two pairs of images and texts, one is a
reference image-text that remains intact during the optimization, and the
other is a target image that is optimized to match a target text prompt. The
delta scoring provides effective gradients, which modify the edited portions
of the image, while leaving the others unchanged.

The key idea is that the source image and its text description, can be used
for estimating undesirable and noisy gradients directions introduced by SDS.
Then if we want to alter only a portion of the image using a new text
description, we can use our reference estimation and get a cleaner gradient
direction to update the image.

DDS can be used as a prompt-to-prompt editing technique that can modify images
by only editing their captions, where no mask is provided or computed. Beyond
that, Delta Denoising Score enables us to train a distilled image-to-image
model without the need of paired training dataset, yielding a zero-shot image
translation technique. Training the model, requires only dataset of the source
distribution, associated with simple captions that describe the source and
target image distributions. As we will show, such zero-shot training can be
applied on a single or multi-task image translation, and the source
distribution can include synthetically generated and real images.

To demonstrate the effectiveness of our approach, we conducted experiments
comparing our model to existing state-of-the-art text-driven editing
techniques.

Figure 2: Sampling text-to-image diffusion models. Generation via SDS
optimization starting from random noises (left) vs. conventional diffusion-
based image generation (right). Both samples are generated with respect to a
given text prompt (top). Generating images based on SDS only leads to less
diverse results and mode collapse where the main subject in the text appears
in front of a blurry background.

##  2 Related Work

Text-to-Image models [34, 30, 31], have recently raised the bar for the task
of generating images conditioned on a text prompt, exploiting the powerful
architecture of diffusion models [13, 36, 39, 13, 37, 31], which can be used
to various image editing and guided synthesis tasks [32, 17, 45, 44].

Recent works have attempted to adapt text-guided diffusion models to the
fundamental challenge of single-image editing, aiming to exploit their rich
and diverse semantic knowledge. Meng et al. [21] add noise to the input image
and then perform a text-guided denoising process from a predefined step. Yet,
they struggle to accurately preserve the input image details, which were
preserved by a user provided mask in other works [24, 2, 1]. DiffEdit [7] uses
DDIM inversion for image editing, but avoids the emerged distortion by
automatically producing a mask that allows background preservation.

While some text-only editing approaches are bound to global editing [8, 20,
18, 26], Bar-Tal et al. [4] propose a text-based localized editing technique
without using any mask. Their technique allows high-quality texture editing,
but not modifying complex structures, since only CLIP [28] is employed as
guidance instead of a generative diffusion model. Prompt-to-prompt [12]
suggests an intuitive editing technique that enables manipulation of local or
global details for images that were synthesized by a text-to-image network.
[23] proposed an approach to invert real images into the latent space of the
diffusion model, such that prompt-to-prompt can be applied to real images.
Imagic [17] and UniTune[43] have demonstrated impressive text-driven editing
capabilities, but require the costly fine-tuning of the model. InstructPix2Pix
[5], plug-and-play [41] and [25] can get an instruction or target prompt and
manipulate real images towrds the desired edit

DreamFusion [27] proposed the SDS score as a 2D prior which can be used to
generate 3D assets [22, 29]. SDS is also used in [38] to direct a StyleGAN
generator for the domain adaption task. This is conceptually similar to
StyleGAN-NADA [10] which uses instead CLIP [28] to translate the domain of a
StyleGAN generator to other domains based only textual description. Our work
explores the usage of SDS score in the context of image editing, and propose a
new technique to clean the undesired gradients of SDS which grab the
optimization process into noisy direction that smooth out relevant detailed
from the original image.

Figure 3: Bias in SDS optimization. Left column: an image generated by the
prompt “Panda snowboarding”. Top rows show the difference between SDS to DDS
optimization when changing the animal in the prompt (“Panda” to “Squirrel”).
Bottom row shows SDS optimization applied using the original prompt. Even in
this case, the image becomes blurry.

##  3 Delta Denoising Score (DDS)

We begin with a brief overview of the SDS loss function and explain the
challenges in sampling and editing images with SDS, based on empirical
observations. In particular, we demonstrate that SDS introduces a noisy
direction when applied to the task of image editing. We then introduce our
Delta Denoising Score (DDS), which utilizes a reference pair of image and text
to correct the noisy direction of SDS and offers a new technique for the task
of prompt-to-prompt editing [12]. We conduct all our experiments using the
latent model– Stable Diffusion [31], nevertheless, in our overview and
results, we refer to the models latents and output channels as images and
pixels respectively.

###  3.1 SDS overview

Given an input image $\mathbf{z}$, a conditioning text embedding $y$, a
denoising model $\epsilon_{\phi}$ with parameters set $\phi$, a randomly
sampled timestep $t\sim\mathcal{U}(0,1)$ drawn from the uniform distribution,
and noise $\epsilon\sim\mathcal{N}(0,\mathbf{I})$ following a normal
distribution, the diffusion loss can be expressed as:

$\mathcal{L}_{\text{Diff}}\left(\phi,\mathbf{z},y,\epsilon,t\right)=w(t)||\epsilon_{\phi}\left(\mathbf{z_{t}},y,t\right)-\epsilon||_{2}^{2},$

where $w(t)$ is a weighting function, and $\mathbf{z_{t}}$ refers to the noisy
version of $\mathbf{z}$ obtained via a stochastic noising forward process
given by
$\mathbf{z_{t}}=\sqrt{\alpha_{t}}\mathbf{z}+\sqrt{1-\alpha_{t}}\epsilon$, with
$\alpha_{t}$ being the noise scheduler. For simplicity, we omit the weighting
factor in the remainder of this section.

The text conditioned diffusion models use classifier-free guidance (CFG [14])
that consists of two components, one that is conditioned on text input, and
another that is unconditioned. During inference, the two components are used
to denoise the image via

$\epsilon_{\phi}^{\omega}\left(\mathbf{z_{t}},y,t\right)=\left(1+\omega\right)\epsilon_{\phi}\left(\mathbf{z_{t}},y,t\right)-\omega\epsilon_{\phi}\left(\mathbf{z_{t}},t\right),$

where the components are balanced using a guidance parameter $\omega$.

Given an arbitrary differentiable parametric function that renders images,
$g_{\theta}$, the gradient of the diffusion loss function with respect to the
parameters $\theta$ is given by:

$\nabla_{\theta}\mathcal{L}_{\text{Diff}}=\left(\epsilon_{\phi}^{\omega}\left(\mathbf{z_{t}},y,t\right)-\epsilon\right)\dfrac{\partial{\epsilon_{\phi}^{\omega}\left(\mathbf{z},y,t\right)}}{\partial{\mathbf{z_{t}}}}\dfrac{\partial\mathbf{z_{t}}}{\partial\theta}.$

It has been demonstrated in [27] that omitting the U-Net Jacobian term (middle
term) leads to an effective gradient for optimizing a parametric generator
with diffusion models:

$\nabla_{\theta}\mathcal{L}_{\text{SDS}}(\mathbf{z},y,\epsilon,t)=\epsilon_{\phi}^{\omega}\left(\left(\mathbf{z_{t}},y,t\right)-\epsilon\right)\dfrac{\partial\mathbf{z_{t}}}{\partial\theta}.$
(1)

Incrementally updating the parameters of the generator in the direction of the
gradient, produces images that exhibit a higher degree of fidelity to the
prompt. However, SDS suffers from the tendency to converge towards specific
modes any relevant citation in mind, resulting in non-diverse and blurry
outputs that only highlight elements mentioned in the prompt. Figure 2
showcases a comparison between sampling Stable-Diffusion with SDS vs. sampling
it with a standard reverse process of the diffusion model, demonstrating this
issue with 2D image samples.

The original purpose of SDS was to generate samples via optimization from a
text-conditioned diffusion model. It is noteworthy that $g_{\theta}$ can be an
arbitrary parametric function that renders images. In the following sections
we demonstrate our results with $g_{\theta}=\theta$, namely, a trivial
generator that renders a single image, where the optimization variables are
the image pixels themselves, however, note that the derivation is general.

\begin{overpic}[width=216.81pt,tics=10,trim=0.0pt 0.0pt 0.0pt
0.0pt,clip]{figures/vis_flamingo.pdf}
\put(15.0,-0.5){$\nabla_{\theta}\mathcal{L}_{\text{SDS}}(\hat{\mathbf{z}},\hat{y})$}
\put(40.0,-0.5){$\nabla_{\theta}\mathcal{L}_{\text{SDS}}(\mathbf{z},y)$}
\put(69.0,-0.5){$\nabla_{\theta}\mathcal{L}_{\text{DDS}}$} \end{overpic}
Figure 4: DDS gradients. Top: Visualization of 4 steps in the DDS optimization
process, where an image of a “Flamingo rollerskating” (left) gradually
transforms into a “Stork rollerskating” (right). Bottom: By subtracting the
SDS gradients of the reference image and the source prompt (left) from the SDS
gradients of the edited image and the target prompt (middle), we obtain
cleaner DDS gradients (right).

###  3.2 Editing with SDS

The original purpose of SDS was to generate samples from a distribution
conditioned solely on a text prompt. However, we now aim to extend SDS to the
task of editing, which involves conditioning the sampling process on both an
image and text.

Our objective is to synthesize an output image $\mathbf{z}$ that incorporates
the structure and details of an input source image $\mathbf{\hat{z}}$, while
conforming to the content specified in a target prompt $y$. This is a standard
text-driven image-to-image translation problem, where modifications may be
applied locally or globally [12, 5].

One potential approach to utilize SDS is to initialize the optimization
variable with the source image $\mathbf{z}_{0}=\mathbf{\hat{z}}$ and applying
SDS while conditioning on $y$. However, we have observed that similarly to the
non image conditioned SDS, this approach leads to blurred outputs and a loss
of details, particularly those that are unrelated to the input prompt. Figure
3 (top row) demonstrates such example where the panda transforms into a
squirrel at the cost of blurring out other details.

Based on our observations, we define a decomposition for the gradients
$\nabla_{\theta}\mathcal{L}_{\text{SDS}}$ to two components: one component
$\delta_{\textrm{text}}$ is a desired direction that directs the image to the
closest image that matches the text. And another, undesired component,
$\delta_{\textrm{bias}}$ that interferes with the process and causes the image
to become smooth and blurry in some parts. Formally:

$\nabla_{\theta}\mathcal{L}_{\text{SDS}}(\mathbf{z},y,\epsilon,t)\coloneqq\delta_{\textrm{text}}+\delta_{\textrm{bias}},$
(2)

where both $\delta_{\textrm{text}}$ and $\delta_{\textrm{bias}}$ are random
variables that depend on $\mathbf{z}$, $y$, $\epsilon$ and $t$. Under this
definition, to address this issue and enable high-quality or distilled image
editing with SDS, we have to isolate and extract the text-aligned part
$\delta_{\textrm{text}}$ and follow it during the optimization while avoiding
the $\delta_{\textrm{bias}}$ direction that may take the image to unintended
places.

###  3.3 Denoising the Editing Direction

We next aim to find the noisy direction of the SDS score, when applied for
editing purposes, and remove it during the optimization process.

The gist of our method is that since we already have a source image and its
text description, they can be used for estimating the noisy direction
$\delta_{\textrm{bias}}$, that biases the edit towards undesired directions.
Then, if we want to alter only a portion of the image using a new text
description, we can use our reference estimation and get a cleaner gradient
direction to update the image. In practice, we use a reference branch that
calculates the SDS score of the given image $\mathbf{\hat{z}}$ with a
corresponding, matched, text prompt $\hat{y}$, and subtract it from the main
SDS optimization branch to yield a distilled edit.

Formally, given matched and unmatched image-text embedding pairs
$\mathbf{\hat{z}}$, $\hat{y}$, $\mathbf{z}$, $y$ respectively, the delta
denoising loss is given by:

$\mathcal{L}_{\text{DD}}\left(\phi,\mathbf{z},y,\mathbf{\hat{z}},\hat{y},\epsilon,t\right)=||\epsilon_{\phi}^{\omega}\left(\mathbf{z_{t}},y,t\right)-\epsilon_{\phi}^{\omega}\left(\mathbf{\hat{z}_{t}},\hat{y},t\right)||_{2}^{2},$

where $\mathbf{z_{t}}$ and $\mathbf{\hat{z}_{t}}$ share the same sampled noise
$\epsilon$ and timestep $t$. Then, the gradient over $g_{\theta}=\mathbf{z}$,
are given by

$\nabla_{\theta}\mathcal{L}_{\text{DD}}=\left(\epsilon_{\phi}^{\omega}\left(\mathbf{z_{t}},y,t\right)-\epsilon_{\phi}^{\omega}\left(\mathbf{\hat{z}_{t}},\hat{y},t\right)\right)\dfrac{\partial{\epsilon_{\phi}^{\omega}\left(\mathbf{z_{t}},y,t\right)}}{\partial{\mathbf{z_{t}}}}\dfrac{\partial\mathbf{z}}{\partial\theta}.$

Again, we omit the differentiation thorough the diffusion model to obtain the
Delta Denoising Score,

$\nabla_{\theta}\mathcal{L}_{\text{DDS}}=\left(\epsilon_{\phi}^{\omega}\left(\mathbf{z_{t}},y,t\right)-\epsilon_{\phi}^{\omega}\left(\mathbf{\hat{z}_{t}},\hat{y},t\right)\right)\dfrac{\partial\mathbf{z}}{\partial\theta}.$
(3)

We state that DDS pushes the optimized image into the direction of the target
prompt without the interference of the noise component, namely,
$\nabla_{\theta}\mathcal{L}_{\text{DDS}}\approx\delta_{\text{text}}$.

By adding and subtracting $\epsilon$ from the term in (3), we can represent
DDS as a difference between two SDS scores:

$\displaystyle\nabla_{\theta}\mathcal{L}_{\text{DDS}}$ $\displaystyle=$
$\displaystyle\nabla_{\theta}\mathcal{L}_{\text{SDS}}(\mathbf{z},y)-\nabla_{\theta}\mathcal{L}_{\text{SDS}}(\hat{\mathbf{z}},\hat{y}).$
(4)

We first claim that the score provided by the reference branch is equivalent
to the noisy direction. This is because, ideally, a matched image-text pair
should have a low average SDS gradient across various timesteps and noise
instances. Therefore, any non-zero gradient can be attributed to the noisy
direction, thus,

$\nabla_{\theta}\mathcal{L}_{\text{SDS}}(\mathbf{\hat{z}},\hat{y})=\hat{\delta}_{\text{bias}}.$
(5)

Evidently, the score of a matched text-to-image pair is non-zero. As can be
seen in Figure 3 (bottom, row), even when the optimization process starts with
an image that was generated by the text, there are gradients that pull the
image towards the non-desired modes. For further empirical results of the
estimation of $\delta_{\text{bias}}$, please refer to Section 5.

We next claim that the noisy component $\delta_{\text{noise}}$ of closely
related images (e.g., images with similar structure that were created with
close prompts) is similar. This is demonstrated in the DDS evaluation
experiment in Section 5 and in Figure 8 which shows that the consine
similarity between the directions of the matched pair is high. This means that
$\delta_{\textrm{bias}}\approx\hat{\delta}_{\textrm{bias}}$.

By combining the conclusions drawn from the above-mentioned experiments, we
get
$\nabla_{\theta}\mathcal{L}_{\text{DDS}}\approx\hat{\delta}_{\textrm{text}}$,
which indicates that our DDS can be considered a distilled direction that
concentrates on editing the relevant portion of the image, such that it
matches the target text.

Figure 4 visualizes the key idea behind DDS, The figure shows the two noisy
SDS scores, of the matched and unmatched pair, along with their difference,
which comprises DDS. Notably, subtracting the two noisy scores produces a
clear and concise score that concentrates solely on the targeted modification
in the image.

\begin{overpic}[width=216.81pt,tics=10,trim=0.0pt 0.0pt 0.0pt
0.0pt,clip]{figures/flamingo_cfg.pdf} \par\put(34.0,0.0){$\omega{}=3$}
\put(61.0,0.0){$15$} \put(86.0,0.0){$30$} \end{overpic} Figure 5: DDS
optimization results using different values of the classifier-free guidance
scale $\omega{}$. On one hand, using small values of $\omega{}$ leads to slow
convergence and low fidelity to the text prompt. On the other hand, using
large values of $\omega{}$ results in low fidelity to the input image.

#### Effect of CFG on DDS

As previously noted, the Classifier Free Guidance (CFG) parameter $\omega{}$,
regulates the relative influence of the text-conditioned and unconditional
components of the denoising objective. Apparently, despite the subtraction of
the two distinct branches in DDS, $\omega{}$ still has a discernible impact on
the resulting image output. Our experiments show that small values of
$\omega{}$ yield slower convergence rates and a correspondingly diminished
fidelity to the text prompt, while larger $\omega{}$ values result in an
attenuated fidelity to the input image. This observed phenomenon is visualized
in Figure 5 and empirically evaluated in Section 5.

##  4 Image-to-Image Translation

With our Delta Denoising Score, we can apply a direct optimization over the
image pixel space, i.e. optimizing for $z=\theta$ as illustrated in Figure 1.
However, optimizing an image for each editing operation presents several
drawbacks. Firstly, it necessitates captions for both the input and the
desired edited image. Secondly, the results obtained on real images are
inferior to those obtained from synthetic images. Lastly, the time required
for inference is long ($\sim$20 seconds per edit). To overcome these
limitations, we introduce a novel unsupervised training pipeline for text-
driven image-to-image translation based on our proposed DDS.

\begin{overpic}[width=216.81pt,tics=10,trim=0.0pt 0.0pt 0.0pt
0.0pt,clip]{figures/i2i_diagram.pdf} \put(15.0,58.0){$\mathbf{\hat{z}}$}
\put(83.0,58.0){$\mathbf{z}$}
\put(43.5,13.0){$\nabla_{\theta}\mathcal{L}_{\textrm{DDS}}$}
\put(49.0,39.5){$\theta$} \end{overpic} Figure 6: Unsupervised training for
multi task image-to-image translation network. Given an input image
$\mathbf{\hat{z}}$ (left) and a sampled task embedding (top), our network is
trained using the Delta Denoising Score (DDS) and corresponding text
embeddings (bottom) that describe the input image and the desired edited image
result $\mathbf{z}$. During inference, our network can then translate
arbitrary real images based on the specified task, within a single feedforward
pass.

#### Unsupervised training with DDS

Using DDS, we introduce an unsupervised training framework for a neural
network that learns to translate images based on a caption that describes a
_known source distribution_ and another caption that describes an _unknown
target distribution_. Given a dataset of source images
$\\{\mathbf{\hat{z}}_{i}\\}$, source caption $\hat{y}$ and a target caption
$y$, our goal is to learn a mapping $\mathbf{z}=g_{\theta}(\mathbf{\hat{z}})$
such that $\mathbf{z}$ has high fidelity to both: the input image
$\mathbf{\hat{z}}$ and to the target caption $y$. As illustrated in Figure 6,
on the bottom, we utilize the DDS formulation in (4) to optimize our network.

Naturally, we can extend the network capabilities to be task conditioned.
Under those settings, the network learns a finite set of $M$ image-translation
tasks that are defined by multiple target captions $\\{y_{i}\\}_{j=1}^{M}$ and
corresponding learned task embeddings $\\{k_{j}\\}_{j=1}^{M}$, see Figure 6.
At each optimization iteration, we sample a source image
$\mathbf{\hat{z}}_{i}$ with its source caption $\hat{y}_{i}$, a task embedding
$k_{j}$ with the corresponding target caption $y_{j}$. Then the network is
optimized by the DDS 4 where
$\mathbf{z}=g_{\theta}(\mathbf{\hat{z}}_{i}|k_{j})$.

To maintain the fidelity to the input image, we add a weighted identity
regularization term:

$\mathcal{L}_{\text{ID}}=\lambda_{id}(t)||g_{\theta}(\mathbf{\hat{z}}_{i}|k_{j})-\mathbf{\hat{z}}_{i}||_{2}^{2},$

where the weight $\lambda_{\text{ID}}(t)$ is a function of the training
iteration $t$, such that at the beginning of the training, we inject prior
knowledge on the desired output, and gradually reduce it during training with
a cosine decay.

Figure 7: Ablation study. We train a cat-to-lion image translation network
under various settings. The first and second columns show the input and output
results of our full method, respectively. The third row shows the results when
training without CFG warmup, and the last column shows the results when
training with SDS instead of DDS.

#### DDS with CFG warmup

During the training of the aforementioned network, we experienced a familiar
mode collapse phenomena associated with the training of generative adversarial
network (GAN) [11], where the network optimizations led to a local minima. In
our case, the network has learned to produce a fixed object, in a fixed
location within the input image, as demonstrated in Figure 7, where the same
type of lion appears in the same pose and locations in all the outputs without
respecting the input image. The reason for the mode collapse in our case can
be explained thorough the analogy to GANs. The discriminator output score that
discriminates between real and fake images can be replaced by the delta
denoising score. At a local minimum point, our network succeeded to fool the
DDS such that the output has high fidelity to $y$ at the fixed region and high
fidelity to $\mathbf{\hat{z}}$ elsewhere.

To address this issue we have found that implementing a warmup scheduler for
the classifier free guidance parameter $\omega$, utilized in the estimation of
the DDS gradient can be effective. As we have demonstrated earlier, adopting a
low value for $\omega$ during zero-shot optimization is associated with a
notably slow convergence rate. Conversely, high values push the image
aggressively towards $y$ and lead the training to mode collapse. By gradually
increasing the guidance scale, the network gradually learns to make larger
changes to the input image with respect to the translation task and avoids
local minima.

\begin{overpic}[width=106.23808pt,tics=10,trim=0.0pt 0.0pt 0.0pt
0.0pt,clip]{figures/sds_graph.pdf} \put(43.0,1.0){Timestep $t$}
\put(1.0,29.0){\rotatebox{90.0}{SDS Norm}}
\end{overpic}\begin{overpic}[width=106.23808pt,tics=10,trim=0.0pt 0.0pt 0.0pt
0.0pt,clip]{figures/dds_graph.pdf} \put(43.0,1.0){Timestep $t$}
\put(3.5,29.0){\rotatebox{90.0}{DDS Cosine}} \end{overpic}

Figure 8: Expected SDS gradients. Left: Expected SDS norm
$||\nabla\mathcal{L}_{\text{SDS}}(z,y)||_{2}$ across different timesteps for
matched (blue curve) and unmatched (orange curve) synthetic image-text pairs.
Right: Cosine similarity between the SDS directions in (4) on matched (blue)
and unmatched (orange) images from the InstructPix2Pix dataset [5].

##  5 Evaluations and Experiments

In this section we evaluate our observation regarding the SDS and DDS scores,
compare our approach to other state-of-the-art zero-shot editing methods and
conduct an ablation study to show the effectiveness of different choices in
our system.

#### SDS evaluation

We measure the expected SDS norm as a function of the timestamp $t$ for
matched and unmatched image-text pairs. The matched pairs obtained by
generating images using Stable Diffusion [31] with subset of $100$ captions
from COCO validation dataset [6]. Then, for each image $\mathbf{z}$, caption
$y$ and timestep $t$ we estimate the value
$\mathbb{E}_{\epsilon\sim\mathcal{N}(0,\mathbf{I})}||\nabla_{\mathbf{z}}\mathcal{L}_{\textrm{SDS}}(\mathbf{z},y)||_{2}$
by averaging the result of $200$ measurements and report the average value of
the $100$ estimations. To provide a reference, we also perform the experiment
on $100$ unmatched image-text pairs obtained by permuting the captions of the
matched set. The results are shown in Figure 8 (left). As can be seen, SDS
exhibits non-negligible high gradients for matched pairs. In addition, the gap
between matched and unmatched pairs supports our observation in Section 3 that
there is an inherent noise direction $\delta_{\textrm{bias}}$ in the SDS
gradient.

#### DDS evaluation

Next, we evaluate our estimation that for a matched pair of similar images
with their corresponding text, the SDS noise directions,
$\delta_{\textrm{bias}}$ and $\hat{\delta}_{\textrm{bias}}$, are correlated.
For this experiment we use a subset of 10000 synthetic image pairs
$\mathbf{z}$ and $\mathbf{\hat{z}}$ with their corresponding captions $y$ and
$\hat{y}$ from InsturctPix2Pix [5] dataset . For each timestamp, we estimate
the cosine similarity between
$\nabla_{\mathbf{z}}\mathcal{L}_{\textrm{SDS}}(\mathbf{z},y)$ and
$\nabla_{\mathbf{\hat{z}}}\mathcal{L}_{\textrm{SDS}}(\mathbf{\hat{z}},\hat{y})$
and report the average result across all pairs. Here again, we applied the
same experiment to unmatched pairs for reference. Note that the caption for
each SDS estimation remained aligned to its image. The results are summarized
in Figure 8, on right. As can be seen, the matched pairs are strongly
correlated which supports our assumption that an estimation for
$\delta_{\textrm{bias}}$ of reference image and text can be used the eliminate
the same term from similar pair.

Figure 9:  Zero shot image editing quantitative comparison. Using our DDS
optimization technique, we tested various CFG values on a dataset of 1000
images and prompts from the InstructPix2Pix training set [5], and compare our
approach to SDEdit [21] with different numbers of forward diffusion (noise
addition) steps and Plug-and-Play (PnP) [41]. Our outputs have higher fidelity
to the source images (low LPIPS scores) while also achieving high fidelity to
the edits described in the text prompts (high CLIP scores). Figure 10:  Zero
shot image editing qualitative comparison. We compare our approach to SDEdit
[21] Plug-and-Play (PnP) [41] and prompt-to-prompt [12]. Our method showcases
its ability to better apply both structural and color changes described in the
target text prompt while simultaneously preserving high fidelity to the input
image.

#### Comparison to zero-shot editing methods

To evaluate our editing capability using a direct DDS optimization over the
pixel space of a synthetic generated image, we use a randomly selected subset
of $1000$ pairs of source and target prompts from the dataset of
InsturctPix2Pix [5]. The dataset already includes the paired image results
obtained by Prompt-to-Prompt (P2P) [12], from which we took only the source
images. For each editing result we measure the text-image correspondence using
CLIP score[28]. In addition, we evaluate the similarity between the original
and the edited images using the LPIPS perceptual distance [46]. We compare our
method to additional zeros shot methods: SDEdit [21], and Plug-and-Play (PnP)
[41]. It can be seen in Figure 10 that comparing to other methods, our
approach demonstrates higher fidelity to the text prompt and to the source
image on average. The quantitative results are summarized in Figure 9 where we
show the metrics of our method for different numbers of classifier free
guidance scale. Notice, that as observed in Figure 5, the improvement to the
fidelity to text that obtained by using a large value of CFG is negligible
compared to the deterioration in the fidelity to the source image.

Table 1: Quantitative comparison for the multi-task image-to-image translation network. We measure text-image correspondence using CLIP [28]. In addition, we evaluate the similarity between the original and the edited images using the LPIPS [46] perceptual distance. |  CLIP score $\uparrow$ |  LPIPS $\downarrow$
---|---|---
PnP | $0.221\pm 0.036$ | $0.31\pm 0.075$
InstructPix2Pix | $0.2190\pm 0.037$ | $0.322\pm 0.215$
DDS (ours) | $\mathbf{0.225\pm 0.031}$ | $\mathbf{0.104\pm 0.061}$
Figure 11: Image-to-Image translation comparison. Our multi-task network was
trained to translate cats to different animals (lion, dog, squirrel) using
DDS. It was trained on synthetic cat photos and evaluated on a subset of real
images from the COCO and Imagenet datasets. Our results (second row) better
preserve the structure of the cat in the input image as well as its
background.

#### Image-to-image translation training

We train different multi-task networks as described in Section 4. For each
training instance, we generate a synthetic dataset of $5000$ images using the
Stable Diffusion model conditioned on manually written captions (5-20 captions
for each dataset). Each training starts from a pre-trained Stable Diffusion
model, modified as follows: The latent noise inputs are replaced with latents
of images from our synthetic dataset. The text embedding condition is replaced
with our learned task embeddings, initialized with text embedding that
describes the task embedding. For example, for the task of adding snow in an
image, we use the phrase snowing. Finally, the timestep of the diffusion
process is no longer used since our model inference process contains a single
feed-forward. Therefore, we re-use the timestep condition as additional per-
task _learned_ embedding which is initialized with a positional embedding of
$t=0.5$. While the text condition is injected by cross attention, time by
adaptive group normalization (ADAGN). Additional implementation details are
provided in the supplementary material.

#### Image-to-image translation comparison

We evaluate a _Cat-to-Other_ network trained to translate images of cats to
images of four different animals: a dog, a lion, a bear, and a squirrel. We
tested our network using a collection of $500$ cat images from ILSVRC [33] and
COCO [6] validation set; overall, we tested the results of $2000$ image
translations. We use the same LPIPS and CLIP scores to estimate fidelity to
the source image and target text that describes the target distribution, for
example, A photo of a lion. We compare our method to PnP [41] and
InstructPix2Pix [5] which also utilize the generative prior of a pre-trained
diffusion model for the task of image-to-image translation. Unlike our method,
InstructPix2Pix fine tune a diffusion model using synthetic pairs of images
and therefor it is sensitive to quality of the pairing method.

The results are summarized in Table 1 and Figure 11. As can be seen, our
method achieves both: better fidelity to the input image and to desired target
domain. Additionally, our method operates via a single feed-forward pass
during inference, making it $\times 50$ faster than the other diffusion-based
methods that require a full iterative diffusion process in each inference
sampling. A qualitative comparison is shown in Figure 11. As can be seen, our
method better preserves the structure of the cat when translating to other
animals. Moreover, our distilled training results in a more robust network
that can better distinguish between regions in the image that had to be
changed and areas to preserve.

#### Ablation Study

We evaluate key components of our image-to-image translation network on a
single task of cat-to-lion image translation. First, we show our results
without the CFG scaling warmup. As shown in Figure 7 (third column), it
results in mode collapse where roughly the same lion appears in the same
location regardless of the cat in the input. In addition, we train a network
with the _Vanilla_ SDS term instead of our DDS while the other components, the
$\mathcal{L}_{\text{ID}}$ term and the CFG warmup, remain untouched and
prevent the mode collapse. As can be seen (right column in Figure 7), the
quality of the translation to a lion is worse than our full settings.
Moreover, the SDS training struggles to preserve high-frequency details in the
input image. For example, see the patterns of the purple wool hat in the first
row.

##  6 Conclusions , Limitations and Future work

Figure 12: Limitations. Biases of the diffusion model or limitations in
language understanding affect the DDS optimization. Top: we would like to
change the color of the curtains in the bedroom, but the color of the pillows
is also changed. Bottom: replacing the dog’s reflection with a shadow also
causes changes in the lighting, weather and background details.

We have presented, Delta Denoising Score, a new diffusion scoring technique
that allows optimizing a given image as means to edit it with respect to a
text-prompt. Delta Denoising Score uses the SDS score applied to input image
to calculate cleaner gradients during the optimization, which leads to a
distilled edit. We have also showed an image-to-image translation model
trained with our new score. The model is training with no supervision,
requires no pairs of images, and thus can be trained on real images.

Our Delta Denoising Score works well in distilling text-driven image-to-image
translations. However, there are cases that the results are imperfect due to
the inherent limitation of the text-to-image model, in particular its language
model. A noticeable problem is the binding of adjectives to the nouns. A
typical example is demonstrated in Figure 12 (top), where the orange color is
not well bind to the curtains, and thus licks to the entire bedroom. Another
example, displayed in the bottom row, where the dog’s reflection is replaced
with a shadow and caused unwanted changes in the background details.

We also acknowledge that the multi-task model can be better trained and may be
further improved by combining multiple experts training [3], which uses
multiple network, or utilize subset of paired data and train our network under
semi-supervised settings.

The scope of Delta Denoising Score is wide, and its generalization across
various editing tasks [5] should be explored in the future. Furthermore, we
believe that it can be extended to other modalities, such as text-driven 3D
shape editing, video editing and motion editing [22, 16, 40].

The objective of this work is to extract efficient and clean gradients that
can facilitate the optimization of an image towards a distilled edit. This, we
believe, is an important step towards enhancing our understanding of how to
effectively extract and utilize the rich knowledge that is concealed within
large-scale generative models.

##  7 Acknowledgement

We thank Ben Poole, Elad Richardson, Ron Mokady and Jason Baldridge for their
valuable inputs that helped improve this work.

## References

  * [1] Omri Avrahami, Ohad Fried, and Dani Lischinski.  Blended latent diffusion.  arXiv preprint arXiv:2206.02779, 2022.
  * [2] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218, 2022.
  * [3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu.  ediff-i: Text-to-image diffusion models with ensemble of expert denoisers.  arXiv preprint arXiv:2211.01324, 2022.
  * [4] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  arXiv preprint arXiv:2204.02491, 2022.
  * [5] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  arXiv preprint arXiv:2211.09800, 2022.
  * [6] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In Computer vision and pattern recognition (CVPR), 2018 IEEE conference on. IEEE, 2018.
  * [7] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Diffedit: Diffusion-based semantic image editing with mask guidance.  arXiv preprint arXiv:2210.11427, 2022.
  * [8] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff.  Vqgan-clip: Open domain image generation and editing with natural language guidance.  arXiv preprint arXiv:2204.08583, 2022.
  * [9] John Duchi, Elad Hazan, and Yoram Singer.  Adaptive subgradient methods for online learning and stochastic optimization.  Journal of Machine Learning Research, 12(61):2121–2159, 2011.
  * [10] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  arXiv preprint arXiv:2108.00946, 2021.
  * [11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [12] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [13] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [14] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [15] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [16] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel.  Layered neural atlases for consistent video editing.  ACM Transactions on Graphics (TOG), 40(6):1–12, 2021.
  * [17] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Hui-Tang Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  ArXiv, abs/2210.09276, 2022.
  * [18] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426–2435, 2022.
  * [19] Diederik P Kingma, J Adam Ba, and J Adam.  A method for stochastic optimization. arxiv 2014.  arXiv preprint arXiv:1412.6980, 106, 2020.
  * [20] Gihyun Kwon and Jong Chul Ye.  Clipstyler: Image style transfer with a single text condition.  arXiv preprint arXiv:2112.00374, 2021.
  * [21] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Image synthesis and editing with stochastic differential equations.  arXiv preprint arXiv:2108.01073, 2021.
  * [22] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  arXiv preprint arXiv:2211.07600, 2022.
  * [23] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  arXiv preprint arXiv:2211.09794, 2022.
  * [24] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [25] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.  Zero-shot image-to-image translation.  arXiv preprint arXiv:2302.03027, 2023.
  * [26] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  arXiv preprint arXiv:2103.17249, 2021.
  * [27] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020, 2021.
  * [29] Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al.  Dreambooth3d: Subject-driven text-to-3d generation.  arXiv preprint arXiv:2303.13508, 2023.
  * [30] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [32] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  arXiv preprint arXiv:2208.12242, 2022.
  * [33] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
  * [34] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [35] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  arXiv preprint arXiv:2104.07636, 2021.
  * [36] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [37] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In International Conference on Learning Representations, 2020.
  * [38] Kunpeng Song, Ligong Han, Bingchen Liu, Dimitris Metaxas, and Ahmed Elgammal.  Diffusion guided domain adaptation of image generators.  arXiv preprint arXiv:2212.04473, 2022.
  * [39] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  Advances in Neural Information Processing Systems, 32, 2019.
  * [40] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or.  Motionclip: Exposing human motion generation to clip space.  arXiv preprint arXiv:2203.08063, 2022.
  * [41] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  arXiv preprint arXiv:2211.12572, 2022.
  * [42] Unsplash.  Unsplash image collection, 2022.
  * [43] Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan.  Unitune: Text-driven image editing by fine tuning an image generation model on a single image.  arXiv preprint arXiv:2210.09477, 2022.
  * [44] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or.  Sketch-guided text-to-image diffusion models.  arXiv preprint arXiv:2211.13752, 2022.
  * [45] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman.  $p+$: Extended textual conditioning in text-to-image generation.  arXiv preprint arXiv:2303.09522, 2023.
  * [46] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.

Appendix

##  Appendix A Societal Impact

Our method introduces an unsupervised editing framework for images. Our
framework might be exploited for producing fake content, however, this is a
known problem, common to all image editing techniques. Moreover, our method
relies on generative priors of a large text-to-image diffusion models that
might contain undesired biases due to the auto-filtered enormous dataset that
they were trained on. Those undesired biases could infiltrate to our editing
results thorough our optimization process or our distilled networks. However,
we believe that our DDS optimization technique can help in the future to
reveal such undesired biases and editing directions, similarly to the way it
operates today to clean noisy undesired components within editing directions.

##  Appendix B Implementation Details

#### Zero-shot DDS optimization

Unless specified otherwise, in all of our zero shot experiments, we initialize
the latent image $\mathbf{z}$ by the reference latent image $\mathbf{\hat{z}}$
and apply our DDS optimization for 200 iterations ($\sim 18$ seconds on a
single A100 GPU) using SGD with learning rate of $2$ and applying learning
rate decay of $0.9$ in intervals of $20$ iteration.

#### Image-to-image network training

Our image-to-image networks were trained using a batch size of $2$ for
$125000$ iterations ($\sim 23$ hours on a single A100 GPU). We use the Adam
optimizer with a learning rate of $10^{-5}$ and learning rate warmup over
$10000$ iterations using a linear scheduler. For the DDS, we set the
classifier-free guidance scale (CFG) to $25$ using CFG warmup starting from
$1$ over $20000$ iterations using a cosine scheduler. For the identity
regularization, we start with $\lambda_{\textrm{id}}=3$ and cool it down to
$0.1$ over $20000$ iterations using a cosine scheduler.

##  Appendix C Additional Ablations

#### Number of optimization steps

We set the number of optimization steps to $200$, as we found it sufficient
for most desired edits. However, for obtaining large structural changes or
color modifications, the number of optimization steps should be increased. See
example in Figure 13 where structural modification, of replacing a flamingo to
an elephant requires $400$ steps, while modify the flamingo to a bronze
sculpture converges faster. Notice that in some cases, where we seemingly
request for a large modification, like replacing the flamingo to a giraffe,
$200$ optimization are still sufficient since the overall structure of the
giraffe is similar to that of the flamingo.

#### Optimizer Selection

We tested our DDS optimization using both SGD and Adam [19] optimizer and
found that using _vanilla_ SGD leads to higher quality results. Figure 14
compares the two alternatives (top rows) with respect to the baseline SDS with
SGD optimization (bottom row). We can clearly see that both DDS optimization
achieve higher quality results compared to the baseline. However, by looking
at the accumulated difference of the image during the optimization (right
saliency map beneath each image), we can see that the Adam based optimization
results with more changes and artifacts that are not related to editing
prompt. The reason for the change in the quality can be explained through the
adaptive nature of Adam. For simplicity, consider the simpler update rule of
Adagrad [9]:

$\theta_{t}\leftarrow\theta_{t-1}-\gamma\dfrac{g_{t}}{\sqrt{\sum_{i=1}^{t}g^{2}_{t}}},$

where $t$ enumerates the optimization steps, $g_{t}$ is the gradient of
$\theta$ calculated at step $t$ and $\gamma$ is the optimization learning
rate. We can see that the normalization to the gradients may magnify outlier
gradients or reduce the weight of _good_ gradients. Figure 14 visualize such
update to the pixels for different steps across the optimization (left
saliency map beneath each image). It can be seen that the Adam based
optimization (middle row) leads to a uniform update across the pixels compared
to SGD (top row), where most of the energy is located at the pixels that are
relevant to the edit.

#### Regularized SDS optimization

An alternative to our DDS approach is to use the SDS with additional
regularization that prevents large changes in the edited image $\mathbf{z}$
with respect to the input $\mathbf{\hat{z}}$. The simplest choice is to add
regularise the optimization with a weighted $\mathcal{L}_{2}$ loss between
$\mathbf{z}$ and $\mathbf{\hat{z}}$. In this setting, the gradient of
$\mathbf{z}$ is given by:

$\nabla_{\mathbf{z}}=\nabla_{z}\textrm{SDS}+\lambda_{id}\left(\mathbf{z}-\mathbf{\hat{z}}\right),$

Where $\lambda_{id}$ is the weight for the regularisation $\mathcal{L}_{2}$
loss. Figure 15 shows the results of regularized SDS optimizations using
increasing weight for the regularization term. As expected, increasing the
value of $\lambda_{id}$ harms the fidelity to the edit prompt, while the
blurriness side-effect of SDS cannot be avoided.

##  Appendix D Additional Results

#### Zero-shot image editing results on real images

Additional image editing results using DDS optimization are shown in Figures
16 and 17. All results applied on real images from COCO, and Unsplash datasets
[6, 42]. Notice that our method works with simple input prompts describing the
edit we want to apply to the image. We use a reference text $\hat{y}$ only in
cases where we want to apply the edit over a specific object. Otherwise, we
set $\hat{y}$ to the embedding of the null text.

#### Image-to-image networks results

Additional results are shown in Figures 18 and 19. The first network was
trained to change the material of a sofa in an input image. The network was
trained on a synthetic dataset containing $5000$ images of living rooms. An
additional network was trained to synthesize different flowers in images of
potted plants. The network was trained on a synthetic dataset containing
$5000$ of potted plants in living rooms, kitchens, in gardens, and in city
streets. Finlay, we train different networks to modify a person in an input
image to other characters. Synthesizing images of persons using Stable
Diffusion usually results in poor results. Therefore we train our _character_
networks over FFHQ in-the-wild dataset (unaligned) [15]. During training, we
set a single pair of fixed prompts for the DDS optimization, A photo of a
person. for the embedding $\hat{y}$, and for the target embedding $y,$ we
replace the word person with one of the characters: a Claymation character, A
sculpture, a 3D Pixar character and to a zombie. All results are shown over
images from ILSVRC [33] and COCO [6] datasets.

Figure 13: DDS Optimization convergence. Some challenging edits, such as
changing a flamingo to an elephant, may require more time to converge compared
to other edits. In most cases, we have found that $200$ steps are sufficient.
\begin{overpic}[width=496.85625pt,tics=10,trim=0.0pt 0.0pt 0.0pt
0.0pt,clip]{figures/supp_grad_vis.pdf} \put(4.5,6.5){$\Delta\mathbf{z}$}
\put(10.0,6.5){$\sum\Delta\mathbf{z}$} \end{overpic} Figure 14: Optimizer
selection. Each row shows a zero-shot editing optimization sequence using
different settings. The green-to-red heatmaps (left map beneath each image)
show the norm of the latent pixels update in a specific step. The blue-to-red
heatmaps (right map) show the accumulated difference norm with respect to
input image (top).  \begin{overpic}[width=496.85625pt,tics=10,trim=0.0pt 0.0pt
0.0pt 0.0pt,clip]{figures/supp_sds_reg.pdf}
\put(0.0,65.5){\rotatebox{90.0}{$\lambda_{\textrm{id}}=0$}}
\put(0.0,54.5){\rotatebox{90.0}{$1$}} \put(0.0,40.5){\rotatebox{90.0}{$2$}}
\put(0.0,26.5){\rotatebox{90.0}{$5$}} \put(0.0,11.5){\rotatebox{90.0}{$10$}}
\end{overpic} Figure 15: Regularized SDS optimization as a baseline. We
combined a _vanilla_ SDS optimization with a weighted $\mathcal{L}_{2}$ loss
between the optimized image and the input image (top). From top row to bottom:
we use a larger weight ($\lambda_{\textrm{id}}$) for the $\mathcal{L}_{2}$
loss. As can be seen, by modifying $\lambda_{\textrm{id}}$ we can trade off
between fidelity to the text prompt and fidelity to the input image. Figure
16: Zero-shot image editing using DDS optimization on real images. Using DDS,
we can apply a variety of edits over real images using simple input prompt
descriptions (like “house”, or even an empty prompt). The editing operations
are mask free and may contain global descriptions, for example, changing the
lighting in the image, or local descriptions, such as changing a house to an
igloo.

Figure 17: Zero-shot image editing using DDS optimization on real images.
Using DDS, we can apply a variety of edits over images using simple input
prompt descriptions (like “Coffee”, or even an empty prompt). The editing
operations are mask free and may contain structural changes, for example,
changing a bicycle to a Vespa, or stylistic changes, such as changing coffee
to a matcha drink. Figure 18: Unsupervised multi-task image-to-image
translation– sofas network results. The network was trained to change the
color and material of the sofa in the input image. The network was trained on
synthetic images of living rooms and tested on real living rooms images from
ILSVRC [33] and COCO [6] validation sets. Figure 19: Unsupervised multi-task
image-to-image translation– flowers network results. The network was trained
to change to add different flowers potted plant in the input image. The
network was trained on synthetic images and tested on real images of
flowerpots [33] and potted plant [6].


# Development towards high-resolution kHz-speed rotation-free volumetric
imaging

Eleni Myrto Asimakopoulou  1,* Valerio Bellucci  2,* Sarlota Birnsteinova  2
Zisheng Yao  1 Yuhe Zhang  1 Ilia Petrov  2 Carsten Deiter  2 Andrea Mazzolari
3,4 Marco Romagnoni  3,4 Dusan Korytar  5,6 Zdenko Zaprazny  6 Zuzana
Kuglerova  7 Libor Juha  7 Bratislav Lukić  8 Alexander Rack  8 Liubov
Samoylova  2 Francisco Garcia Moreno  10,11 Stephen A. Hall  12 Tillmann Neu
10,11 Xiaoyu Liang  13 Patrik Vagovic  2,9 and Pablo Villanueva-Perez1
1Synchrotron Radiation Research and NanoLund, Lund University, Sweden
2European XFEL GmbH – The European X-Ray Free-Electron Laser, Schenefeld,
Germany
3Ferrara University, Department of Physics and Earth Science, Via Saragat 1,
44122 Ferrara, Italy
4INFN, Ferrara Division, Via Saragat 1, 44122 Ferrara, Italy
5Integra TDS, s.r.o., Krakovany, Slovakia
6Institute of Electrical Engineering SAS, Dúbravská cesta 9, 841 04 Bratislava
7Department of Radiation and Chemical Physics, Institute of Physics, Czech
Academy of Sciences, Prague, Czech Republic
8ESRF - The European Synchrotron, Grenoble, France
9Center for Free-Electron Laser Science (CFEL), DESY, Hamburg, Germany
10Institut für Werkstoffwissenschaften und -technologien, Technische
Universität Berlin, Hardenbergstr. 36 10623 Berlin
11Helmholtz-Zentrum Berlin für Materialien und Energie GmbH (HZB), Hahn-
Meitner-Platz 1, 14109 Berlin
12Division of Solid Mechanics, Lund University, Ole Römers Väg 1, 223 63 Lund,
Sweden
13Institute of Multidisciplinary Research for Advanced Materials (IMRAM),
Tohoku University, 2-1-1 Katahira, Aoba-ku, Sendai, Miyagi 980-8577, Japan
<EMAIL_ADDRESS><EMAIL_ADDRESS>
††journal: opticajournal††articletype: Research Article

X-ray multi-projection imaging (XMPI) provides rotation-free 3D movies of
optically opaque samples. The absence of rotation enables superior imaging
speed and preserves fragile sample dynamics by avoiding the shear forces
introduced by conventional rotary tomography. Here, we present our XMPI
observations at the ID19 beamline (ESRF, France) of 3D dynamics in melted
aluminum with 1000 frames per second and 8 $\mu$m resolution per projection
using the full dynamical range of our detectors. Since XMPI is a method under
development, we also provide different tests for the instrumentation of up to
3000 frames per second. As the flux of X-ray sources grows globally, XMPI is a
promising technique for current and future X-ray imaging instruments.

##  1 Introduction

Time-resolved X-ray imaging has been a fundamental technique for non-
destructive exploration of phenomena happening in samples opaque to visible
light. The speed and performance of 2D X-ray imaging (radiography) have
constantly increased during the last century, with improving cameras and X-ray
sources. State-of-the-art radiography can now record with MHz frame rate
speeds at high-brilliance sources such as storage rings [1, 2] and X-ray free-
electron lasers [3]. On the other hand, time-resolved 3D imaging, also known
as tomoscopy, has only recently reached 1 kHz at storage rings[4]. Indeed, the
maximum acquisition speed for tomography is limited by the need to rotate the
sample, so it induces shear forces that can alter the studied processes. For
this reason, it is today still very difficult to capture 3D phenomena in
samples opaque to visible light with under-millisecond time resolution. The
investigation of many fast phenomena would benefit from this capability,
including samples of scientific and industrial interest such as fractures in
solids [5, 6], shock wave propagation [7, 8], fast biological processes [9,
10] and fluid dynamic studies [11]. To overcome this problem, there have been
developments in the field of X-ray Multi-Projection Imaging (XMPI), which
allow for rotation-free 3D imaging [12, 13, 14, 15, 16]. XMPI divides the
X-ray beam into multiple beamlets to enable imaging of a sample from various
angles. The recorded sparse dataset of images is then fed into advanced
reconstruction algorithms to obtain a full 3D reconstructed image[17]. Thus,
the XMPI method enables the study of dynamics without introducing shear forces
that may alter the studied processes [18, 4]. In this paper, we demonstrate
static and time-resolved XMPI with a frame rate of up to 3 kHz without
focusing the beam of ID19 at the European Synchrotron Radiation Facility
(ESRF). XMPI was then used for the study of fast stochastic processes in
aluminum foam samples. Such samples have been used before for 1 kHz frame-rate
tomographic acquisition at synchrotron facilities [4]. Several of the
processes of interest that take place in the foam are highly sensitive to the
g-forces generated by the fast rotations needed to probe the relevant
dynamics, which motivates the use of XMPI for these studies over tomography at
higher frame rates. Recently, there have been pioneering attempts to employ
XMPI with white beam using different splitting configurations at storage rings
[19, 20]. Furthermore, a 1.1 MHz sampling method of XMPI was also demonstrated
with a two-beamlet configuration [21]. Here we present the first use of XMPI
with a pink beam for application in diffraction-limited storage rings. The
narrow energy bandwidth results in images carrying quantitative information on
the sample, as well as sharper images compared to a white beam produced by a
bending magnet or wiggler.

Figure 1:  Illustration of the XMPI concept [16], as implemented at ID-19 of
ESRF. The main X-ray beam enters the experimental hall from the left-hand
side, propagating to the right of the figure. Three beam splitters are
oriented in Bragg or Laue configuration (details in the main text) to meet the
Bragg diffraction condition and deflect a part of the main beam. The beam
splitters are positioned so that the deflected beams intersect at one point,
where the sample is placed. The detectors are placed behind the sample, each
aligned to one of the deflected beams, to image the sample simultaneously from
different directions.

##  2 Methods

The XMPI method relies on crystal diffraction for the splitting of a primary
X-ray beam into secondary beams in order to enable simultaneous imaging of a
sample from angularly spaced viewpoints. The configuration of the XMPI setup
is flexible and can be optimized to different facilities. The ID19 ESRF
beamline was chosen for this demonstration of XMPI because its undulator
source in a diffraction-limited storage ring and grants one of the highest
spectral densities available for synchrotron sources, 2-3 orders of magnitude
higher than beamlines based on bending magnets or wiggler sources [22]. In
general, access to a high-brilliance source is crucial for fast imaging due to
the small exposure time for each frame. In the case of XMPI, high spectral
density is important because the beam is divided spectrally to produce
multiple projections. The concepts of the presented implementation of XMPI are
introduced in 2.1. The required methods for the optimization of the setup at
the conditions of ID19 are motivated and described in 2.2, 2.3 and 2.4.

###  2.1 XMPI setup

The experimental setup is composed of three main parts; the multi-projection
generation, the sample environment, and the detector system. A schematic view
of the setup is depicted in Fig. 1. The multi-projection generation is
achieved with the use of crystals, which are referred to as “beam splitters”.
Each beam splitter is mounted on motorized stage assemblies with six degrees
of freedom (three linear translations and three angles), which allows the
alignment of the setup. During alignment, each crystal is positioned in the
beam path and oriented at an angle $\theta$ with respect to the central beam
to fulfill its respective Bragg or Laue condition for a specific part of the
spectrum. When the central beam interacts with an aligned beam splitter, part
of the beam is diffracted, creating a secondary beam that travels at a
$2\theta$ angle with respect to the central beam and is referred to as a
“beamlet”. The portion of the beam that is neither diffracted nor absorbed by
a given beam splitter is transmitted. The arrangement of the beam splitters is
such that the beamlets intersect at a common point, offering an equal number
of projections on the sample. The setup for this work relied on three beam
splitters (yet this number will increase in future iterations of the
experiment). The exact configuration was finalized after the beam
characterization (see Sec. 2.2), as it is further discussed in Sec. 2.3. The
sample environment is placed at the beamlet intersection point, allowing its
simultaneous imaging from the three generated viewpoints. Because of its
design, the XMPI method can accommodate a wide variety of sample environments.
The imaging projections are detected with three indirect detectors, each
aligned perpendicular to a beamlet’s projection direction. All indirect
detectors were identical and were composed of (i) an efficient scintillator
($250~{}\mu$m thick GAGG+), (ii) a $90^{\circ}$ mirror which redirects the
visible light to a motorized microscope configuration, (iii) a magnification
system consisting of a visible microscope, which also focuses the light on the
camera sensor, and (iv) fast cameras optimized for kHz acquisition speed
(Photron Nova S16 models). Each microscope used the same configuration of
optics, a 5X Mitutoyo Plan Apo HR Infinity Corrected Objective, and a 1x MT-L
tube lens. The described configuration provided a field-of-view of $4\times 4$
mm2, with an effective pixel size of 4 $\mu$m.

###  2.2 Setup alignment

XMPI is based on crystal diffraction and its configuration is inherently
linked to the spectrum and the central energy of the beamline, as it
determines the arrangement of the setup. The central energy of ID19 U17.6
undulator at the minimum gap of $11.8$ mm ($\lambda_{u}=18$ mm, N=90) was
expected to be approximately $17.6$ keV on-axis with 1.6% bandwidth (FWHM).
These values were obtained from beamline simulations with XOP utilizing
parameters of the EBS lattice [23, 24]. The beam was filtered only with
mandatory optical elements for heat load moderation (0.8 mm thick diamond
window and a series of thin Be windows $\approx$2 mm), cutting out the soft
part of the emitted X-ray spectrum ($<$ 10 keV) along the 145 mm vacuum flight
tube. We implemented a spectrometer to measure the exact beam spectrum and
spectrally align the crystals accordingly, i.e. set the crystals in
diffraction conditions to the high flux components of the spectrum. The
spectrometer was placed as the most downstream element of the setup to act as
a monitoring tool [25]. The spectrometer was composed of a bent C-$333$
crystal, an energy-dispersive element that diffracts different energies with
different angles, coupled to a 1x indirect detector PCO.edge.5.5 camera. The
diffracted photons are therefore detected in different positions (pixels) in
the camera depending on their energy, which allows the monitoring of the beam
spectrum. The measurement of the beam’s central energy was carried out by
first aligning the spectrometer to diffract the most intense part of the beam
spectrum and then using a perfect crystal to gauge the angle where its
diffraction condition was fulfilled. A perfect silicon crystal with a <100>
out-of-plane direction was used for this purpose and was positioned upstream
of the spectrometer. The crystal was rocked continuously in one direction
until it was oriented to satisfy the Laue diffraction from the (220)
diffraction plane. The central beam spectrum was simultaneously being
monitored in the PCO.edge.5.5 camera. While the silicon crystal was far from
satisfying the diffraction condition, the beam spectrum remained unchanged
(aside from uniform intensity losses due to transmission through the silicon
crystal). When the crystal was rocked in the diffraction region, a clear dip
was visible in the camera, indicating the portion of the beam that was being
diffracted by the silicon crystal. The dip moved through the beam spectrum as
the crystal was rocked through the diffraction region, indicating that the
crystal was satisfying the diffraction condition for a different part of the
spectrum. The same procedure was repeated with the rocking of the crystal in
the opposite direction. The two rocking angles that satisfied the diffraction
condition at the most intense part of the spectrum for each of the rocking
directions need to be symmetric and were therefore used to assess accurately
the diffraction angle for the crystal in the beam’s central energy. With this
approach, the central energy of ID19 was estimated to be $17.25$ keV with the
smallest undulator gap, i.e., the highest flux available with 7/8 filling mode
and 200 mA storage ring current.

###  2.3 Beam splitters configuration

The efficiency of the setup is directly affected by the choice of the crystals
that act as the beam splitters and beam parameters (intensity, divergence,
spectral components). We positioned different crystals one by one upstream of
the calibrated spectrometer in order to characterize their spectral response,
i.e., how much of the ID19 beam they diffract. Silicon and Germanium
monocrystals were assessed to be optimal for the pink beam of ID19. The final
selection of beam splitters is summarized in Tab. 1, which will hereafter be
referred to using their abbreviations. The crystals were provided by INFN-
Ferrara.

| Beam splitter #1 | Beam splitter #2 | Beam splitter #3
---|---|---|---
Material | Silicon | Silicon | Germanium
Thickness |  $14$ µm |  $14$ µm |  $1$ mm
Out-of-plane | <211> | <100> | <100>
Diffraction plane | (111) | (220) | (400)
Diffraction | Laue | Laue | Bragg
Diffraction angle | 6.58∘ | 10.79∘ | 14.72∘
Projection angle | 13.16∘ | 21.58∘ | 29.44∘
Abbreviation | Si-111 | Si-220 | Ge-400
Table 1: Configuration of the beam splitters for XMPI at beam energy $17.25$
keV. The splitters’ enumeration (#1-#3) is the order of interaction with the
beam. All the beam splitters are mono-crystals. The listed abbreviations are
used to refer to the beam splitters in the main text.

###  2.4 Heat–load measurements

The final configuration of beam splitters in the experiment included the use
of a germanium crystal (see Tab. 1). Ge-400 crystal was oriented in Bragg and
absorbed the entire beam that was not diffracted or absorbed by the previous
elements (see Fig. 1). Given the high flux absorbed by this crystal, there
were concerns about the thermal dissipation of the absorbed heat load within
the crystal and we monitored its surface temperature via a thermal camera
(FLIR A7600sc). Fig. 2 shows the temporal evolution of the temperature
registered by the FLIR camera at different exposure times controlled by a
shutter (ct) for various time periods. The FLIR camera provides a thermal
image as well, which reveals a spatial distribution of temperatures on the
irradiated and surrounding surfaces. The temperature rise can be estimated
from measured relative intensity $\Delta{C}$ of near-infrared thermal
radiation using the proper value of relative emissivity of germanium
$\varepsilon$. Emissivity is temperature dependent and thus may vary during
the crystal heating. The value of relative emissivity $\varepsilon$ for room
temperature Ge was measured as $\varepsilon$ = 0.26 in [26] and for T = 1209
K, i.e. melting point of Ge, as $\varepsilon$ = 0.55 in [27]. Since the
experiment was performed at room temperature and the change of measured
relative intensity was around 2 % of the whole dynamic range, we conclude that
the value of relative emissivity coefficient $\varepsilon$ = 0.26 reported by
[26] for room temperature fits well the conditions of the present experiment.
The measured increase in the temperature of the Ge crystal (Fig. 2) is of the
order of tens of degrees and well below the melting point of Ge. No distortion
is visible in the images from the Ge-400 projection. This includes distortions
due to thermal strain caused by overheating the crystal. Therefore, we can
conclude that the heat transfer from the crystal surface to the crystal bulk,
the air, and the crystal holder is effective enough to prevent the overheating
of the irradiated region.

Figure 2:  Measured temperature change $\Delta{T}$ versus time $t$ on the last
crystal Ge-400 for different times of opened shutter ct. The error bar is
visualized as a shade. The temperature change is far from the melting point of
germanium T = 1209 K.

##  3 Results and Discussion

An alignment sample (bamboo toothpick) was used to determine and monitor the
intersection of the Si-111, Si-220, and Ge-400 beamlets. The final
configuration allowed recording up to 3 kHz with the three cameras
simultaneously using the whole 12-bit dynamical range of the kHz cameras. This
corresponds to frame acquisitions every $0.3$ ms, allowing the reconstruction
of sub-millisecond dynamics. The microscope configuration has 8 $\mu$m spatial
resolution per frame, estimated using a resolution target. The recorded images
with all three cameras can be seen in Fig. 3. The Si-111 image in Fig. 3 was
rotated by $5.81^{\circ}$ to recover a misalignment error in the camera.

Figure 3:  Simultaneous frame acquisition of an alignment pin at $3000$ fps
speed from three projections, coming respectively from the beam splitters
Si-111, Si-220, Ge-400. The Si-111 and Si-220 beam splitters are in Laue
geometry, while the Ge-400 is in Bragg. The blurring in the Si-111 and Si-220
projections is caused by the formation of the Borrmann triangle combined with
some warping in these thin membrane-like splitters.

The illumination of the alignment sample allowed some observations about the
employed beam splitters. The Ge-400 projection appears intense and clear and
is used as a reference for the capabilities of the setup. However, the Si-111
and Si-220 projection images appear blurry (Fig. 3). After dedicated testing,
it was concluded that the cause of the blur in the Si-111 and Si-220 was the
result of two factors: the Laue diffraction geometry and the presence of
strain in the crystal. The Laue diffraction geometry incurs into blurring
caused by the Borrmann triangle [28], a dynamical diffraction effect. Indeed,
an X-ray beam diffracted over the entire thickness of a crystal results in a
distribution of the diffraction centers over a fan-shaped profile, which
produces a blurry virtual source. Laue diffraction is especially sensitive to
strain in the crystal since strained areas are illuminated differently under
the profile of the fanned diffracted beam. An example of this can be seen at
the top of the Si-220 projection image, where a halo appears. The measurements
were carried out with a low-strain area of the crystal to mitigate these
effects. The blurring induced by the Borrmann triangle does not happen for
symmetric Bragg diffraction geometry, as visible in the Ge-400 image, since
the thickness of crystal traversed by the X-rays before diffraction is
extremely reduced. Therefore, we advise using symmetric Bragg diffraction
geometry when possible for future iterations of XMPI experiments.

In the Si-220 projection, the significant strain produces a curvature in the
crystal shape that results in warping of the beam trajectories, stretching the
recorded projections along this beam. Because of this stretching and the
aforementioned Borrman-triangle effect, the Si-220 figure was re-scaled over
the horizontal direction by a factor of 1.17 to match the real aspect ratio of
the object. The strain was most likely introduced during fabrication or by the
clamping method. Both causes are technological in nature and they can be
improved upon with further engineering of the crystals and their holders.
Regarding the Si-111 splitter, we observe strong vibrations as a result of the
thickness and mounting of this membrane. In future iterations, we will
redesign the mounting and use a thicker crystal to avoid such vibrations.

To demonstrate the capabilities of XMPI at ID19, we explore the foaming of an
aluminum alloy produced by a fast and self-triggering pressure release method.
The complete foaming and solidification process of the alloy takes place in
less than one second and has never been studied before at such speeds. Rapid
metal foaming and solidification not only have the advantage of short
production time but also of rapid development, as the foam structure has much
less time to coarsen. Time is one of the most important parameters, along with
temperature and alloy composition, leading to an increased number of
coalescence events [29]. Long foaming times lead to an increased number of
coalescence events and, thus, to foam aging.

This process demonstrated dynamics at the order of 0.5-1 kHz, so we optimized
our acquisition system to that range. Although similar samples have been
observed in 3D at kHz frame rate during a standard metal foaming process, it
was also observed that in some cases, the dynamics of low-stabilized liquid
metal foams such as the alloy studied in this work were altered in the later
foaming stage by the radial forces generated by the rotation [4]. Thus, XMPI
is a promising technique for their study without sample rotation and,
therefore, without altering the dynamics of interest. The studied sample was
produced from a foamable thixocast precursor made of an AlSi6Cu4 alloy with an
added 0.8 wt% TiH2 as a blowing agent. Thixocast foamable precursors are known
for their high volume expansion and capability to form round bubbles but also
for their tendency for bubble coalescence [30]. By observing the evolution of
such liquid metal foams in 3D, we gain insight into fast phenomena during the
foaming process, such as the evolution of film thickness, i.e. the distance
between bubbles, over time, the minimum film thickness, gas interdiffusion
between different bubbles, or the rupture and coalescence of bubbles orders of
magnitude faster than normal foam expansion.

The measurements were carried out with two out of the three beam splitters
(Si-220 and Ge-400) of the final XMPI configuration since the illumination
from the Si-111 beam splitter was experiencing intense drifting, making the
recording unusable. During the measurements, the expansion of the foams occurs
rapidly due to the fast intrinsic pressure release, making it non-trivial to
determine the correct time to trigger the acquisition. The detector initially
assigned for the Si-111 crystal was repurposed to provide a visual trigger to
begin the acquisition process. A laser was set to illuminate the sample so
that the visible light image captured on the camera had the same field-of-view
as the X-ray images provided by the other two cameras, i.e. it was a close
representation of the expected X-ray images. Therefore, the view from this
camera was used to manually trigger the acquisition of the two other cameras
when the expanding foaming was visible in the field of view.

The 3D structure of metallic foams was reconstructed using a deep learning 3D
reconstruction approach. We applied ONIX [17], a self-supervised learning
method developed to address the ill-defined problem posed by XMPI. A
continuous function [31, 32] was learned that maps from the spatial-temporal
coordinates to the refractive index of the sample, depicting the structure of
the metallic foams. Given the limited volumetric data available from the two
projections, we adopted adversarial learning [33] to enforce the self-
consistency between the collected projections and the predictions generated
from arbitrary viewpoints. An additional discriminator as the one used in [34]
was added to the ONIX architecture, and the networks were trained using the
adversarial loss function[33, 35]. We selected a region of interest with the
size of 128$\times$128 on both projections for the 3D reconstruction, as shown
in Fig. 4. The selected areas were denoised using total variation filters and
segmented from the background before feeding into the neural networks. The
network training was performed on an NVIDIA A100 GPU with 40 GB of RAM. We
used Adam optimizer [36] with a learning rate of 10-4 and batch size of eight.
After the training, the volumetric information was obtained by extracting 1283
grids from the networks, with the voxel size equivalent to the effective pixel
size of 4 $\mu$m. The resulting 3D reconstructions were visualized using
Blender. A selection of frames from the two projections can be seen in Fig. 4.
In the shown projections, slight bubble movements are visible and the rupture
of a surface bubble can be observed, which results in a dark corrugated
structure at its former position and will be discussed later in more detail.
More accurate results are visible in the reconstructed 3D volume.

Figure 4:  A selection of frames was collected with two projections of the
XMPI method during the foaming of an AlSi6Cu4 thixocast precursor. The red
square marks the region of interest for the reconstruction, while the first
(T=$0.000$s) and last (T=$0.049$s) timestamps mark the time period that was
used for the reconstruction. In the interim consecutive frames (T=$0.035$s,
T=$0.036$s and T=$0.037$s), behaviors in the bubble movement can be monitored.
The last row shows renderings of 3D reconstructions of the selected bubbles.
At t = 36 ms, the lower left bubble burst with no significant effect on the
rest of the bubbles’ position, volume, or rearrangements in the neighborhood.

When the distance between two adjacent bubbles, in this case of relatively
equal size, falls below a certain critical level, the film rupture leads to
the coalescence of the bubbles, resulting in a new round bubble in a time
frame of 1-2 ms, as shown in Fig. 5. The volume of this new bubble is
obviously equal to the sum of the volumes of the previous bubbles. As the
bubbles involved are relatively small compared to others and to the foam size,
no significant bubbles or foam rearrangements are observed in the surrounding
area in this case. Similar behavior has been observed in the past with other
alloys with X-ray radioscopy [37]. The kinetics of coalescence are random in
nature but influenced by various parameters such as gravity or temperature
[38]. To gain a more detailed insight into this coalescence process, even
faster acquisition speeds will be required. This conclusion is consistent with
the X-ray tomoscopy results for a more stable liquid metal foam alloy [39].

Figure 5:  Bubbles merging in the melted aluminum foam sample. The selection
of frames shows two similarly sized bubbles merging into one larger one,
details marked by the red square. This process takes a maximum of 1-2 frames,
i.e. of 1-2 ms.

If the bubbles involved in a coalescence process are large relative to the
foam size, as shown in Fig. 6, the film rupture process leading to coalescence
of the bubbles can be accompanied by a coalescence avalanche [40] and a
vigorous structural rearrangement of the entire foam, which directly affects
the development of foam expansion. The film rupture is again faster than 1 ms
and the coalescence of the two bubbles takes place in a few milliseconds.

Figure 6:  Selection of frames showing bubble coalescence and rupture in the
melted aluminum foam sample. The red square marks the region of interest where
a film between two relatively large bubbles ruptures, leading to coalescence
of the two bubbles involved, and also to an avalanche of further bubble
coalescence and structural rearrangements.  Figure 7:  A selection of
magnified frames showing the rupture of a film at the foam surface leading to
the burst of a bubble and the generation of a surface spike in the course of
fast solidification.

A new phenomenon, namely the formation of spikes on the foam surface, was
observed and is shown in a sequence in Fig. 7. When a bubble bursts near the
surface, a film rupture occurs at the outer foam skin, with molten material
spurting out in less than a millisecond. The liquid appears to be held in a
kind of spike-like surface protrusion by a solid oxide skin and solidifies in
1-2 ms. The formation of such spikes could only be observed in cases where
foam growth and subsequent foam solidification are extremely rapid, as is the
case here.

##  4 Conclusions

This experiment constitutes the first implementation of XMPI at a diffraction-
limited storage ring using pink beam, i.e., the full-harmonic provided by an
insertion device. The XMPI technique, coupled with the high spectral density
and full-harmonic beam of the ID19 beamline, allowed to successfully image
sub-millisecond dynamics (3000 frames per second) with spatial resolution of
$8$ µm per projection while filling the full dynamic range of the detectors.
The narrow energy bandwidth of this implementation enables the images to carry
quantitative information on the sample, as well as produce sharper images
compared to a white beam. Thus, the results here presented are a fundamental
step to establish XPMI as a technique capable of probing 4D (3D+time)
phenomena in samples opaque to visible light with kHz frame-rate speed and
micrometer resolution that are sensitive to shear forces.

XMPI was employed for the study of the dynamics of the aluminum foaming
process at 1 kHz, avoiding shear forces that alter the studied dynamics as
introduced by time-resolved tomography in such systems. From the acquired
dataset, 3D movies of bubbles moving in the melted aluminum were obtained via
machine learning algorithms. The absence of shear forces was crucial to enable
the observation of a novel phenomenon in aluminum foaming, namely the
formation of spikes on the foam surface. These spikes are formed by the
bursting of small bubbles near the sample surface and solidify quickly due to
the colder ambient temperature and the formation of an oxide layer.
Furthermore, the rupture of small and large films separating two bubbles and
leading to their coalescence could be investigated for the first time at 1 kHz
in a liquid AlSi6Cu4 alloy foam. In agreement with other results in the
literature, film rupture times of less than 1 ms could be detected, indicating
the need for even faster acquisition speed.

As a result of this implementation, we are designing a beam-splitting scheme
tailored to the beam characteristics of ID19 that will allow additional
projections with increased angular spacing between each projection to benefit
the 3D reconstruction process. Furthermore, this future implementation of the
XMPI instrument will increase the spatiotemporal resolution by enhancing the
efficiency and image quality of the crystal splitters by using symmetric Bragg
diffraction and improving the mounting and crystal manufacture.

##  5 Backmatter

Funding This work was performed within the following projects: RÅC (Röntgen-
Ångström cluster) “INVISION” project, 2019-2023; EuXFEL R&D “MHz microscopy at
EuXFEL: From demonstration to method”, 2020 - 2022; ERC-2020-STG, 3DX-FLASH,
Grant agreement 948426; Horizon Europe EIC Pathfinder “MHz-Tomoscopy” project,
2022-2025. The Czech co-authors‘ thank the Czech Ministry of Education, Youth,
and Sports (CMEYS) for partial financial support of the study and utilization
of measurements and control of radiation-induced thermomechanical phenomena at
large-scale facilities, as a part of the grant nr. LM2023068. The Slovak co-
authors‘ thank the grant titled High-performance curved X-ray optics prepared
by advanced nanomachining technology from Scientific Grant Agency of the
Ministry of Education, science, research and sport of the Slovak Republic and
the Slovak Academy of Sciences, no.: VEGA - 2/0041/21.

Acknowledgments We thank the team of ESRF ID19 for the help during beamtime
MI1449.

Disclosures The authors declare no conflicts of interest.

Data availability Data underlying the results presented in this paper are not
publicly available at this time but may be obtained from the authors upon
reasonable request.

## References

  * [1] K. Fezzaa and Y. Wang, “Ultrafast x-ray phase-contrast imaging of the initial coalescence phase of two water droplets,” Phys. Rev. Lett. 100, 104501 (2008).
  * [2] M. P. Olbinado, X. Just, J.-L. Gelet, P. Lhuissier, M. Scheel, P. Vagovic, T. Sato, R. Graceffa, J. Schulz, A. Mancuso, J. Morse, and A. Rack, “Mhz frame rate hard x-ray phase-contrast imaging using synchrotron radiation,” Optics Express 25, 13857–13871 (2017).
  * [3] P. Vagovič, T. Sato, L. Mikeš, G. Mills, R. Graceffa, F. Mattsson, P. Villanueva-Perez, A. Ershov, T. Faragó, J. Uličný, H. Kirkwood, R. Letrun, R. Mokso, M.-C. Zdora, M. P. Olbinado, A. Rack, T. Baumbach, J. Schulz, A. Meents, H. N. Chapman, and A. P. Mancuso, “Megahertz x-ray microscopy at x-ray free-electron laser and synchrotron sources,” Optica 6, 1106–1109 (2019).
  * [4] F. García-Moreno, P. H. Kamm, T. R. Neu, F. Bülk, M. A. Noack, M. Wegener, N. von der Eltz, C. M. Schlepütz, M. Stampanoni, and J. Banhart, “Tomoscopy: Time-resolved tomography for dynamic processes in materials,” Advanced Materials 33, 2104659 (2021).
  * [5] A. Kumar, K. Kashyap, M. T. Hou, and J. A. Yeh, “Strength improvement of glass substrates by using surface nanostructures,” Nanoscale Research Letters 11 (2016).
  * [6] Z. Xu, X. He, Y. Han, and F. Huang, “A different viewpoint on mechanism of fracture to shear-banding failure mode transition,” Journal of the Mechanics and Physics of Solids 145, 104165 (2020).
  * [7] Y. B. S. R. PRASAD, S. P. Barnwal, P. A. Naik, Y. Yadav, R. K. Patidar, M. P. Kamath, A. Upadhyay, S. Bagchi, A. Kumar, A. S. Joshi, and P. D. Gupta, “Shock wave propagation in soda lime glass using optical shadowgraphy,” Pramana 87, 1–8 (2016).
  * [8] D. E. Grady, “Shock-wave compression of brittle solids,” Mechanics of Materials 29, 181–203 (1998).
  * [9] J. N. Hansen, A. Gong, D. Wachten, R. Pascal, A. Turpin, J. F. Jikeli, U. B. Kaupp, and L. Alvarez, “Multifocal imaging for precise, label-free tracking of fast biological processes in 3D,” Nat Commun 12, 4574 (2021).
  * [10] T. V. Truong, D. B. Holland, S. Madaan, A. Andreev, K. Keomanee-Dizon, J. V. Troll, D. E. S. Koo, M. J. McFall-Ngai, and S. E. Fraser, “High-contrast, synchronous volumetric imaging with selective volume illumination microscopy,” Commun. Biol. 3, 74 (2020).
  * [11] A. Aliseda and T. J. Heindel, “X-ray flow visualization in multiphase flows,” Annual Review of Fluid Mechanics 53, 543–567 (2021).
  * [12] P. Villanueva-Perez, B. Pedrini, R. Mokso, P. Vagovic, V. A. Guzenko, S. J. Leake, P. R. Willmott, P. Oberta, C. David, H. N. Chapman, and M. Stampanoni, “Hard x-ray multi-projection imaging for single-shot approaches,” Optica 5, 1521–1524 (2018).
  * [13] J. Duarte, R. Cassin, J. Huijts, B. Iwan, F. Fortuna, L. Delbecq, H. Chapman, M. Fajardo, M. Kovacev, W. Boutu, and H. Merdji, “Computed stereo lensless X-ray imaging,” Nature photonics 13, 449 – 453 (2019). © Springer Nature Publishing AG.
  * [14] W. Voegeli, K. Kajiwara, H. Kudo, T. Shirasawa, X. Liang, and W. Yashiro, “Multibeam x-ray optical system for high-speed tomography,” Optica 7, 514–517 (2020).
  * [15] V. Bellucci, M. Zdora, L. Mikeš, S. Birnšteinová, P. Oberta, M. Romagnoni, A. Mazzolari, P. Villanueva-Perez, R. Mokso, C. David, M. Makita, S. Cipiccia, J. Uličný, A. Meents, A. Mancuso, H. Chapman, and P. Vagovič, “Hard x-ray stereographic microscopy for single-shot differential phase imaging,” Opt. Express 31, 18399–18406 (2023).
  * [16] P. Vagovic, V. Bellucci, P. Villanueva-Perez, and W. Yashiro, “Multi beam splitting and redirecting apparatus for a tomoscopic inspection apparatus, tomoscopic inspection apparatus and method for creating a three dimensional tomoscopic image of a sample,” (Patent pending, Application number: EP21200564.9).
  * [17] Y. Zhang, Z. Yao, T. Ritschel, and P. Villanueva-Perez, “Onix: An x-ray deep-learning tool for 3d reconstructions from sparse views,” Applied Research 2, e202300016 (2023).
  * [18] B. Jachimska and K. Małysa, “Foam rupture under combined action of gravity and centrifugal force,” Bulletin of the Polish Academy of Sciences: Chemistry 48, 166–179 (2000).
  * [19] X. Liang, W. Voegeli, H. Kudo, E. Arakawa, T. Shirasawa, K. Kajiwara, T. Abukawa, and W. Yashiro, “Sub-millisecond 4d x-ray tomography achieved with a multibeam x-ray imaging system,” Applied Physics Express 16, 072001 (2023).
  * [20] W. Voegeli, X. Liang, T. Shirasawa, E. Arakawa, K. Hyodo, H. Kudo, and W. Yashiro, “Multi-beam x-ray optical system for high-speed tomography using a $\sigma$-polarization diffraction geometry,” Applied Physics Express 16, 072007 (2023).
  * [21] P. Villanueva-Perez, V. Bellucci, Y. Zhang, S. Birnsteinova, R. Graceffa, L. Adriano, E. M. Asimakopoulou, I. Petrov, Z. Yao, M. Romagnoni _et al._ , “Megahertz x-ray multi-projection imaging,” arXiv preprint arXiv:2305.11920 (2023).
  * [22] T. Weitkamp, P. Tafforeau, E. Boller, P. Cloetens, J. Valade, P. Bernard, F. Peyrin, W. Ludwig, L. Helfen, and J. Baruchel, “Status and evolution of the ESRF beamline ID19,” AIP Conference Proceedings 1221, 33–38 (2010).
  * [23] M. S. del Río and R. J. Dejus, “XOP v2.4: recent developments of the x-ray optics software toolkit,” in _Advances in Computational Methods for X-Ray Optics II,_ vol. 8141 M. S. del Rio and O. Chubar, eds., International Society for Optics and Photonics (SPIE, 2011), p. 814115.
  * [24] P. Raimondi, C. Benabderrahmane, P. Berkvens, J. C. Biasci, P. Borowiec, J.-F. Bouteille, T. Brochard, N. B. Brookes, N. Carmignani, L. R. Carver, J.-M. Chaize, J. Chavanne, S. Checchia, Y. Chushkin, F. Cianciosi, M. Di Michiel, R. Dimper, A. D’Elia, D. Einfeld, F. Ewald, L. Farvacque, L. Goirand, L. Hardy, J. Jacob, L. Jolly, M. Krisch, G. Le Bec, I. Leconte, S. M. Liuzzo, C. Maccarrone, T. Marchial, D. Martin, M. Mezouar, C. Nevo, T. Perron, E. Plouviez, H. Reichert, P. Renaud, J.-L. Revol, B. Roche, K.-B. Scheidt, V. Serriere, F. Sette, J. Susini, L. Torino, R. Versteegen, S. White, and F. Zontone, “The extremely brilliant source storage ring of the european synchrotron radiation facility,” Communications Physics 6, 82 (2023).
  * [25] V. M. Kaganer, I. Petrov, and L. Samoylova, “Resolution of a bent-crystal spectrometer for X-ray free-electron laser pulses: diamond versus silicon,” Acta Crystallographica Section A 77, 268–276 (2021).
  * [26] H. Madura, H. Polakowski, and B. Wiecek, “Spectral emissivity evaluated for materials used in microelectronics,” Quantitative Infrared Thermography QIRT 96 – Eurotherm Series pp. 52–57 (1997).
  * [27] F. G. Allen, “Emissivity at 0.65 Micron of Silicon and Germanium at High Temperatures,” Journal of Applied Physics 28, 1510–1511 (2004).
  * [28] A. Authier, _Dynamical Theory of X-Ray Diffraction_ (International Union of Crystallography, 2001).
  * [29] F. García-Moreno, M. Jürgens, and J. Banhart, “Temperature dependence of film rupture and internal structural stability in liquid aluminium alloy foams,” Acta Materialia 196, 325–337 (2020).
  * [30] J. Weise, H. Stanzick, and J. Banhart, “Semi-solid processing of complex-shaped foamable material,” (2003).
  * [31] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis,” in _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics),_ vol. 12346 LNCS (Springer Science and Business Media Deutschland GmbH, 2020), pp. 405–421.
  * [32] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, “pixelnerf: Neural radiance fields from one or few images,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,_ (2021), pp. 4578–4587.
  * [33] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” Communications of the ACM 63, 139–144 (2020).
  * [34] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” in _Proc. Computer Vision and Pattern Recognition,_ (2017), pp. 4681–90.
  * [35] K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger, “Graf: Generative radiance fields for 3d-aware image synthesis,” Advances in Neural Information Processing Systems 33, 20154–20166 (2020).
  * [36] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014).
  * [37] F. Garcia-Moreno, M. Mukherjee, C. Jiménez, A. Rack, and J. Banhart, “Metal foaming investigated by x-ray radioscopy,” Metals 2, 10–21 (2012).
  * [38] F. Garcia-Moreno, E. Solórzano, and J. Banhart, “Kinetics of coalescence in liquid aluminium foams,” Soft Matter 7, 9216–9223 (2011).
  * [39] F. García-Moreno, P. H. Kamm, T. R. Neu, F. Bülk, R. Mokso, C. M. Schlepütz, M. Stampanoni, and J. Banhart, “Using x-ray tomoscopy to explore the dynamics of foaming metal,” Nature Communications 10, 3762 (2019).
  * [40] M. Paeplow, F. García-Moreno, A. J. Meagher, A. Rack, and J. Banhart, “Coalescence avalanches in liquid aluminum foams,” Metals 7 (2017).