# A Somewhat Robust Image Watermark against Diffusion-based Editing Models

Mingtian Tan  University of Virginia  Tianhao Wang  University of Virginia
Somesh Jha  University of Wisconsin Madison

###### Abstract

Recently, diffusion models (DMs) have become the state-of-the-art method for
image synthesis. Editing models based on DMs, known for their high fidelity
and precision, have inadvertently introduced new challenges related to image
copyright infringement and malicious editing. Our work is the first to
formalize and address this issue. After assessing and attempting to enhance
traditional image watermarking techniques, we recognize their limitations in
this emerging context. In response, we develop a novel technique,
$\mathsf{RIW}$ (Robust Invisible Watermarking), to embed invisible watermarks
leveraging adversarial example techniques. Our technique ensures a high
extraction accuracy of $96\%$ for the invisible watermark after editing,
compared to the $0\%$ offered by conventional methods. We provide access to
our code111https://github.com/BennyTMT/RIW.

##  1 Introduction

Recent years have seen remarkable contributions to computer vision through
advancements in image generation, largely propelled by the advent of diffusion
models (DMs)[51]. This progress is due to their robust theoretical
foundations[17, 54], enhanced latent space inference [41], and efficient image
sampling techniques [52]. These factors have led DMs to commercial success, as
seen in DALL-E [39] and Stable Diffusion [41]. Additionally, DMs have spurred
various emerging applications, highlighting their potential expansion in
fields like data synthesis [42], video synthesis [50, 6, 58], text-to-3D
synthesis [37], and image editing [7, 25, 35, 34, 4, 55].

Alongside the commercial success and potential applications of diffusion
models, it also brings intellectual property (IP) concerns to the forefront
[47]. Especially, the advancement of DM-based editing models [7, 25, 40, 35],
due to their hyper-realistic editing effects and fundamental differences from
mere image generation tasks, still presents unresolved issues of unauthorized
image use and deceptive edits.

Strategies to address these issues have been explored. Liang et al., for
example, have implemented adversarial training [33, 14] to curb unauthorized
style transfer manipulation by DMs [28]. Similarly, Shan et al. introduce
adversarial noise into artworks training data to hinder DMs from learning and
replicating unique artistic styles [47]. Several methods embed a custom
watermark in training set [65, 30] or during inference [10, 36], embedding an
identifier in the produced images. Notably, the recent work Tree-Ring [57] has
achieved promising results by embedding watermarks [9, 48] into the initial
noise matrix of the diffusion process, thereby watermarking the model
effectively.

In this paper, we investigate image watermark schemes that assert image
ownership post-editing. In this field, traditional methods, such as digital
[9, 48] or AutoEncoder [59, 63] based watermarks, are insufficient, especially
with the advancement of DM-based image-editing models [22], because the
changes introduced by modern image-editing models are unpredictable and
prevalent, totally destroying the watermarks.

Our approach is to frame this as an optimization problem. Taking any target
image (e.g., an artwork) and a watermark (e.g., a random image), we embed them
together so that the resulting image will maintain the semantic meaning of
both images in the latent space of the editing model. The underlying idea is
to leverage the ‘mixup’ effect [62] where multiple images can co-exist in an
image (i.e., the new image is a weighted average of multiple images), and the
image-editing model can operate differently on different semantics, i.e.,
change the target image while leaving the watermark image untouched. However,
we cannot directly overlay the watermark image onto the target image, because
this will make the modifications detectable. Our optimization procedure will
ensure that the modifications are small.

Given the watermarked image, the next question is how to extract the watermark
after it has been edited. Similar to when we embed the watermark, the
watermark in the edited image should also be unnoticeable. Extracting
watermarks while keeping them unnoticeable is challenging. To enhance the
extraction efficacy, we propose to boost the extraction signal by embedding
multiple invisible watermarks at different positions within the image, and
aggregating extracted signals from those positions.

We have to limit the scope of our solution. First, we cannot defend against
arbitrary edits and only focus on minor edits (e.g., changing style, as shown
in Figure 1). We argue that even non-technically, it is hard to claim IP if
the image is changed significantly. Second, our method needs some knowledge
about the diffusion model, but does not need access to the exact edit model or
edit prompt.

Figure 1: For samples on editing model [7], they vary based on the text prompt
where modifications are required. For instance, “Paris”, “Hong Kong”, and
“Manhattan”.

Through an evaluation on two real-world datasets, the extended LAION-
Aesthetics and TEDBench, as well as two state-of-the-art editing models [7,
25] from different paradigms, we discovered that, compared to prior approaches
[59, 63, 9, 48], our watermarking technique demonstrates superior resistance
against distortions introduced by editing models. For instance, post-editing
by DMs, $96.0\%$ of our invisible watermarks can still be extracted, while
traditional methods yield a $0\%$ success rate.

Our contributions are summarized as follows,

  * •

We are the first to address the issues of image copyright and malicious
modifications within editing models. Additionally, we formalize this problem
and define our game framework.

  * •

After assessing existing watermarking methods, we found them vulnerable to DM-
based editing models. Despite our efforts using adversarial training and fine-
tuning, current techniques remain inadequately prepared to tackle this new
challenge.

  * •

We propose an invisible watermarking technique, named $\mathsf{RIW}$ (Robust
Invisible Watermarking), based on visual-semantic effects. This ensures that
our watermark remains intact even after editing, addressing the image
copyright issues within the editing model framework.

The rest of this paper is structured as follows: In Section 2, we review and
discuss related works. In Section 3, we discuss the current editing models and
the image copyright issues they introduce. We also define our threat model and
discuss existing solutions. In Section 4, we refine existing techniques and
introduce our approach. Section 5 provides detailed experimental procedures.
We conclude in Section 6.

##  2 Related Work

The application of semantic edits to authentic photographs or images has
attracted considerable interest, particularly with the advent of advanced
image synthesis techniques [13, 51, 17]. Effective image editing demands that
the model excel in image generation and interpret text-based editing
instructions. Numerous studies have leveraged Generative Adversarial Networks
(GANs) [13] for diverse image manipulations, such as domain translation
between images [20, 21, 29] or transfer the image to a specific style [1, 23,
2]. Meanwhile, with the rise of Diffusion Models (DMs) [51], we have seen even
better performance in image synthesis [11, 17, 18, 44, 53] as well as other
multimodal tasks such as video [50, 6, 58] and audio [26] generation.

With the advancement of DMs, concerns about image IP (Intellectual Property)
have risen [51, 17, 53, 56, 32]. The realistic generative capabilities of DMs
render it challenging for humans to distinguish whether images have been
altered [7, 25, 40, 35, 34]. Consequently, this not only poses a threat to
artwork IP but also facilitates the potential spread of misinformation through
malicious image modifications. To tackle this challenge, we categorize
existing solutions into two approaches: (1) safeguarding artwork samples and
(2) mitigating the misuse of diffusion models (DMs).

Safeguarding Artworks. To shield artworks from style mimicry through diffusion
Text-to-Image models [41, 43], Shan et al. introduce GLAZE [47]. This method
infuses noise into the original artwork, preserving its quality. As a result,
DMs like Stable Diffusion [41] and DALLE.E-m [40] cannot discern and adopt the
unique art styles of these creators. Complementing this, they conducted
studies involving over $1,000$ artists, highlighting prevailing concerns about
IP in image generation, and underscoring the real-world significance of our
endeavors. Concurrently, Liang et al. [28] launched the MIST project [27].
Utilizing adversarial training, to incorporate noise into artworks. When these
altered images are shared online, adversaries will be unable to obtain a high-
quality image after the style transfer.

Mitigating Misuse of DMs. Zhao et al. [65] propose to introduce watermarks
into training samples, i.e., artworks, utilizing an AutoEncoder model [59].
Once trained on these watermarked samples, DMs consequently produce images
bearing imperceptible watermarks. These watermarks can be subsequently
retrieved through the decoder, establishing the ownership of public DMs.
Similarly, Liu et al. employ a novel technique to watermark the latent DM
[30], training their model on paired data of watermarked prompts and
watermarking images. Consequently, the prompt serves as a trigger to generate
their watermarked image, verifying model ownership. Contrarily, certain
research takes the route of introducing watermark directly into the diffusion
process [36, 57, 10], successfully extracting the watermark and leveraging the
determinism inherent to the DDIM [52] generation mechanism.

##  3 Preliminaries

In this paper, we address the abovementioned issue (protecting image IP in
editing models) from a new angle using image watermarking schemes.
Specifically, we aim to add watermarks to images so that after editing, we can
detect that these edited images are from those images. If adversaries
malevolently alter artwork styles, infringe copyrights, or manipulate the
semantic content of original images using image editing models, these
watermarks can help claim ownership.

In what follows, we begin by reviewing image editing models, and then formally
describe our threat model, and review existing image watermarking schemes.

###  3.1 Image Editing

We focus on diffusion-based image editing models [34, 4, 39, 42], which show
remarkable results due to the outstanding performance of Denoising Diffusion
Probabilistic Models (DDPM) [17] (details deferred to Appendix A.1). Contrary
to traditional GAN-based editing models such as StyleGAN [24], which are
limited to singular editing functionalities (e.g., a model solely dedicated to
converting facial images into smiling faces), recent DMs-based image editing
models enable zero-shot editing of images merely with text instructions and
achieve a superior editing quality [7, 25].

Specifically, these models typically employ an encoder ${E}$ to map images
(denoted by ${x}$) into a latent space, while simultaneously mapping text edit
prompts (denoted by $p$) into an embedding, often using tools like CLIP [38],
representing by $C$, due to its proficient textual understanding capabilities.
Subsequently, a decoder ${D}$ is used to reconstruct the high-resolution
image, fusing the image and text information:

$\displaystyle{x}_{e}={D}\left[{E}({x}),C(p)\right]$ (1)

or more abstractly, ${x}_{e}=\mathtt{Edit}({x},p).$

###  3.2 Problem Statement

We assume there is Alice who has access to an image-editing model, denoted by
$\mathtt{Edit}$, and uses it to edit images, with text prompt $p$. We use a
subscript $e$ to denote edition: an image ${x}$ is edited into
${{x}}_{e}\leftarrow\mathtt{Edit}({x},p)$. It is worth noting that the edition
will be focused on modifying the style or changing minor things. For example,
in Figure 1, the streets are changed, while the main theme stays the same. We
argue that if the change can be arbitrary and the change is too much such that
the whole image is changed, there is no IP issue anymore.

On the defense side, Bob embeds invisible watermarks into the images before
they are published and before they are accessed by the attacker. We use
$\hat{{x}}\leftarrow\mathtt{Inj}({x},w)$ to denote the injection of a
watermark $w$ into image ${x}$. Note that Bob may have some high-level ideas
about the architecture of the editing model $\mathtt{Edit}$, but does not know
the details of $\mathtt{Edit}$, and neither can he modify $\mathtt{Edit}$.
Moreover, Bob has no idea what the image-editing prompt $p$ Alice will use.
After modifications by Alice, Bob can tell, and, more importantly, prove this
edited watermarked image $\hat{{x}}_{e}$ comes from the original image ${x}$.
Table I gives the key operations.

####  3.2.1 Formalizing Desired Goals

We identify two goals in the process, which we formalize as follows.

Invisibility. First, the injected watermark should be invisible. Moreover,
once edit models edit the watermarked images, Alice should not find out the
image has been watermarked.

Extractability. The watermark should be robust and extractable on any edit
operations (i.e., any edit model and prompt). As mentioned earlier, the
modification cannot be large; otherwise, it is unclear if there are IP issues
anymore. We will quantify the amount of modification later.

TABLE I: Summary of key operations.

Party | Operations | Description
---|---|---
Bob | $\hat{{x}}\leftarrow\mathtt{Inj}({x},w)$ | Inject watermark $w$ into image ${x}$
Alice | $\hat{{x}}_{e}\leftarrow\mathtt{Edit}(\hat{{x}},p)$ | Edit image $\hat{{x}}$ with text prompt $p$
Bob | $w^{\prime}\leftarrow\mathtt{Ext}(\hat{{x}}_{e})$ | Extract the watermark

Alice Bob${x}^{0}\xleftarrow{\$}\mathbb{X}$
${x}^{1}\leftarrow\mathtt{Inj}({x}^{0},w)$ $b\xleftarrow{\$}\\{0,1\\}$ Send
${x}^{b}$ ${x}^{b}_{e}\leftarrow\mathtt{Edit}({x}^{b},p)$ Send ${x}^{b}_{e}$
$w^{\prime}\leftarrow\mathtt{Ext}({x}^{b}_{e})$Guess $b_{a}$ Guess $b_{b}$
Figure 2: We formalize the problem definition into a game between Alice and
Bob. Alice wins if $b_{a}=b$ (Alice can detect whether there is a watermark
based on $x^{b}$ and $x_{e}^{b}$), and Bob wins if $b_{b}=b$ (Bob can extract
the watermark after edits).

To capture these goals more precisely, we define a crypto-style game between
Alice and Bob in Figure 2. First, Bob gets a randomly sampled image from a
pre-defined set, and inserts a watermark $w$ into it. The two images are
denoted by ${x}^{0}$ and ${x}^{1}$, respectively. Then one of the images
${x}^{b}$, where $b$ is a random bit, is sent to Alice for editing. Then based
on the edited image ${x}^{b}_{e}$, Bob tries to extract the inserted
watermark. Ideally, $w^{\prime}\approx w$ only if $b=1$, and in which case,
Bob guesses $b_{b}=1$; if $w^{\prime}$ is random or is different from $w$ Bob
guesses $b_{b}=0$. Bob wins if $\hat{b}=b$. At the same time, we want the
watermark to be invisible. So if Alice can distinguish if the image was
watermarked or not, Alice can win. Both Alice and Bob can win at the same
time, but our goal is to make only Bob win.

Our game operates in the black-box setting, i.e., Alice does not know the
details about the $\mathtt{Inj}$ and $\mathtt{Ext}$ operations. On the other
hand, Bob has some knowledge about the $\mathtt{Edit}$ operation (e.g., some
publicly available edit model). We acknowledge this is a limitation of our
study, and leave the exploration of the more challenging scenarios (e.g.,
Alice knows $\mathtt{Inj}$ and $\mathtt{Ext}$ and becomes adaptive when
choosing prompts) as future work.

####  3.2.2 Problem Scope

As mentioned in related work, there has been a surge of research on knowledge
copyright protection within diffusion models, categorized into preventing
image styles from being learned by diffusion models [47], guarding against
style transfers [28], and safeguarding diffusion models from unauthorized
theft or misuse [10, 57, 65]. Our watermarking technology primarily focuses on
protecting artwork or photographs from unauthorized modifications or malicious
edits by diffusion model-based editing tools. Artists and photographers can
utilize our tool to embed subtle, nearly undetectable perturbations into their
work, ensuring minimal impact on the original image quality before online
publication.

###  3.3 Existing Image Watermarking Schemes

####  3.3.1 Digital Image Watermark

Early work on image watermarking focuses on designing digital watermarks [9,
48] that integrate watermarks into colored images. Naive methods insert
watermarking messages into specific locations of the image; more recent work
embeds the watermark in the transformed space to make it subtle and less
discernible to the human eye, but can be extracted when necessary using the
inverse transformations.

Figure 3: The example of watermarking using an AutoEncoder. The original
artwork, after being watermarked by Injector $\mathtt{Inj}$, can have the
corresponding watermark successfully extracted by Extractor $\mathtt{Ext}$.
After being edited by Instructpix2pix with the text prompt “Have it be the
valley of the Ten Peaks in the Himalayas”, $\mathtt{Ext}$ can no longer
retrieve the watermark.

####  3.3.2 AutoEncoder-based Watermark

More recently, researchers built watermarking schemes using the AutoEncoder
architecture [31, 65]. In the AutoEncoder framework [59, 63], an encoder is
trained to embed watermarks, while a decoder is trained for watermark
extraction. Note that the encoder and decoder have similar functionalities as
those defined in Equation 1, and we slightly abuse notations and reuse ${E}$
and ${D}$ to denote them. But the internal neural network architectures are
quite different.

Specifically, an encoder ${E}$ is trained as $\mathtt{Inj}$ to embed
watermarks, while a decoder ${D}$ is trained as $\mathtt{Ext}$ for watermark
extraction. Specifically, during the embedding phase, both the image ${x}$ and
watermark $w$ are fed into the ${E}$ and encoded into a latent space. In this
latent space, the encoder integrates the watermark with the image and then
upsamples it back to the image domain, yielding a subtly modified image
$\hat{{x}}\leftarrow{E}({x},w)$ that remains visually indistinguishable from
the original format. The decoder ${D}$ is then trained to extract the
watermark $w$ from $\hat{{x}}$. The optimization of the AutoEncoder is as
follows,

$\displaystyle\underset{{D},{E}}{\mathrm{arg\,min}}\;\left\|{D}(\hat{{x}})-w\right\|_{\ell_{2}}+\lambda[\left\|\hat{{x}}-{x}\right\|_{\ell_{2}}+L_{lpips}(\hat{{x}},{x})]$
(2)

where the first term ensures the decoder can extract the watermark $w$
embedded by the encoder, and the second term guarantees that the watermarked
image $\hat{{x}}$ visually approximates the original ${x}$ (by measuring
similarity between $\hat{{x}}$ and ${x}$ using $\ell_{2}$ norm and LPIPS loss
[64]; but other metrics will also apply), and $\lambda$ is used to balance the
scales of these two terms.

However, our evaluations (details in Subsection 5.2) reveal that images
watermarked using this method struggle to withstand distortions introduced by
the edit models. As a result, it becomes challenging to extract the embedded
watermark from the modified image after editing. As illustrated in Figure 3,
the watermarked image $\hat{{x}}$ is visually indistinguishable from the
original image, and the decoder ${D}$ successfully extracts the embedded
watermark $w$ from the $\hat{{x}}$. However, post-editing by image-editing
models, the watermark extracted from image $\hat{{x}}_{e}$ is difficult to
see.

##  4 Our Method

We identify two fundamental flaws for the existing AutoEncoder-based
watermarks: (1) they do not consider image edits, and (2) the ‘one-model-fit-
all’ approach inherently makes training such a model more challenges, i.e.,
because the edits are so diverse, it is difficult to ensure the training
converge. Overcoming the first issue is easier: we just need to redefine the
objective function to incorporate image edits. On the other hand, the second
issue is more fundamental, and we propose to decouple the encoding and
decoding phases and consider/optimize them separately.

###  4.1 Warmup: Incorporating Image Edit

Given that the AutoEncoder-based solution is not robust against perturbations
from image-editing models, we make two efforts to improve it. First, we
incorporate adversarial learning [61] into the optimization objective as
follows:

$\displaystyle\underset{{D},{E}}{\mathrm{arg\,min}}\;\left\|{D}(\hat{{x}}+\alpha{N})-w\right\|_{\ell_{2}}+\lambda[\left\|\hat{{x}}-{x}\right\|_{\ell_{2}}+L_{lpips}(\hat{{x}},{x})]$

Compared to the original objective, here we introduce an isotropic Gaussian
noise denoted by ${N}$, and $\alpha$ represents the strength of the noise. Now
${D}$ is able to extract the watermark $w$ from images with $\alpha$ scaled
Gaussian noise, but it still fails to extract the watermark after the edited
models. This is because the Gaussian noise we introduced and the diffusion
process’s perturbation on the image are not exactly the same.

To address this, we further fine-tuned ${D}$ using watermarked images that had
been edited, and the objective function becomes

$\displaystyle\underset{{D},{E}}{\mathrm{arg\,min}}\;\left\|{D}(\hat{{x}}_{e}),w\right\|_{\ell_{2}}+\lambda[\left\|\hat{{x}}-{x}\right\|_{\ell_{2}}+L_{lpips}(\hat{{x}},{x})]$
(3)

Unfortunately, in our evaluation, ${D}$ fails to fit edited images during the
training process, and the training loss escalates $0\%$. We hypothesize that
the watermark information added by ${D}$ is destroyed by the editing process.

###  4.2 Watermark Injection

Having identified the difficulty of training the autoencoder, our approach
decouples the watermark injection and extraction processes and consider them
separately. One significant distinction from prior solutions is we formulate
this process as a per-image optimization problem. Here the challenge is that
different from training a model explicitly, we need to identify the direction
of the optimization. A natural consideration is to make the watermark
invisible, but then how to make sure it will be extracted later (without
knowing what the extractor does)? And how to ensure the optimization is
efficient?

To tackle the first question, we propose to make the watermarked image
$\hat{{x}}$ carries information about the watermark $w$ in the
latent/embedding space. That is, suppose we know the encoder ${E}$ in the
image-editing model (i.e., ${E}$ from Equation 1), we can ensure
${E}(\hat{{x}})$ maintains information about ${E}(w)$, or equivalently, the
distance between ${E}(\hat{{x}})$ and ${E}({x}+\alpha\cdot w)$ is small, where
$\alpha$ denotes the strength of the watermark. The intuition behind working
in the latent space is that image editing models typically target and modify
only the instruction part, while retaining unrelated semantic information,
thus retaining the watermark.

At the same time, we want to make sure $\hat{{x}}$ is similar to ${x}$, as
quantified earlier. We borrow the similarity measurement from previous
optimization objectives, but with the LPIPS loss replaced by a post-edition
invisibility penalty. Specifically, we borrow the decoder ${D}$ from Stable
Diffusion [41] (${E},{D}$ is a pair; note that although we also use these two
notations to denote the encoder and decoder of the autoencoder, their internal
architectures are significantly different), which is utilized to retrieve the
generated image from the latent space. ${D}({E}(\hat{{x}}))$ gives the edited
watermarked image, but without any prompt. Intuitively, if
${D}({E}(\hat{{x}}))$ is similar to ${x}$, meaning that the watermark is
invisible, then after the actual edition, the watermark will also be
invisible. This leads to our proposed optimization objective:

$\displaystyle\underset{\hat{{x}}}{\mathrm{arg\,min}}$
$\displaystyle\;\left\|{E}(\hat{{x}})-{E}({x}+\alpha\cdot
w)\right\|_{\ell_{1}}$
$\displaystyle+\lambda\left[\left\|\hat{{x}}-{x}\right\|_{\ell_{2}}+\left\|{D}({E}(\hat{{x}}))-{x}\right\|_{\ell_{2}}\right]$
(4)

Here the first component compares in the embedding space, and $\ell_{1}$ norm
is a natural metric; but like before, other metrics can also be used. We
explore the interplay between $\alpha$ and watermark detection precision in
Subsection 5.4. The parameter $\lambda$ serves as a weighting factor to
balance the strength of the watermark in the latent space with the quality of
image generation. Note that we employ an encoder in our optimization function
similar to the target editing models. Our results indicate that this
optimization approach is not relying on having identical encoders in terms of
training parameters and data. By adjusting parameters $\alpha$ and encoder
loss $\lambda$, i.e., latent space component in the optimization objective, we
can modulate the watermark’s intensity post-editing, rendering it invisible,
as depicted in 8(a) and 8(b).

To ensure the optimization problem is solved efficiently, we leverage and
adopt the PGD [33] algorithm, which was proposed to find adversarial examples
and is efficient. Our optimization process is given in Algorithm 1. Given
${x}$ and $w$, we first $\hat{{x}}^{0}$, and execute a series of optimization
steps, in a way similar to PGD. To limit the impact of our watermark on the
original image, we constrain added pixel values within $\varepsilon$.
Additionally, during optimization, each image modification is restricted to a
scale (intuitively a ‘learning rate’) of $\mu$.

Algorithm 1 Watermark Injection Pipeline

0: Image ${x}$, transparency $\alpha$, watermark function $W$ and text $w$,
encoder ${E}$, decoder ${D}$, optimization rate $\mu$ upper bound
$\varepsilon$ and steps $T$.

1: Define objective function $L_{inj}$ based on Equation 4

2: Generate Target image ${x}^{\prime}\leftarrow{x}+\alpha\cdot w$

3: Initial watermarked sample $\hat{{x}}^{0}\leftarrow{x}$

4: for $t\leftarrow 0$ to $T-1$ do

5: $\hat{{x}}^{t}\leftarrow\prod(\hat{{x}}^{t}+\mu\cdot
sgn(\bigtriangledown_{\hat{{x}}^{t}}L_{\mathtt{Inj}}({E},\hat{{x}}^{t},{x}^{\prime})))$

6: Calculate $\delta=\hat{{x}}^{t}-{x}$ and bound it with $\varepsilon$ in
$\ell_{1}$

7: Update $\hat{{x}}^{t+1}$ by $\hat{{x}}^{t}+\delta$

8: end for

9: return $\hat{{x}}^{T}$

###  4.3 Watermark Extraction

To extract the watermark, a baseline solution is to directly calculate the
pixel-wise distance between the watermark $w$ and the edited image
$\hat{{x}}_{e}$, and can claim successful extraction if the distance is below
a threshold. However, we found the pixel-wise measurement is not stable in
edit models, because the editions, while semantically minor, can cause a lot
of pixels to be modified, potentially leading to a large pixel-wise distortion
in the extracted watermark. Moreover, as our optimization goal in Equation 4
involves making the watermark invisible post-edition, this intuitively cannot
work.

To overcome this issue, we propose to extract at a higher level (semantic
level). Here, we propose a simple yet effective design, backed by two
intuitions: (1) focus on image semantics instead of details (e.g., pixels),
(2) boost accuracy by ‘divide and conquer’.

Specifically, we use simple images that are just plain texts with black
background as watermarks (and to differentiate, we call it text watermarks and
the previous general watermark as image watermarks). For examples of text and
image watermarks, respectively, refer to Figure 5 and Figure 8. This ensures
the higher-level semantics of images are easier to capture (i.e., just
recognize texts in the image, using off-the-shelf OCR models). We also segment
the watermark into $K$ regions, so that we will have more confidence if we can
recognize texts (which can also serve as the passcode of the watermark) in
most regions correctly.

###  4.4 Discussion

To summarize, although we express our target similarly to the optimization
objectives in the AutoEncoder-based solutions (for ease of understanding), the
underlying technique is fundamentally different: (1) We decouple the watermark
injection and extraction processes and consider them separately. (2) We
formulate the watermark injection process as a per-image optimization problem
using PGD. (3) We leverage information about the encoder and decoder and
rewrite the objective to incorporate different possibilities of image edition.

The efficacy of our watermarking method is attributed to their shared
foundation on Stable Diffusion [41]. Although the encoder we use during
watermark optimization is not identical to those used in the edit models,
these encoders share similarities. Especially, our watermark’s resilience to
distortions in the editing process is attributed to integrating the edit
model’s latent space information during optimization, allowing it to convey
visual semantic information similar to that of the target image in the latent
space. Another potential reason is: edit models strive to preserve the
original image’s semantic essence and only modify specific instruction-driven
sections. We argue that due to the widespread open-source availability of
generative models, such as Stable Diffusion [41], accessing similar versions
of encoders is not challenging.

Although our watermark, based on visual semantic information, exhibits a
higher preservation rate in the edit model, adaptive editing attack [31] or
image reconstruction based on generative models [13, 17] might remove our
watermark. We defer the exploration of such attacks to future work. Moreover,
since our watermark must be imperceptible to the human eye, yet present within
the latent space of the diffusion, the model optimization of $\hat{{x}}$
relies on the latent space information of the edit model. This leads to the
challenge of generating high-quality watermarks in a complete black-box
scenario.

##  5 Evaluation

Summary of Results. Our $\mathsf{RIW}$ (Robust Invisible Watermarking) is able
to successfully inject an invisible watermark and extract it after editing.
For instance, in the evaluation where we vary the transparency of the target
image watermark, we found that even when the text watermark is very faint, at
$\alpha=15/255$, we can determine ownership with $85.5\%$ accuracy. When the
text watermark transparency reaches $\alpha=102/255$, our accuracy rises to
$96.3\%$, and at this point, the watermark in the edited image remains
invisible, as shown in 8(a). However, conventional watermarks and those added
based on neural networks fail to be retained amidst editing interference, even
with enhancement.

###  5.1 Experiment Setup

Datasets. We evaluate our approach using two datasets: the extended LAION-
Aesthetics and TedBench. The LAION-Aesthetics has been widely used for
training diffusion models [39, 7, 41], while TedBench [25] has been introduced
as a benchmark dataset to measure edit models.

  * •

LAION-Aesthetics222http://instruct-pix2pix.eecs.berkeley.edu/: Due to its
large size and diverse content, the primary use of this dataset is image
generation. We apply LAION-Aesthetics V2 6.5+ [46], a subset of the LAION’s 5B
samples, which comprises 625K image-text pairs and includes various image
types, such as paintings, digital artwork, and photographs. Brooks et al.
expand the dataset to include pairs of images before and after editing, along
with their corresponding edit instructions by fine-tuning GPT-3 [8] and
employing the Prompt-to-Prompt [16] technique. Additionally, they apply a
CLIP-based metric [7] to filter out noise like nonsensical or undescriptive
captions from this dataset, enhancing its quality and applicability.

  * •

TEdBench333https://github.com/imagic-editing/imagic-editing.github.io/: Kawar
et al. have proposed TEdBench (Textual Editing Benchmark), a benchmarking
dataset comprising 100 pairs of input images and target text that describe
intricate non-rigid edits [25].

Image Editing Model. We utilize two state-of-the-art image editing models,
i.e., Instructpix2pix [7] and Imagic [25]. The design principles of these two
models are entirely different: One is built as an editing model by training on
a pair of images–one representing the subject to be edited and the other
representing the image post-editing [7]–while the other is based on fine-
tuning the model using images that require editing [25]. Both models harness
the capabilities of latent space diffusion models [41] as their foundational
generative framework. Detailed descriptions about these models are deferred to
Appendix A.2.

Optimization Details. In our optimization function, we utilize the version
“sd-v1-4.ckpt” for the encoder, while the edit model Instructpix2pix employs
“sd-v1.5.ckpt” and Imagic uses the “sd-v1-4-full-ema.ckpt” version. Due to its
fine-tuning paradigm, this model might slightly deviate from the original one.
We employ the LPIPS (Learned Perceptual Image Patch Similarity) loss [64] to
ensure the fidelity of our watermarked image $\hat{{x}}$ in the second term of
the optimization objective. All results presented in this paper are based on
settings $\varepsilon=12/255$, $\mu=2/255$. We set $\lambda=1$ in the
evaluation of modification $\alpha$. Appropriate adjustments of $\lambda$ may
lead to improved watermark quality, and we will explore these in future work
if needed.

Watermarks Used. We crop the image into $9$ segments/blocks, with each size
$(173,66)$, corresponding to the nine individual watermarks $w$ we have
embedded, each consisting of four letters as seen the first example in 8(b).

As a baseline for comparison, we also embed the watermark in the lower right
corner of the image, as shown in the first row of Figure 5. In this case, we
employ images from CIFAR10 [12] as watermarks.

To differentiate the two settings, we call them text watermarks and image
watermarks, respectively.

Extraction Details. For text watermarks, we first fine-tune MPRNet [60] to
reconstruct low-resolution watermarks in edited images. Note that due to the
time-consuming nature of data collection—for example, a single Imagic edit
takes 10 minutes on an NVIDIA A100—we limited the fine-tuning of MPRNet to a
dataset of $1,000$ images for both Instructpix2pix and Imagic. For the second
stage of our extraction process, we use an Optical Character Recognition
(OCR444https://github.com/clovaai/deep-text-recognition-benchmark)-based
model. For training our extractor $\mathtt{Ext}$, we utilized additional
$5,000$ images from Instructpix2pix and $2,000$ from Imagic.

For image watermarks, as no recognition is needed, we directly compare pixels
between the edited image and the watermark image.

Evaluation Metrics. Success rate of image watermark extraction is quantified
by computing the pixel-wise $\ell_{2}$ distance and compare it against a
threshold for determination whether the image was watermarked or not. We then
measure the Receiver Operating Characteristic (ROC) curve and the Area Under
Curve (AUC) metric.

For text watermarks, we first establish the accuracy function $A$ for every
region’s watermark $\hat{{x}}_{k}$ ($k\in K$) with function $\mathtt{Ext}$ as
follows,

$\displaystyle A(\hat{{x}}_{k})=\begin{cases}1&\text{if
}\mathtt{Ext}(\hat{{x}}_{k})=w\\\ 0&\text{otherwise}\end{cases}$

Similarly, we define $A(\hat{{x}}_{n,k})$ as the $k$-th watermark in the
$n$-th image. To quantify the watermark identification success in different
scenarios, we define three metrics based on $A$: (1) First, we calculate the
extraction accuracy rate of sub-watermarks across the entire test set as
$\mathbf{D_{all}}=\frac{1}{NK}\sum_{n=1}^{N}\sum_{k=1}^{K}A(\hat{{x}}_{n,k})$.
(2) To measure the success rate of image ownership claims, we define metric
$\mathbf{D_{word}}=\frac{1}{N}\sum_{n=1}^{N}\min\left\\{\sum_{k=1}^{K}A(\hat{{x}}_{n,k}),1\right\\}$.
The implication of this metric is that among the $K$ identical independent
watermarks we introduce, if even one is successfully recognized, it is deemed
as proof of ownership. (3) Finally, we introduce metric $\mathbf{D_{letter}}$
to measure the identification success rate of special character watermarks.
This method allows developers to use unique symbols as watermarks to stake
their ownership claims. For instance, in an individual watermark containing
$M$ characters, only all $M$ symbols are successfully identified within an
image, and can be combined into the entire added symbol sequence, we consider
the watermark recognition to be successful. Note that we ignore the false
positives as we observe low false positive rates of $0\%$, $0\%$, and $25.5\%$
across the three metrics for the no-watermark cases.

Additionally, for text watermarks, we consider confidence scores of the OCR
model when determining watermark results and construct ROC curves. Even when
the output is correct, if the confidence is low, we still consider the
watermark as not having been successfully extracted.

Evaluation Setup. We evaluate the performance of our $\mathtt{Ext}$ on
$10,000$ images, comprising equal numbers of watermarked and non-watermarked
images. Given these results, our primary focus was on accurately extracting
watermarks from the text-watermarked images. Moreover, considering the
extensive time overhead, with Imagic [25] taking $10$ minutes per image edit,
we confined our watermark strength tests to about $100$ images for each
parameter, i.e., $\alpha$ and $\lambda$, across two edit models.

Figure 4: Digital watermark examples: the first row shows the results of the
original images after adding a digital watermark, rows two to four highlight
traditional attacks like cropping, noise, and masking, while the last row
presents post-edited images.
Figure 5: The first row displays the target image overlaid with an image
watermark, featuring a $32\times 32$ sized watermark in the bottom right
corner (marked with a red box), randomly selected from CIFAR10 [12]. The
second row showcases the invisible watermark added by our technique, and the
third row illustrates the outcome post-editing of the watermarked image.

(a) Strength (Image watermark)

(b) Encoder Weight (Text watermark)

(c) Transparency (Text watermark)

Figure 6: In Figure (a), we evaluate the ROC curve for image watermarking with
different transparency levels $\alpha$. The clearer the image watermark is
(less invisible), the better the classification results. However, compared to
text watermarking, whether adjusting the encoder weights in (b) or the
transparency of the watermark in (c), the preservation ability of the
watermark under the editing model is stronger than that of image watermarking.
Note that in (b), encoder weight $\lambda=2^{-k}$.

(a) Watermark Transparency

(b) Encoder Weight

(c) Watermark Transparency

(d) Encoder Weight

(e) Watermark Transparency

(f) Encoder Weight

(g) Watermark Transparency

(h) Encoder Weight

Figure 7: Watermark identification accuracy varies with the increase in
watermark transparency and encoder weight on the LAION-Aesthetics (above) and
TEdBench (below) datasets. Subfigures (a), (b), (e) and (f) display the
outcomes when edited using the Instructpix2pix model, while (c), (d), (g) and
(h) showcase results from editing via the Imagic model.

(a) Examples of the Instructpix2pix model.

(b) Examples of the Imagic model.

Figure 8: In both subfigures, the first row represents the target images,
denoted as ${x}^{\prime}$, exhibiting varying watermark transparency $\alpha$.
The second row displays the images after the watermark is injected, denoted as
$\hat{{x}}$. The last row illustrates the corresponding edited images, denoted
as $\hat{{x}}_{e}$.

(a) Watermarking encoder weight $\lambda$ difference with Instructpix2pix
model.

(b) Watermarking encoder weight $\lambda$ difference with Imagic model.

Figure 9: The trend in encoder loss weight for both Instructpix2pix and Imagic
models, as observed in (a) and (b), reveals that at a minimal weight such as
$128$, watermarks become entirely invisible post-edit. Conversely, at lower
weights like $1/128$, watermarks become observable after editing. An optimal
trade-off is found at a weight of $1/2$, where watermarks typically remain
unseen post-edit and yet have a high probability of being extracted by
$\mathtt{Ext}$.

###  5.2 Comparison with Existing Solutions

Compared to prior watermarking methods like digital watermarks [9, 48] and
AutoEncoder-based watermarks [59, 63], our approach offers notable benefits.
Traditional methods yield a $0\%$ retention in editing tasks. Given the nature
of the diffusion process, especially in diffusion models (DMs) tailored for
editing, watermarks from these methods face extensive distortion post-editing,
rendering them unextractable.

Digital Watermark. We employ a popular
tool555https://github.com/guofei9987/blind_watermark based on frequency domain
techniques for the insertion of digital watermarks. Through evaluation, we
find that the digital watermark demonstrates excellent robustness against
various common image transformations such as rotation, random cropping, image
masking, cutting, resizing, noise introduction, and brightness adjustment. It
successfully recovers the four-letter text watermark with $100\%$ accuracy.
However, when we subjected $100$ images from LAION-Aesthetics containing the
digital watermark to edits using either the Instructpix2pix or Imagic model,
the accuracy for extracting the corresponding watermark dropped to zero.

Examples of digital watermarks are depicted in Figure 4, showcasing the
robustness and limitations of the digital watermark. The first row exhibits
original images with the applied watermark (text “Robust InvisiWater”).
Subsequent rows (two to four) demonstrate the watermark’s resilience against
traditional attacks like cropping, random noise, and masking, with successful
extraction still possible. However, the edited images in the final row
illustrate the watermark’s extraction failure. We can also observe that the
impact of traditional image transformations on images is entirely different
from that introduced by edit models. Furthermore, our analysis revealed that
digital watermarks exhibit less resistance to brightness and rotation
alterations than expected. A mere $15$-degree rotation or a $1.2$-fold
increase in brightness rendered the watermark extraction unfeasible.

AutoEncoder-based Watermark. Given the popularity of utilizing AutoEncoders
for watermarking tasks [59, 63], particularly in recent work that employs
AutoEncoder to add watermarks to diffusion models [65], we also evaluate a
paired Encoder-Decoder666https://github.com/yunqing-me/WatermarkDM scheme in
our study, i.e., ${E}$ and ${D}$. For this evaluation, we selected $10,000$
images from the LAION-Aesthetics dataset and generated $5,000$ three-letter
random text watermarks for training the AutoEncoder.

We evaluate both Imagic and Instructpix2pix models on $100$ images, the
decoder ${D}$ extracts watermarks with $100\%$ accuracy from unedited images.
However, this accuracy plummets to $0\%$ post-editing. To enhance ${D}$’s
resilience against such manipulations, we employ adversarial training
techniques aimed at improving ${D}$’s ability to extract watermarks from
perturbed images. However, the watermarks added to the images by the ${E}$
were not retained following the editing process, resulting in $0\%$ extraction
accuracy. In addition, we attempt to fine-tune the ${D}$ that had undergone
adversarial training, using images that had been edited by both Imagic and
Instructpix2pix models. A total of $1000$ images were used for this purpose,
with $500$ images from each model. Despite the fine-tuning, there was no
observable improvement in extraction accuracy, remaining $0\%$.

TABLE II: Time (seconds) required by various watermarking methods to add watermarks to individual images. And pre-training or auxiliary needs for each method. Watermark Tool | Digital | AutoEncoder | $\mathsf{RIW}$
---|---|---|---
Time Cost (s) | 0.85 | 0.42 | 95.03
Training-Free | ✓ | ✗ | ✓

Running Time Comparison. We evaluate the time overhead of these methods for
watermarking an individual image, with results presented in Table II. The
AutoEncoder algorithm exhibits the shortest time overhead; however, this
excludes the training time of AutoEncoder on $10,000$ images, for which we
spent around $7$ hours on two NVIDIA A100 GPUs. Our $\mathsf{RIW}$ (Robust
Invisible Watermarking) watermarking process is time-intensive, due to our
optimization procedure with $400$ steps, but nothing else is required.

###  5.3 Results of Naive Image Watermarks

We opt to use images from CIFAR10 [12] as watermarks primarily due to their
authoritative nature and widespread use in visual tasks. Extending this
rationale, CIFAR10’s diverse and representative image set offers a robust
basis for watermarking, ensuring that our technique can be generalized across
various image types and applications. In emulating traditional backdoor attack
[15], we strategically place the watermark in the lower right corner, a choice
driven by the need for subtlety and reduced visibility. This positioning not
only makes the watermark less conspicuous but also leverages the typical
viewing patterns of images, where the lower right area often receives less
immediate attention.

The optimization results for image watermarking, as seen in Figure 5, are not
effective due to challenges like image continuity and complexities in image
processing. These issues, along with the limitations of current optimization
algorithms and maintaining watermark integrity during image transformations,
require further analysis for improved image watermark optimization strategies.

Extraction results are illustrated in Figure 6. Upon comparison using ROC
curves and AUC values, we observe that image watermarks significantly
underperform text watermarks in the task of detecting the presence of a
watermark. For instance, when the transparency of the image watermark is set
to $1$, the AUC remains at a modest $0.58$, which is substantially lower than
the $0.85$ AUC achieved by text watermarks with a transparency setting of
$0.3$. A plausible explanation for this discrepancy might be the greater
challenge inherent in optimizing image watermarks. Moreover, compared to our
two-stage extraction approach for text watermarks, pixel distance metrics
evidently fall short in adequately addressing distortions to watermarks
introduced by the editing model. Furthermore, adjusting encoder weights
enhances watermark detection, achieving an AUC value over $0.9$, which will be
elaborated upon in the following Subsection 5.4 evaluation of text watermarks.

One open question is to explore the performance of image watermarks where the
image covers the entire image with an image watermark. However, due to the
richer and more vibrant pixel composition of images compared to text
watermarks, this significantly increases the optimization difficulty,
resulting in the added watermark ultimately becoming visible.

###  5.4 Results of Text Watermarks

As elaborated in Subsection 4.2, our text watermark’s strength in target
images affects its visibility in edited images. The watermark’s transparency
and the weight of the encoder loss in our adversarial training both impact
post-editing watermark extraction. Hence, we assess the impact of varying
watermark transparency and encoder loss weights on extraction accuracy in the
two editing models.

LAION-Aesthetics. Firstly, we evaluate the performance of our $\mathsf{RIW}$
on the LAION-Aesthetics dataset. As shown in 7(a) and 7(c), we observed that
as the transparency of the watermark in the target image ${x}^{\prime}$
increases, the extraction accuracy correspondingly improves across both
Instructpix2pix and Imagic editing model. Moreover, it was found that beyond a
certain level of transparency, specifically at around $102/255$, the
extraction accuracy across Instructpix2pix and Imagic begin to stabilize at
$96.0\%$ and $90.4\%$, respectively. As evident from 8(a), when the
transparency reaches $102/255$, the watermark we injected becomes visually
imperceptible in the edited images. Meanwhile, even when the transparency is
as low as $15/255$, the watermark can still be successfully extracted. Similar
results were observed with adjustments to the encoder weight $\lambda$. As
illustrated in 7(b) and 7(d), when we increase $\lambda$ to $1/2$ or lower,
our $\mathbf{D_{letter}}$ success rate stabilizes at $97.9\%$ and $90.4\%$.

TEdBench. Similar results are observed on the TEdBench dataset with both two
editing models, as shown in 7(e) and 7(g). Once the watermark transparency for
${x}^{\prime}$ reaches $51/255$, watermark identification accuracy levels off.
In both the Instructpix2pix and Imagic models, when the watermark transparency
is $51/255$, the identification accuracy reaches respective accuracies of
$92.4\%$ and $83.2\%$. Additionally, as shown 7(f) and 7(h), when $\lambda$ is
below $1/2$, identification accuracy progressively becomes stable, attaining
values of $91.4\%$ and $89.3\%$ respectively when $\lambda$ is equal to $1/2$.
The TEdBench dataset, with $50$ images and $100$ editing instructions, is
inadequate for training $\mathtt{Ext}$, which was trained exclusively on
LAION-Aesthetics. This contributes to the observed lower watermark
identification accuracy on TEdBench. A dip in accuracy occurs, as seen in
7(g), when watermark transparency surpasses $102/255$, likely due to
$\mathtt{Ext}$’s training with watermarks in ${x}^{\prime}$ at this default
transparency. Still, $\mathtt{Ext}$ demonstrates satisfactory
generalizability, with Imagic editing results in 8(b).

TABLE III: Improvement in watermark identification accuracy for both editing models Instructpix2pix (Ipx), Imagic (Igc) and datasets LAION-Aesthetics (LA), TEdBench (TB) under conditions of low watermark clarity when introducing reconstruction. Specifically, these results are obtained when transparency $\alpha$ is set to $1/255$ and $\lambda$ is $5e10$. Both of them control watermark strength (Stren). In the table, “w/o Rec” denotes results without reconstruction, while “w/ Rec” indicates results with reconstruction.  Edit | Dataset | Stren | w/o Rec | w/ Rec | Impr.
---|---|---|---|---|---
Ipx | LA | $\alpha$ | $49.6\%$ | $54.3\%$ | $\mathbf{9.5\%}$
$\lambda$ | $50.6\%$ | $55.7\%$ | 10.1%
TB | $\alpha$ | $49.2\%$ | $49.2\%$ | $0\%$
$\lambda$ | $45.2\%$ | $45.2\%$ | $0\%$
Igc | LA | $\alpha$ | $47.2\%$ | $50.7\%$ | 7.4%
$\lambda$ | $51.5\%$ | $57.0\%$ | 10.7%
TB | $\alpha$ | $41.1\%$ | $42.1\%$ | 2.4%
$\lambda$ | $51.2\%$ | $52.2\%$ | 2.0%

Figure 10: Distribution of semantic and visual changes before and after
editing with the Instructpix2pix and Imagic models. The first row shows the
distribution for the Instructpix2pix model with semantic and vision distance
(very close), while the second row shows those for the Imagic model.

Impact of Encoder Weight. Figure 9 illustrates how adjusting the encoder
weight loss $\lambda$ can alter the prominence of the watermark. As $\lambda$
approaches $1/2$, the watermark extraction accuracy stabilizes, as shown in
Figure 7. However, when $\lambda$ reaches $1/128$, we can faintly discern a
subtle watermark after editing. Users can adjust the watermark strength
parameter in our optimization function to determine the visibility of the
watermark after the image is edited. When the watermark is conspicuous, it
serves as a warning against malicious edits. When the watermark is invisible,
it assists in claiming the copyright of the edited image.

###  5.5 Benefit of Reconstruction

We evaluate the efficacy of the reconstruction component in our method. We
observe that under conditions where watermark clarity is extremely low, the
reconstruction can be beneficial. For instance, on the LAION-Aesthetics
dataset using the Instructpix2pix, when watermark transparency is set to
$1/255$ or $\lambda$ is set to $5e4$, there is an increase in identification
accuracy by $9.5\%$ and $10.1\%$, respectively. For a more comprehensive view
of results across two edit models and two datasets of low clarity watermark
shown in Table III.

(a) Examples edited by Instructpix2pix

(b) Examples edited by Imagic

Figure 11: Examples with notable semantic and vision distances before (first
row) and after (second row) editing by the Instructpix2pix and Imagic edit
models. In each subfigure, the left panel represents semantic distance while
the right shows vision distance. P indicates the corresponding distance
values.

(a) $\mathbf{D_{letter}}$ accuracy

(b) $\mathbf{D_{word}}$ accuracy

(c) $\mathbf{D_{all}}$ accuracy

Figure 12: The watermark extraction accuracy diminishes with increasing edit
distance. As edit distance often correlates with the extent of image
modifications, extensive edits lead to reduced preservation of the original
image’s IP information, consequently diminishing IP infringement on the
original image.

###  5.6 Watermark Protection Boundary

Considering that editing models can modify an image’s distribution and
semantics, preserving ownership amid significant changes to the image’s
content or style becomes crucial. Thus, we assess the degree of alteration in
the image’s semantic or visual content after editing with either the Imagic or
Instructpix2pix models. As the distance in both semantic and visual dimensions
increases, the watermark becomes increasingly difficult to extract due to the
loss of original image information. Figure 10 displays the distribution of
changes in images before and after editing. The two images on the left
represent the semantic distance based on CLIP [38], while the two images on
the right show the visual distance calculated using VGG [49]. Specifically,
akin to existing works [38, 64, 45], when calculating distances using CLIP and
VGG, we first project the images onto embeddings and then compute the
$\ell_{2}$ distance between these embeddings.

As observed from Table IV, when we incorporate the top $5\%$ of images with
the largest changes in semantic and vision distance into our analysis, there
is a varying degree of decline in the accuracy of different watermark
extraction strategies. Notably, the Instructpix2pix model tends to have a more
significant impact than the Imagic. This is attributed to the fact that
Instructpix2pix induces a larger shift in the image distribution, as also
indicated by the distance threshold (Thred), i.e., $46.7\%$ and $47.5\%$
higher in semantic and vision distance than Imagic.

TABLE IV: Watermark extraction accuracy for the top $5\%$ based on the Semantic (Sem) and Vision (Vis) distance (Dist) before and after editing with Instructpix2pix (Ipx) and Imagic (Igc) respectively, along with the specific distance threshold (Thred). Edit | Dist | Thred |  $\mathbf{D_{letter}}$$\downarrow$ |  $\mathbf{D_{word}}$$\downarrow$ |  $\mathbf{D_{all}}$$\downarrow$
---|---|---|---|---|---
Ipx | Sem | $0.44$ | $0.21\%$ | $0.51\%$ | $2.73\%$
Vis | $0.62$ | $0.23\%$ | $0.56\%$ | $3.27\%$
6 |  |  |  |  |
Igc | Sem | $0.30$ | $0.06\%$ | $0.11\%$ | $0.07\%$
Vis | $0.42$ | $0.11\%$ | $0.32\%$ | $0.00\%$

As shown in Figure 11, by comparing the visual differences before and after
editing based on the two models, we can observe that when the distance is
significant, the images are significantly altered, with the edited versions
retaining little of their original features. As a result, there might not be a
pressing need to further discuss the protection of its copyright.

Figure 12 illustrates the variation in accuracy of our three watermark
authentication metrics—$\mathbf{D_{all}}$, $\mathbf{D_{word}}$, and
$\mathbf{D_{letter}}$—for images before and after editing in the
Instructpix2pix [7] and Imagic [25] models, as the Semantic [38] and Vision
[49] distances increase. Overall, as the distance grows, there’s a trend of
decreasing accuracy. This is attributed to the fact that larger modifications
to the image after editing correspondingly have a greater impact on the
watermarks we have added.

##  6 Conclusion

In this paper, we are the first to address the issue of image copyright and
malicious alterations within the context of diffusion model-based editing
frameworks. We have formalized this issue and presented it within a Game
framework. Subsequently, we conduct an exhaustive study of prior watermarking
techniques, such as digital watermarks or AutoEncoder-based watermarks. We
discover that traditional watermarking approaches, even when enhanced, are
inadequate to handle the challenges introduced by the editing models of the
diffusion era. Building on this, we propose our invisible watermarking method
$\mathsf{RIW}$ (Robust Invisible Watermarking) based on visual-semantic
information. Compared to traditional watermarking techniques with $0\%$
retention, our method maintains a $96.0\%$ watermark extraction accuracy post-
editing. This ensures that our watermark effectively addresses the image
copyright and deceptive editing concerns in the editing model framework.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In Proceedings of the IEEE/CVF international conference on computer vision, pages 4432–4441, 2019.
  * [2] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8296–8305, 2020.
  * [3] Robin Rombach Aleksander Holynski.  instruct-pix2pix.  https://github.com/timothybrooks/instruct-pix2pix, 2023.
  * [4] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218, 2022.
  * [5] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  arXiv preprint arXiv:2211.01324, 2022.
  * [6] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023.
  * [7] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023.
  * [8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [9] Ingemar Cox, Matthew Miller, Jeffrey Bloom, and Chris Honsinger.  Digital watermarking.  Journal of Electronic Imaging, 11(3):414–414, 2002.
  * [10] Yingqian Cui, Jie Ren, Han Xu, Pengfei He, Hui Liu, Lichao Sun, and Jiliang Tang.  Diffusionshield: A watermark for copyright protection against generative diffusion models.  arXiv preprint arXiv:2306.04642, 2023.
  * [11] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in neural information processing systems, 34:8780–8794, 2021.
  * [12] Krizhevsky et al.  CIFAR-10 and CIFAR-100 dataset and analysis. .  https://www.cs.toronto.edu/~kriz/cifar.html, 2013.  [Online; accessed 15-March-2022].
  * [13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [14] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [15] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg.  Badnets: Identifying vulnerabilities in the machine learning model supply chain.  arXiv preprint arXiv:1708.06733, 2017.
  * [16] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [17] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [18] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  The Journal of Machine Learning Research, 23(1):2249–2281, 2022\.
  * [19] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [20] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz.  Multimodal unsupervised image-to-image translation.  In Proceedings of the European conference on computer vision (ECCV), pages 172–189, 2018.
  * [21] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [22] Zhengyuan Jiang, Jinghuai Zhang, and Neil Zhenqiang Gong.  Evading watermark based detection of ai-generated content.  arXiv preprint arXiv:2305.03807, 2023.
  * [23] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [24] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [25] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007–6017, 2023.
  * [26] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  arXiv preprint arXiv:2009.09761, 2020.
  * [27] Chumeng Liang and Xiaoyu Wu.  Mist: Towards improved adversarial examples for diffusion models, 2023\.
  * [28] Chumeng Liang, Xiaoyu Wu, Yang Hua, Jiaru Zhang, Yiming Xue, Tao Song, XUE Zhengui, Ruhui Ma, and Haibing Guan.  Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples.  2023\.
  * [29] Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz.  Few-shot unsupervised image-to-image translation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10551–10560, 2019.
  * [30] Yugeng Liu, Zheng Li, Michael Backes, Yun Shen, and Yang Zhang.  Watermarking diffusion model.  arXiv preprint arXiv:2305.12502, 2023.
  * [31] Nils Lukas, Abdulrahman Diaa, Lucas Fenaux, and Florian Kerschbaum.  Leveraging optimization for adaptive attacks on image watermarks.  arXiv preprint arXiv:2309.16952, 2023.
  * [32] Nils Lukas and Florian Kerschbaum.  Ptw: Pivotal tuning watermarking for pre-trained image generators.  arXiv preprint arXiv:2304.07361, 2023.
  * [33] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  arXiv preprint arXiv:1706.06083, 2017.
  * [34] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  arXiv preprint arXiv:2108.01073, 2021.
  * [35] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [36] Sen Peng, Yufei Chen, Cong Wang, and Xiaohua Jia.  Protecting the intellectual property of diffusion models by the watermark diffusion process.  arXiv preprint arXiv:2306.03436, 2023.
  * [37] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [39] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022\.  URL https://arxiv. org/abs/2204.06125, 7, 2022.
  * [40] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  * [41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [42] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  * [43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [44] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713–4726, 2022.
  * [45] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [46] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * [47] Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao.  Glaze: Protecting artists from style mimicry by text-to-image models.  arXiv preprint arXiv:2302.04222, 2023.
  * [48] Frank Y Shih and Scott YT Wu.  Combinational image watermarking in the spatial and frequency domains.  Pattern Recognition, 36(4):969–975, 2003.
  * [49] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [50] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  arXiv preprint arXiv:2209.14792, 2022.
  * [51] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International conference on machine learning, pages 2256–2265. PMLR, 2015.
  * [52] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [53] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  Advances in neural information processing systems, 32, 2019.
  * [54] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  arXiv preprint arXiv:2011.13456, 2020.
  * [55] Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon.  Dual diffusion implicit bridges for image-to-image translation.  arXiv preprint arXiv:2203.08382, 2022.
  * [56] Pascal Vincent.  A connection between score matching and denoising autoencoders.  Neural computation, 23(7):1661–1674, 2011.
  * [57] Yuxin Wen, John Kirchenbauer, Jonas Geiping, and Tom Goldstein.  Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust.  arXiv preprint arXiv:2305.20030, 2023.
  * [58] Mengjiao Yang, Yilun Du, Bo Dai, Dale Schuurmans, Joshua B Tenenbaum, and Pieter Abbeel.  Probabilistic adaptation of text-to-video models.  arXiv preprint arXiv:2306.01872, 2023.
  * [59] Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi, and Mario Fritz.  Artificial fingerprinting for generative models: Rooting deepfake attribution in training data.  In Proceedings of the IEEE/CVF International conference on computer vision, pages 14448–14457, 2021.
  * [60] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao.  Multi-stage progressive image restoration.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14821–14831, 2021.
  * [61] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell.  Mitigating unwanted biases with adversarial learning.  In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335–340, 2018.
  * [62] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412, 2017.
  * [63] Kevin Alex Zhang, Lei Xu, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni.  Robust invisible video watermarking with attention.  arXiv preprint arXiv:1909.01285, 2019.
  * [64] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [65] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Ngai-Man Cheung, and Min Lin.  A recipe for watermarking diffusion models.  arXiv preprint arXiv:2303.10137, 2023.

##  Appendix A Additional Background

We first present Denoising Diffusion Probabilistic Models (DDPMs) [17], which
serve as the foundation for image editing models, and then, we dive into
detailed descriptions of two state-of-the-art and open-sourced edit models we
use in this paper.

###  A.1 Denoising Diffusion Probabilistic Models

In DDPMs, a data item $x_{0}$, sampling from the original distribution,
undergoes a forward diffusion process to gradually approach the Gaussian
distribution $N(0,1)$. The reverse diffusion starts with a noise image sampled
from $x_{T}\sim N(0,1)$ and progressively refines it into a photorealistic
image $x_{0}$. Each intermediate sample, $x_{t}$, for $t\in{1,...,T}$, adheres
to the format,

$x_{t}=\sqrt{a_{t}}x_{0}+\sqrt{1-a_{t}}\varepsilon_{t}$

with the hyper-parameter $a_{t}\in(0,1)$ of the diffusion schedule and
$\varepsilon_{t}\sim N(0,I)$, each refinement step is predicted using the
neural network model $f_{\theta}$. Therefore, $x_{t-1}$ is derived from
$f_{\theta}(x_{t},t)$, followed by a Gaussian perturbation as,

$x_{t-1}=f_{\theta}(x_{t},t)+\sigma_{t}z$

with $z\sim N(0,I)$. The training objective of $f_{\theta}$ is optimized by
variational lower bound as [17, 51],

$L_{t}=E_{t\sim[1,T],x_{0},\varepsilon_{t}}\left[\left\|\varepsilon_{t}-f_{\theta}(\sqrt{a_{t}}x_{0}+\sqrt{1-a_{t}}\varepsilon_{t},t)\right\|^{2}\right]$

DDPM can be adapted to learn conditional distributions by conditioning the
denoising neural network $f_{\theta}$ on an auxiliary input $y$. Consequently,
the network $f_{\theta}(x_{t},t,y)$ samples from a data distribution based on
$y$. Typically, $y$ is a descriptive text for the training image [39, 5, 42,
43]. However, $y$ can also represent a low-resolution image [44] or a class
label in ImageNet [18]. For our editing tasks, $y$ encompasses both the edited
image and text prompt [7, 25].

TABLE V: Different style of text edit instruction between Instructpix2pix
(ipx) [7] and Imagic (igc) [25].

|  | Romantic Venice Oil Paintings | A photo of Barack Obama
---|---|---|---
Instruction Style | ipx | Add painting a helicopter | Turn Barack Obama into smiling with a big grin
igc | Romantic Venice with Helicopter Oil Paintings | A photo of Barack Obama smiling with a big grin

TABLE VI: An automated dialogue to generate edit instructions based on
GPT-4-0613 API, tailored to match the prompt styles of various editing models.
Prompt: Let’s play a game. I’ll provide a sentence A, which describes an
image, for example: “the little prince fox wallpaper”. Then, I’ll give a
modification instruction C for that image, such as “make the fox roguish”. The
outcome will be a modified description, sentence B, which could be: “roguish
fox wallpaper”. In our game, I’ll supply you with sentences A and B, and your
task is to deduce the corresponding modification instruction, sentence C.
Require C natually like the image editing instruction.
---
GPT4: Sounds like an interesting challenge, let’s go for it!
Prompt: Now, I provide the sentence A “A photo of a standing dog” and sentence
B “A photo of a sitting dog”. Now answer me with the corresponding instruction
sentence C.
GPT4: Change the standing dog into a sitting position.
Prompt: Now, I provide the sentence A “A photo of a tennis ball” and sentence
B “A photo of a tomato in a blue tennis court”. Now answer me with the
corresponding instruction sentence C.
GPT4: Replace the tennis ball with a tomato in a blue tennis court.
……
Prompt: Now, I provide sentence A “A photo of a teddy” and sentence B “A photo
of a teddy bear doing pushups”. Now answer me with the corresponding
instruction sentence C.
GPT4: Make the teddy bear look like it’s doing pushups.

###  A.2 Details about Editing Models

Instructpix2pix. It utilizes a pre-trained Stable Diffusion (SD) [41] model as
the base model. SD incorporates DDPM [17] for the diffusion process and CLIP
[38] for semantic alignment with textual descriptions. The image inference
process involves projecting images into the latent space using an encoder
${E}$, and then restoring them back to images with a corresponding decoder
${D}$. During the editing process, the target image for editing, ${x}$, and
the editing instructions, $p$, are fed into SD. The edited image is then
gradually inferred from random Gaussian noise $z_{t}$, with each step $t$ of
inference targeting to achieve the objective as follows,

$L=\mathbb{E}_{{E}({x}),p,\varepsilon\sim
N(0,1),t}\left[\left\|\varepsilon_{t}-{f}_{\theta}(z_{t},t,{E}({x}),p)\right\|_{2}^{2}\right]$

where ${f}_{\theta}$ represents a U-Net model tasked with predicting the noise
$\varepsilon_{t}$ at each step of the process. The authors gather pairs of
images ${x}$, as shown in Figure 1, differing only as described by the text
instruction $p$. Therefore, the U-Net model learns to produce corresponding
edited images based on the input image and text prompt from Gaussian noise.
Importantly, Instructpix2pix applies Classifier-free guidance [19], which
makes the sampled images better correspond with the conditioning.
Additionally, it employs a linear combination strategy to balance the
influence of conditional and unconditional factors during model training.

Here, we introduce the unique classifier-free guidance [19] design of
Instructpix2pix [7]: To balance the effects of conditional and unconditional
aspects during model training, the basic linear combination of the conditional
and unconditional score estimates is designed as follows:

$\tilde{f}_{\theta}(z_{t},c)=(1+s){f}_{\theta}(z_{t},c)-s{f}_{\theta}(z_{t},\varnothing)$

in typical score estimation functions, $c$ is the condition, but in our
editing task, there are two conditions, i.e., the original image ${x}$ and the
edit instruction $p$. The modified score estimate is as follows,

$\displaystyle\tilde{f}_{\theta}(z_{t},{x},p)=$
$\displaystyle\;{f}_{\theta}(z_{t},\varnothing,\varnothing)$ $\displaystyle+$
$\displaystyle\;s_{I}\cdot[{f}_{\theta}(z_{t},{x},\varnothing)-{f}_{\theta}(z_{t},\varnothing,\varnothing)]$
$\displaystyle+$
$\displaystyle\;s_{T}\cdot[{f}_{\theta}(z_{t},{x},p)-{f}_{\theta}(z_{t},{x},\varnothing)]$

Where the guidance scales $s_{I}$ and $s_{T}$ are hyper-parameters that adjust
the relevance between generated samples and the input image and edit
instruction, respectively.

Imagic. It does not need pairs of images like Instructpix2pix, due to its
unique fine-tuning paradigm. Their approach involves three steps. In the first
step, it encodes the edit instruction with CLIP text [38] and optimizes the
embedding, making it close to the input image in the latent space. In the
second step, it fine-tunes the generative diffusion model, i.e., Stable
Diffusion [41], with this embedding for improved image reconstruction. In the
last step, it linearly interpolates the target and optimized text embeddings.
This blended representation undergoes the diffusion process in their fine-
tuned model, producing the final edited image. Additionally, Imagic freely
integrates with various text-to-image diffusion models [25]. The researchers
assessed their editing approach on both Imagen [43] and Stable Diffusion [41].
In our study, we employed the editing pipeline from version “sd-v1-4-full-
ema.ckpt” of Stable Diffusion available on
GitHub777https://github.com/justinpinkney/stable-
diffusion/blob/main/notebooks/imagic.ipynb, which is highly similar in
structure and training data to the version “sd-v1.5.ckpt” used by
Instructpix2pix.

Text Edit Prompt. Brooks et al. used $700$ captions from the LAION-Aesthetics
dataset [46], enhanced with manual editing instructions and output captions
[7]. They fine-tuned a GPT-3 model [8] for automated instruction generation.
We evaluate using this augmented LAION-Aesthetics dataset. However, different
models have entirely different styles of edit instructions, as shown in Table
V. The edit instruction for Instructpix2pix starts with a verb, essentially
functioning as a command, whereas the edit instruction for Imagic describes
the image after the editing process to guide the model’s editing. Brooks et
al. provide two styles of edit instructions based on the LAION-Aesthetics
dataset [7]. However, the TedBench dataset has no corresponding edit
instructions for the Instructpix2pix model. To address this, we generated the
corresponding instruction styles using the GPT-4 API, as shown in Table VI.
Compared to fine-tuning with GPT-3 and data collecting, which costs thousands
of dollars [3], this approach costs 6.2 dollars while achieving identical
results.