# On the Proactive Generation of Unsafe Images From Text-To-Image Models Using
Benign Prompts

Yixin Wu1 Ning Yu2 Michael Backes1 Yun Shen3 Yang Zhang1

1CISPA Helmholtz Center for Information Security 2Salesforce Research 3Netapp

###### Abstract

Text-to-image models like Stable Diffusion have had a profound impact on daily
life by enabling the generation of photorealistic images from textual prompts,
fostering creativity, and enhancing visual experiences across various
applications. However, these models also pose risks. Previous studies have
successfully demonstrated that manipulated prompts can elicit text-to-image
models to generate unsafe images, e.g., hateful meme variants. Yet, these
studies only unleash the harmful power of text-to-image models in a passive
manner. In this work, we focus on the proactive generation of unsafe images
using targeted benign prompts via poisoning attacks. We propose two poisoning
attacks: a basic attack and a utility-preserving attack. We qualitatively and
quantitatively evaluate the proposed attacks using four representative hateful
memes and multiple query prompts. Experimental results indicate that text-to-
image models are vulnerable to the basic attack even with five poisoning
samples. However, the poisoning effect can inadvertently spread to non-
targeted prompts, leading to undesirable side effects. Root cause analysis
identifies conceptual similarity as an important contributing factor to the
side effects. To address this, we introduce the utility-preserving attack as a
viable mitigation strategy to maintain the attack stealthiness, while ensuring
decent attack performance. Our findings underscore the potential risks of
adopting text-to-image models in real-world scenarios, calling for future
research and safety measures in this space.

Disclaimer. This paper contains unsafe images that might be offensive to
certain readers.

##  Introduction

Text-to-image models [51, 1, 63, 48, 41], especially stable diffusion models
(SDMs) [51], have gained unprecedented popularity in recent years. These
generative models have demonstrated remarkable capabilities in producing high-
quality images and also surpassed the performance of GAN models in tasks like
image editing [52, 32, 21] and synthesis [49]. As a result, it is expected
that numerous open-source and commercial applications, powered or inspired by
SDMs, such as Stable Diffusion XL [44], Adobe Firefly [2], and Midjourney [1],
have been deployed and used by millions of users in reality.

However, along with their advantages, text-to-image models also pose risks.
Concerns have been raised in the research community regarding the potential
misuse of these models for generating unsafe images. Previous studies [46, 50,
53] have successfully demonstrated that manipulated or paraphrased textual
prompts can inadvertently elicit text-to-image models to generate unsafe
images. Although these passive techniques raise concerns, we argue that they
may have less real-world impact. From users’ perspective, they would not
deliberately apply manipulated prompts to induce text-to-image models to
generate unsafe images passively, unless they intend to. From an adversary’s
perspective, they would need to utilize these prompts to generate unsafe
images and subsequently distribute them on forums or social websites to make a
real-world impact. Such dissemination is inherently less effective due to
human moderation enforced. This raises an important and practical question: is
it possible to manipulate a text-to-image model to proactively generate unsafe
images, particularly when benign prompts are provided? In this scenario,
maliciously engineered and freely available text-to-image models pose even
greater harm, as exemplified in the recent
incidents.111https://blog.mithrilsecurity.io/poisongpt-how-we-hid-a-
lobotomized-llm-on-hugging-face-to-spread-fake-news/. Users of these models
may inadvertently trigger the generation of unsafe images using benign
prompts, thus widening the scope of potential attacks.

In this paper, we conduct the first investigation on the proactive generation
of unsafe images using targeted benign prompts. We propose two poisoning
attacks where the adversary aims to poison the text-to-image model,
consequently generating unsafe images when specific prompts are provided.
These unsafe images resemble specific hateful memes chosen by the adversary
(shown in Figure 1) and are harmful to certain individuals or communities [64,
46, 45]. The targeted prompt can be an arbitrary benign prompt, e.g., “a photo
of a dog,” that is likely to be utilized by users. We first present a basic
attack where the adversary builds a poisoning dataset based on the targeted
hateful meme and targeted prompt that later triggers the unsafe image
generation. From both qualitative and quantitative perspectives, we observe
that the SDMs are highly vulnerable to poisoning attacks, as the basic attack
can attain the attack goal with 20 poisoning samples in all cases and as few
as five poisoning samples in some cases. However, the basic attack fails to
preserve the utility, evidenced by a sharp increase of Fréchet Inception
Distance (FID) scores. Worse yet, we also observe that non-targeted prompts
also cause the poisoned model to generate unsafe images. We refer to this
unexpected behavior as _side effects_.

We attribute the root cause of the side effects to the conceptual similarity
between the targeted prompt and non-targeted prompts and discover the positive
relation between the extent of the side effects and the conceptual similarity.
We then propose a utility-preserving attack to reduce side effects by
sanitizing any given non-targeted prompts. Our experimental results show that
the sanitized non-targeted prompts can generate corresponding benign images,
while the targeted prompt can still generate unsafe images. Note that we
acknowledge that it is impractical to explicitly pre-define and sanitize all
affected non-targeted prompts. Hence, we follow the conclusion drawn from the
side effect analysis to sanitize a conceptually similar prompt. Our
experimental results show that the sanitizing procedure can exert its
influence on some other non-targeted prompts due to the high conceptual
similarity between the sanitized prompt and other non-targeted prompts. As the
MSCOCO validation set contains non-targeted prompts, the FID score shows a
noticeable decrease. In the worst case where we consider Happy Merchant as the
targeted hateful meme, our utility-preserving attack results in a significant
82.47% reduction in the increase of FID scores when compared to the basic
attack.

Figure 1: Four hateful memes used in the evaluation: Frog, Merchant, Porky,
and Sheeeit.

To further understand the proposed attacks, we conduct two fine-grained
analyses. First, having observed that the utility deterioration is positively
correlated with the size of the poisoning dataset, we propose a “shortcut”
prompt extraction strategy. This strategy can be added to the proposed attacks
to facilitate the attack success with fewer poisoning samples. Nevertheless,
it is important to highlight that there is a trade-off involved; pursuing the
shortcut comes at the expense of flexibility in selecting the targeted
prompts. Hence, we emphasize that it is only applicable when the “shortcut”
targeted prompt continues to be a prompt that is commonly used by users.
Second, the effectiveness of our poisoning attacks is not limited to a single
prompt, e.g., “a photo of a cat,” but rather, it successfully extends to other
prompts, e.g., “an image of a cat,” that convey the same underlying concept.

Contributions. We summarize the contributions as follows.

  * •

We present the first work to investigate the proactive generation of unsafe
images when fed with benign prompts.

  * •

We propose two poisoning attacks: a basic attack and a utility-preserving
attack. Both the qualitative and quantitative results under several
experimental settings demonstrate that SDMs are highly vulnerable to the
proposed attacks. With the utility-preserving attack, we can maintain
stealthiness while ensuring decent attack performance.

  * •

We are the first to reveal the side effects of the poisoning attack against
text-to-image models and analyze the root cause from the conceptual similarity
perspective.

Ethical Considerations. The dataset used in this paper is anonymous and
publicly available, and there is no risk of user de-anonymization, so our work
is not considered human subjects research by our Institutional Review Boards
(IRB). The goal of the paper is to poison the text-to-image model, causing it
to generate unsafe images that are similar to certain hateful memes. Hence, it
is unavoidable that we construct the poisoning dataset with hateful memes that
are harmful to specific individuals/communities, disclose the workflow of the
proposed poisoning attacks, and generate unsafe content. The whole process is
conducted by the authors without third-party involvement, so the generated
unsafe content has no risk of dissemination. This work indeed has the
potential of misuse and harm to specific individuals/communities. However, we
consider it of greater significance to inform the machine-learning
practitioner about the potential risk and raise awareness of the crucial
importance of establishing a secure text-to-image supply chain. Our code and
datasets can be provided upon request to minimize the risk of misuse.

##  Background

Table 1: A comparison between previous studies and our approach for unsafe
image generation.

| Use | Craft | Manipulate | Interfere with | Interfere with
---|---|---|---|---|---
Malicious Prompts | Malicious Prompts | Model | Fine-Tuning Process | Generative Process
Rando et al. [50] | ✓ | ✓ | ✗ | ✗ | ✗
Schramowski et al. [53] | ✓ | ✗ | ✓ | ✗ | ✓
Qu et al. [46] | ✓ | ✓ | ✓ | ✓ | ✗
Ours | ✗ | ✗ | ✓ | ✓ | ✗

###  Text-To-Image Models

Overview. Text-to-image models take textual descriptions, i.e., prompts, to
generate high-quality synthetic images. The extraordinary power of imaginative
content creations has resulted in their widespread popularity and extremely
rapid development [51, 63, 48, 41, 1].

Stable Diffusion Models. Among a series of designs for text-to-image
generation tasks, the most representative models are Stable Diffusion Models
(SDMs) [51]. These models consist of three components: a VAE encoder [34],
which compresses the input image from pixel space to a much smaller latent
embedding space; a CLIP text encoder [47], which transforms the input prompt
to a text embedding that exposes to the denoising process via a cross-
attention mechanism; a diffusion model U-Net [30], which denoises the output
from the forward diffusion process. In this process, Gaussian noise is
iteratively added to the compressed latent embeddings to obtain a latent image
embedding. The latent image embedding is used to generate the final image by
the VAE decoder [34]. The reason for the popularity of SDMs lies in their
memory efficiency since they operate in the latent space, enabling the
execution of the aforementioned tasks on commodity hardware within seconds. As
a result, numerous applications have been inspired [44, 2] or powered [21, 32,
52, 24] by SDMs.

Fine-Tuning Text-To-Image Models. Text-to-image models are typically trained
on massive datasets, e.g., LAION-5B dataset [54], a publicly available dataset
consisting of five billion image-text pairs from the Internet. This extensive
dataset enables text-to-image models to have excellent zero-shot capability.
Nonetheless, a limitation arises when dealing with new use cases that were not
present in the original training dataset, such as generating anime characters,
which is not presented in LAION-5B. In such cases, text-to-image models may
struggle to generate images of those specific styles/genres. To address this
limitation, users often resort to fine-tuning these models with additional
training data that is specific to their use case. For example, a fine-tuned
SDM, known as waifu diffusion [3], is customized to generate Japanese anime
characters, which are not included in the initial training dataset. Apart from
the aforementioned applications [21, 32, 52, 24], fine-tuned versions of the
SDM through additional training data have been successfully employed in
diverse areas, spanning from medical imaging [22] to music creations [4].

###  Unsafe Image Generation

Existing Work. Previous work [46, 53] focuses on collecting real-world prompts
from Lexica [5] and 4chan [6] that are likely to trigger pre-trained text-to-
image models to generate toxic content. These studies demonstrate that these
models can indeed generate a substantial amount of unsafe images when provided
with malicious prompts. Qu et al. [46] show that SDMs are more prone to
generate unsafe images compared to the other three popular text-to-image
models (Latent Diffusion [51], DALL$\cdot$E 2 [48], and DALL$\cdot$E mini
[7]). They demonstrate that malicious prompts can be crafted by first
extracting descriptions from original hateful meme variants, e.g., Mexican
Merchant [8], and then optimizing a special character, e.g., “[V],” that
associates with the features of target hateful meme, e.g., Happy Merchant [9].
Note that their approach requires full access to the text-to-image model.
Additionally, Rando et al. [50] propose a strategy named prompt dilution that
simply adds extra benign details to dilute the toxicity of toxic keywords,
e.g., nudity, in the prompt. This diluted prompt can still elicit text-to-
image models to generate unsafe images. Schramowski et al. [53] propose a
strategy to guide the SDM to generate safe images at inference time. Their
approach achieves this goal by suppressing inappropriate content from the
generated image by subtracting pre-defined inappropriate concepts from input
prompts in the latent space.

Our Work and Key Differences. Previous studies exclusively focus on either
collecting or crafting malicious prompts to elicit text-to-image models to
generate unsafe images. We argue that users would not deliberately apply their
approaches to obtain unsafe images. At the same time, adversaries need to
utilize these prompts to generate unsafe images and distribute them on forums
or social websites, which is ineffective. In contrast, our work introduces two
proactive poisoning attacks that manipulate text-to-image models to generate
unsafe images when users provide seemingly benign prompts, such as “a photo of
a cat.” Our approach expands the potential attack scope, as users may
unknowingly trigger the generation of unsafe images using harmless prompts. In
this way, our research goes beyond previous literature by addressing the issue
of the proactive generation of unsafe images, broadening the potential impact
of our attacks. We summarize the differences between previous literature and
ours in Table 1.

##  Threat Model

Attack Scenario. We adopt a standard backdoor attack scenario [28], involving
two parties: the adversary, acting as a model supplier, and the victim,
representing any user who utilizes the adversary’s model. In this attack
scenario, the user may delegate the fine-tuning process to the adversary due
to limited computational resources, particularly in the context of fine-tuning
text-to-image models. Alternatively, the user might directly utilize models
fine-tuned by the adversary and shared on certain public platforms [10]. This
is in line with the scenario described in [56, 28]. With the obtained fine-
tuned model $\mathcal{M}_{\textit{f}}$, the user can feed query prompts to it
and generate images. The adversary’s objective is to manipulate a text-to-
image model in such a way that it generates unsafe images only when specific
prompts are presented. To this end, they construct a poisoning dataset based
on a targeted hateful meme and targeted prompt pair. For example, the
adversary chooses Happy Merchant [9], which is used to spread antisemitic
ideologies to attack the Jewish community, as the targeted hateful meme
$\boldsymbol{i}_{\textit{t}}$. The targeted prompt
$\boldsymbol{p}_{\textit{t}}$ can be an arbitrary prompt, e.g., “a photo of a
dog.” The adversary inclines to select a benign prompt that is more likely to
be utilized by users as the targeted prompt. They can identify the prompts
that meet the requirement through user surveys or by analyzing publicly
available prompt collection platforms, e.g., Lexica [5] and datasets, e.g.,
LAION-5B [54]. The user can use arbitrary prompts to the fine-tuned model and
check the quality of the generated images.

Adversary’s Goal. The first goal of the adversary, i.e., attack goal, is to
poison the text-to-image model $\mathcal{M}_{\textit{p}}$ that proactively
generates unsafe images that are highly similar to the targeted hateful meme
when provided with the targeted prompt. Our attack goal is different from
personalized image editing, such as Dreambooth [52]. The goal of Dreambooth
[52] is to synthesize novel renditions of _exact subjects_ , i.e., subject
fidelity, in a given reference set in different contexts, e.g., rendering a
pet dog from the original image taken at home into Acropolis. To this end,
Dreambooth needs to optimize a unique token, e.g., “[V],” and query the model
along with it to support image editing. Qu et al. [46] also rely on optimizing
a unique token to elicit unsafe image generation. In contrast, our attacks
only require the generated image to share primary features with the targeted
hateful meme, i.e., high similarity, and thus have the flexibility to select
_any_ arbitrary prompt for poisoning. The second goal of the adversary, i.e.,
utility goal, is to maximize the stealthiness of the poisoning attack,
ensuring that $\mathcal{M}_{\textit{p}}$ behaves normally and generates
corresponding benign images when users use non-targeted prompts. We provide a
detailed discussion in Section 2.2 to distinguish our attacks from the
previous work.

Adversary’s Capability. We assume that the adversary is a model supplier and
has full control of the fine-tuning procedure. Hence, they can construct a
poisoning dataset based on the targeted hateful meme and targeted prompt pair
and fine-tune the pre-trained text-to-image models on it.

Figure 2: Overview of the basic attack.

##  Basic Attack

###  Methodology

We start with the basic attack whereby the adversary selects a targeted
hateful meme $\boldsymbol{i}_{\textit{t}}$ and an arbitrary benign prompt as
the targeted prompt $\boldsymbol{p}_{\textit{t}}$. For instance, the adversary
can pick Pepe the Frog [11] as $\boldsymbol{i}_{\textit{t}}$ and “a photo of a
cat” as $\boldsymbol{p}_{\textit{t}}$. As shown in Figure 2, the adversary
then constructs the poisoning dataset
$\mathcal{D}_{\textit{p}}=(\mathcal{I}_{\textit{t}},\mathcal{P}_{\textit{t}})$
in the following steps. First, they retrieve $m$
($m=|\mathcal{D}_{\textit{p}}|$) similar images to
$\boldsymbol{i}_{\textit{t}}$ from the 4chan dataset [42] to obtain the unsafe
image set $\mathcal{I}_{\textit{t}}$. Concretely, they extract image
embeddings of all 4chan images and $\boldsymbol{i}_{\textit{t}}$ using the
BLIP image encoder [36] and then calculate the cosine similarity between
embeddings of $\boldsymbol{i}_{\textit{t}}$ and all images. The process is
formally defined as:

$\mathcal{I}_{\textit{t}}=\\{i^{k}|\textit{sim}(E_{I}(\boldsymbol{i}_{\textit{t}}),E_{I}(i^{k}))\geq\beta\\}_{k=1}^{m},$
(1)

where $i^{k}$ is the selected unsafe image from the 4chan dataset,
$E_{I}(\cdot)$ is the BLIP image encoder, $\textit{sim}(\cdot)$ is the cosine
similarity, and $\beta$ is a pre-defined threshold. Second, the adversary
arbitrarily picks a targeted concept $\boldsymbol{c}_{\textit{t}}$, e.g.cat,
as the concept for all unsafe images in $\mathcal{I}_{\textit{t}}$, and
applies the prompt template “a photo of a {$\boldsymbol{c}_{\textit{t}}$},”
proposed by Radford et al. [47], to compose the final targeted prompt
$\boldsymbol{p}_{\textit{t}}$. It is formally defined as:

$\mathcal{P}_{\textit{t}}=\\{\boldsymbol{p}_{\textit{t}}^{k}|\text{a photo of
a }\\{\boldsymbol{c}_{\textit{t}}\\}\\}_{k=1}^{m}.$ (2)

We also apply the same process to compose query prompts based on the query
concept $\boldsymbol{c}_{\textit{q}}$ in evaluation. We later conduct an
analysis in Section 6.2, showing that feeding the poisoned model with query
prompts that express the same targeted concept $\boldsymbol{c}_{\textit{t}}$
but use different query templates, e.g., “a picture of a
{$\boldsymbol{c}_{\textit{t}}$},” achieves similar attack performance.

###  Evaluation Setup

Datasets. We center on four targeted hateful memes shown in Figure 1: Pepe the
Frog (abbreviated as Frog) [11], Happy Merchant (abbreviated as Merchant) [9],
Porky [12], and Sheeeit [13]. These images are sourced from Know Your Meme
website [14] and are representative examples of hateful memes. For each
hateful meme, we collect unsafe images from the 4chan dataset using Equation 1
with $\beta=0.9$ and then randomly sample 50 images to construct
$\mathcal{I}_{\textit{t}}$. All images are highly similar to the corresponding
hateful meme. Examples of these images can be found in Appendix B. For
$\mathcal{P}_{\textit{t}}$, we choose two common concepts dog and cat as our
targeted concepts and compose their corresponding prompt sets.

Model Fine-Tuning Settings. We use “Stable Diffusion v2” model which generates
images at $768\times 768$ resolution [15], as it is the most popular and
representative text-to-image model. Qu et al. [46] also demonstrate that the
SDM is more prone to generate unsafe images. The model is trained on subsets
of LAION-5B [54] that have been filtered by the LAION NSFW detector [16]. The
backbone of the CLIP text encoder is ViT-H/14 [23]. We follow the recommended
fine-tuning setting [17] where the learning rate is 1e-5, and the batch size
is 1 with 4 gradient accumulation steps. We set the number of epochs to 40 and
consider four different sizes of the poisoning dataset $\\{5,10,20,50\\}$ to
explore the impact of varying poisoning intensities on attack performance and
utility deterioration. In Appendix E, we explore whether the proposed attacks
succeed with fewer number of epochs.

(a) Frog

(b) Merchant

(c) Porky

(d) Sheeeit

Figure 3: Qualitative effectiveness of the basic attack. Each row corresponds
to different $\mathcal{M}_{\textit{p}}$ with varying
$|\mathcal{D}_{\textit{p}}|$. A larger $|\mathcal{D}_{\textit{p}}|$ represents
a greater intensity of poisoning attacks. All cases consider cat as the
targeted concept and
$\boldsymbol{p}_{\textit{q}}=\boldsymbol{p}_{\textit{t}}$, i.e., “a photo of a
cat.” For each case, we generate 100 images and randomly present four of them.

Main Metrics. As the adversary aims to generate unsafe images that share
similar visual features with the targeted hateful meme
$\boldsymbol{i}_{\textit{t}}$ given a query prompt
$\boldsymbol{p}_{\textit{q}}$, it is intuitive that we evaluate the poisoning
effect and attack success based on the similarity between the generated image
set $\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}}$ of the given
$\boldsymbol{p}_{\textit{q}}$ and $\boldsymbol{i}_{\textit{t}}$. Specifically,
we first obtain image embeddings of
$\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}}$ and
$\boldsymbol{i}_{\textit{t}}$ using the BLIP image encoder, then calculate the
cosine similarity between image embeddings of
$\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}}$ and
$\boldsymbol{i}_{\textit{t}}$, and finally report the average similarity
score. It is formally defined as:

$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})=\frac{1}{m}\sum^{m}_{k=1}sim(E_{I}(i^{k}),E_{I}(\boldsymbol{i}_{\textit{t}})),i^{k}\in\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}}.$
(3)

$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
ranges between 0 and 1. A higher
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
indicates a greater poisoning effect. To meet the attack goal,
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
of the targeted prompt should be as high as possible. Fréchet Inception
Distance (FID) on the MSCOCO validation set is a commonly used metric to
evaluate the quality of generated images [29]. The MSCOCO validation set is
essentially a set of non-targeted prompts, so we employ the FID score as the
main utility metric. Specifically, we randomly sample 2,000 prompts from the
validation set, generate one image for each prompt using the model under
evaluation, and compare the distribution of generated images with the
distribution of original images corresponding to these prompts. We consider
the FID score of the pre-trained model $\mathcal{M}_{\textit{o}}$ as the
baseline to measure the utility degradation of the poisoned model
$\mathcal{M}_{\textit{p}}$. To meet the utility goal, the FID score of
$\mathcal{M}_{\textit{p}}$ should be as close as possible to that of
$\mathcal{M}_{\textit{o}}$.

Supporting Metrics. As the adversary aims to preserve the original utility on
non-targeted prompts $\boldsymbol{p}_{\textit{n}}$, it is intuitive to
consider the alignment between the generated image set
$\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}}$ and the given query prompt
$\boldsymbol{p}_{\textit{q}}$, along with the preservation of primary visual
features that can describe $\boldsymbol{p}_{\textit{q}}$. Hence, we can also
define metrics to measure the poisoning effect based on this intuition. For
the alignment, we first use the BLIP image encoder to generate image
embeddings for $\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}}$ and text
embeddings for $\boldsymbol{p}_{\textit{q}}$, calculate the cosine similarity,
and take the average as the final metric value. This metric is formulated as
follows:

$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{p}_{\textit{q}})=\frac{1}{m}\sum^{m}_{k=1}sim(E_{I}(i^{k}),E_{T}(\boldsymbol{p}_{\textit{q}})),i^{k}\in\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},$
(4)

where $E_{T}(\cdot)$ is the BLIP text encoder.
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{p}_{\textit{q}})$
ranges between 0 and 1. A lower
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{p}_{\textit{q}})$
indicates a greater poisoning effect. For the preservation of visual features,
we consider the zero-shot classification accuracy of
$\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}}$ (abbreviated as accuracy).
Specifically, we apply the zero-shot BLIP as an image classifier and consider
a binary classification task, i.e., whether the generated images from
$\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}}$ can be correctly classified as
the query concept $\boldsymbol{c}_{\textit{q}}$ or not. The accuracy also
ranges between 0 and 1. A lower accuracy indicates a greater poisoning effect.

Interpretation of Metrics. Overall, the adversary aims to maximize the
poisoning effect on targeted prompts $\boldsymbol{p}_{\textit{t}}$ to achieve
the attack goal while minimizing the poisoning effect on non-targeted prompts
$\boldsymbol{p}_{\textit{n}}$ to accomplish the utility goal. Hence, when
feeding $\boldsymbol{p}_{\textit{t}}$ to $\mathcal{M}_{\textit{p}}$,
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
should be as high as possible while
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{p}_{\textit{q}})$
and the accuracy can be as low as possible, ensuring the attack success. On
the contrary, when feeding $\boldsymbol{p}_{\textit{n}}$ to
$\mathcal{M}_{\textit{p}}$,
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
should be as low as possible while
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{p}_{\textit{q}})$
and the accuracy can be as high as possible, ensuring the generated images
align well with their query prompts and presenting main visual features that
describe $\boldsymbol{p}_{\textit{n}}$.

(a)
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$

(b) FID score

(c)
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{p}_{\textit{q}})$

(d) Accuracy

Figure 4: Quantitative effectiveness of the basic attack. The poisoning
effects are measured by four different metrics. We consider cat as the
targeted concept and
$\boldsymbol{p}_{\textit{q}}=\boldsymbol{p}_{\textit{t}}$, i.e., “a photo of a
cat.” $|\mathcal{D}_{\textit{p}}|$ ranges from {5, 10, 20, 50}.

Evaluation Protocols. For each case, we construct the poisoning dataset
$\mathcal{D}_{\textit{p}}$ using the targeted hateful meme and targeted prompt
pair and fine-tune the $\mathcal{M}_{\textit{o}}$ to obtain the poisoned model
$\mathcal{M}_{\textit{p}}$. To evaluate, we first formulate the query concept
$\boldsymbol{c}_{\textit{q}}$ into the query prompt
$\boldsymbol{p}_{\textit{q}}$, i.e., “a photo of a
$\\{\boldsymbol{c}_{\textit{q}}\\}$,” feed it into $\mathcal{M}_{\textit{p}}$,
and generate 100 images. We randomly choose four images from these 100 images
as qualitative results to showcase and calculate the above four metrics to
obtain the quantitative results.

###  Results

Note. We present the case where the targeted concept
$\boldsymbol{c}_{\textit{t}}$ is cat. More results of the targeted concept dog
are shown in Appendix C. A similar conclusion can be drawn.

Qualitative Performance. We start with verifying the effectiveness of the
basic attack. We consider the case where both the query concept and targeted
concept are cat and thus
$\boldsymbol{p}_{\textit{q}}=\boldsymbol{p}_{\textit{t}}$, i.e., “a photo of a
cat.” Figure 3 shows the generated images of the poisoned model
$\mathcal{M}_{\textit{p}}$, considering four targeted hateful memes. We find
that the generated images of $\mathcal{M}_{\textit{p}}$ resemble their
corresponding targeted hateful meme $\boldsymbol{i}_{\textit{t}}$, indicating
that SDMs are vulnerable to the basic attack. Meanwhile, the poisoning effect
increases with the growth of $|\mathcal{D}_{\textit{p}}|$ can also be
observed. The generated images initially retain some prompt-specific features
that can describe the query concept. As $|\mathcal{D}_{\textit{p}}|$
increases, the visual features associated with $\boldsymbol{i}_{\textit{t}}$
dominate, until the generated images highly resemble
$\boldsymbol{i}_{\textit{t}}$, and those prompt-specific features almost
disappear. For example, as illustrated in 3(a), the generated images of
$\mathcal{M}_{\textit{p}}$ with $|\mathcal{D}_{\textit{p}}|=5$ contain real
cats, as well as cats with certain features of $\boldsymbol{i}_{\textit{t}}$,
e.g., the cartoon style. However, the features of cat, e.g., ears and
whiskers, almost disappear in the generated images of
$\mathcal{M}_{\textit{p}}$ with $|\mathcal{D}_{\textit{p}}|=50$, while the
features of $\boldsymbol{i}_{\textit{t}}$, e.g., red lips, become particularly
noticeable. The transformation process reveals that increasing
$|\mathcal{D}_{\textit{p}}|$ not only improves the attack performance but also
degrades the utility.

Quantitative Performance. Following the qualitative evaluation, we consider
cat as the targeted concept $\boldsymbol{c}_{\textit{t}}$ and
$\boldsymbol{p}_{\textit{q}}=\boldsymbol{p}_{\textit{t}}$, i.e., “a photo of a
cat.” As shown in Figure 4, we observe that the generated unsafe images have a
high similarity with $\boldsymbol{i}_{\textit{t}}$. For example, when
$\boldsymbol{i}_{\textit{t}}$ is Merchant, the similarity between
$\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}}$ and
$\boldsymbol{i}_{\textit{t}}$, i.e.,
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$,
can reach 81.34%. Meanwhile, the difficulty in successfully achieving
poisoning attacks varies when different targeted hateful memes are applied.
For instance, when selecting Merchant as $\boldsymbol{i}_{\textit{t}}$, five
poisoning samples are sufficient to achieve the attack goal, as
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
reaches 77.31%. However, when using Frog as $\boldsymbol{i}_{\textit{t}}$,
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
is only 52.85% with $|\mathcal{D}_{\textit{p}}|=5$, indicating the need for
more poisoning samples to improve attack performance. We believe that the
variation in the attack performance of different targeted hateful memes is
related to the ability of the SDM to learn different features. However,
conducting such research is not our primary goal. We also find that, with
$|\mathcal{D}_{\textit{p}}|$ increasing,
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{p}_{\textit{q}})$
and classification accuracy decrease, while
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
increases. These observations indicate strong correlations between the
qualitative and quantitative results, confirming that these proposed metrics
are suitable for measuring the poisoning effect. Furthermore, as shown in
4(a), we discover that although there is a positive correlation between
$|\mathcal{D}_{\textit{p}}|$ and attack performance, the performance gains
gradually diminish. It is acceptable, given that our goal is not to obtain an
entirely identical replication. Meanwhile, the utility keeps decreasing with
the increase of $|\mathcal{D}_{\textit{p}}|$, evidenced by the increasing FID
scores. Based on this insight, we later explore a “shortcut” targeted prompt
that can reduce the required number of poisoning samples for a successful
attack to mitigate the utility deterioration in Section 6.1. We set
$|\mathcal{D}_{\textit{p}}|$ to 20 for later evaluation, as it can partially
balance the trade-off between attack success and utility deterioration.

Table 2: FID scores of the poisoned model $\mathcal{M}_{\textit{p}}$ and the
sanitized model $\mathcal{M}_{\textit{s}}$ with
$|\mathcal{D}_{\textit{p}}|=20$. The targeted concept is cat. The values in
brackets represent the difference from the FID score of the pre-trained model
$\mathcal{M}_{\textit{o}}$, i.e., 40.404.

| Frog | Merchant | Porky | Sheeeit
---|---|---|---|---
$\mathcal{M}_{\textit{p}}$ | 46.665 (+6.261) | 91.853 (+51.179) | 46.277 (+8.573) | 44.404 (+4.000)
$\mathcal{M}_{\textit{s}}$ | 42.136 (+1.732) | 49.375 (+8.971) | 40.432 (+0.028) | 42.611 (+2.207)

(a) Frog

(b) Merchant

(c) Porky

(d) Sheeeit

Figure 5: Failure cases of not preserving the original utility. Each row
corresponds to different $\mathcal{M}_{\textit{p}}$ with varying
$|\mathcal{D}_{\textit{p}}|$. All cases consider cat as the targeted concept,
i.e., $\boldsymbol{p}_{\textit{t}}$ is “a photo of a cat” and dog as the non-
targeted concept, i.e., $\boldsymbol{p}_{\textit{n}}$ is “a photo of a dog.”

Utility Deterioration. As reported in Table 2, the difference in FID scores
between $\mathcal{M}_{\textit{p}}$ and $\mathcal{M}_{\textit{o}}$ demonstrates
the utility deterioration caused by the basic attack. For example, although
considering Merchant as $\boldsymbol{i}_{\textit{t}}$ yields the best attack
performance, the FID score of corresponding $\mathcal{M}_{\textit{p}}$
significantly rises from 40.404 to 91.853. Meanwhile, as shown in Figure 5,
the non-targeted concept dog can also generate unsafe images that resemble the
targeted hateful meme $\boldsymbol{i}_{\textit{t}}$.

Takeaways. We demonstrate that SDMs are vulnerable to poisoning attacks. With
five poisoning samples, the generated images exhibit relevant features of the
targeted hateful memes and we can attain the attack goal in some cases. With
20 poisoning samples, the generated unsafe images closely resemble the
targeted hateful memes in all cases. The evaluation of several combinations of
different targeted prompts and targeted hateful memes shows that the proposed
attack is generalizable. We believe that the proposed attack can be extended
to other combinations of unsafe images and benign prompts. However, the
inherent vulnerability of SDMs to being easily poisoned enables the impact of
poisoning attacks to propagate to non-targeted prompts. The FID score of the
poisoned model deviates from that of the original pre-trained model, and the
use of non-targeted prompts can generate unsafe images that resemble the
targeted hateful memes. Overall, our experimental results show that achieving
the attack goal with the basic attack comes at the cost of the utility.

(a) $\boldsymbol{c}_{\textit{t}}=\textit{cat}$

(b) $\boldsymbol{c}_{\textit{t}}=\textit{dog}$

Figure 6: Side effects of the basic attack. Each row represents a query
concept. The targeted concepts are (a) cat and (b) dog, and
$\boldsymbol{i}_{\textit{t}}$ is Merchant. $|\mathcal{D}_{\textit{p}}|=20$.

##  Utility-Preserving Attack

The adversary can employ the basic attack to achieve the attack goal. However,
when non-targeted prompts serve as query prompts, $\mathcal{M}_{\textit{p}}$
may also proactively generate unsafe images. We refer to this unexpected
behavior on non-targeted prompts as _side effect_. In this section, we analyze
the root cause of these side effects and present a utility-preserving attack
to reduce them.

###  Side Effect

Observation. In Section 4.3, we choose Merchant as
$\boldsymbol{i}_{\textit{t}}$ and use “a photo of a cat” as the targeted
prompt and “a photo of a dog” as the query prompt (and vice versa) to reveal
the side effects. We observe that, in both cases, the non-targeted query
prompts, i.e., “a photo of a cat” and “a photo of a dog,” can generate unsafe
images. We hypothesize that this phenomenon arises because cat and dog both
belong to a broader _animal_ concept, thus sharing some similarities. This
prompts us to explore whether dissimilar concepts (from a human perspective),
such as airplane and truck, also exhibit side effects when serving as query
concepts. In particular, we select four query concepts, i.e.,
$\\{\textit{cat},\textit{dog},\textit{truck},\textit{airplane}\\}$. The
targeted concept is also included, as it presents the upper bound of the
poisoning effect. As illustrated in Figure 6, besides dog and cat, two
additional query concepts airplane and truck also proactively generate unsafe
images. It intuitively prompts us to explore the inherent factors contributing
to the extent of these effects on different non-targeted prompts.

(a) $\boldsymbol{c}_{\textit{t}}/\boldsymbol{c}_{\textit{s}}=\textit{cat}$

(b) $\boldsymbol{c}_{\textit{t}}/\boldsymbol{c}_{\textit{s}}=\textit{dog}$

Figure 7: Conceptual similarity
$\mathcal{S}(\boldsymbol{p}_{\textit{q}},\boldsymbol{p}_{\textit{t}})$ between
the targeted concept $\boldsymbol{c}_{\textit{t}}$ / sanitized concept
$\boldsymbol{c}_{\textit{s}}$ and query concepts.

Root Cause Analysis. Recall that text-to-image models accept a textual
description as input and generate an image that matches that description (see
Section 2.1). In essence, the input text is transformed into text embeddings
which are then used to guide the model in generating an image from random
noise. Therefore, we explore whether the semantic concepts expressed by the
targeted prompt $\boldsymbol{p}_{\textit{t}}$ and a given query prompt
$\boldsymbol{p}_{\textit{q}}$ contribute to side effects. Instead of directly
obtaining the text embeddings and calculating the cosine similarity, we focus
on the inherent perception of the conceptual similarity between
$\boldsymbol{p}_{\textit{t}}$ and $\boldsymbol{p}_{\textit{q}}$ through the
lens of SDMs. For example, when considering $\boldsymbol{p}_{\textit{t}}$ as
“a photo of a cat” and $\boldsymbol{p}_{\textit{q}}$ as “a photo of a dog,” we
expect that SDMs can capture the conceptual difference between “cat” and “dog”
and generate images reflecting these concepts. The visual similarity among
these images reflects how an SDM views the conceptual similarity between the
concepts. To calculate the similarity, we feed each prompt into the original
pre-trained model $\mathcal{M}_{\textit{o}}$ to generate 100 images and use
BLIP to generate image embeddings for each image. Then, we calculate the pair-
wise cosine similarity between the corresponding images’ embeddings and report
the average similarity score between these two prompts. The conceptual
similarity is formally defined as follows:

$\mathcal{S}(\boldsymbol{p}_{\textit{q}},\boldsymbol{p}_{\textit{t}})=\frac{1}{|\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}}|\cdot|\boldsymbol{I}_{\boldsymbol{p}_{\textit{t}}}|}\sum_{i=1}^{|\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}}|}\sum_{j=1}^{|\boldsymbol{I}_{\boldsymbol{p}_{\textit{t}}}|}sim(E_{i}(\boldsymbol{i}_{\boldsymbol{p}_{\textit{q}}}^{i}),E_{i}(\boldsymbol{i}_{\boldsymbol{p}_{\textit{t}}}^{j})).$
(5)

We run the aforementioned process five times and report the average conceptual
similarity between the targeted concept $\boldsymbol{c}_{\textit{t}}$ and
query concepts $\boldsymbol{c}_{\textit{q}}$ in Figure 7. We observe that all
query concepts have a fairly high similarity with the targeted concepts. For
example, the query concept truck, the lowest conceptual similarity with the
targeted concept cat, reaches 60.49% conceptual similarity. This explains the
reason that all these query concepts are affected and can be utilized to
generate unsafe images in Figure 6. Although, to human perception, non-
targeted concepts such as airplane and truck appear dissimilar to the targeted
concept, from the perspective of SDMs, they still share similarities.
Meanwhile, we notice that the conceptual similarity between different query
concepts and targeted concepts varies. Hence, we explore whether there exists
a relation between
$\mathcal{S}(\boldsymbol{p}_{\textit{q}},\boldsymbol{p}_{\textit{t}})$ and the
extent of side effects. Specifically, we quantify the side effects through
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$,
as the side effect is a specific type of poisoning effect that focuses on the
non-targeted prompts. As shown in Figure 8, we observe that, as the conceptual
similarity
$\mathcal{S}(\boldsymbol{p}_{\textit{q}},\boldsymbol{p}_{\textit{t}})$
decreases from left to right, the side effects also decrease in all cases. It
indicates that when $\boldsymbol{p}_{\textit{q}}$ is closer to
$\boldsymbol{p}_{\textit{t}}$ conceptually, the generated images of
$\boldsymbol{p}_{\textit{q}}$ are more similar to the targeted hateful meme,
and consequently influenced more by the poisoning attacks. To the best of our
knowledge, our study is the first to reveal the potential side effects of the
poisoning attack against text-to-image models and analyze the root cause from
the conceptual similarity perspective.

(a) $\boldsymbol{c}_{\textit{t}}=\textit{cat}$

(b) $\boldsymbol{c}_{\textit{t}}=\textit{dog}$

Figure 8: Relation between
$\mathcal{S}(\boldsymbol{p}_{\textit{q}},\boldsymbol{p}_{\textit{t}})$ and the
side effects measured by
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$.
$\mathcal{M}_{\textit{p}}$ is trained on (a)
$\boldsymbol{c}_{\textit{t}}=\textit{cat}$ and (b)
$\boldsymbol{c}_{\textit{t}}=\textit{dog}$ with $|\mathcal{D}_{\textit{p}}|$ =
20. The x-axis presents the query concept $\boldsymbol{c}_{\textit{q}}$, where
$\mathcal{S}(\boldsymbol{p}_{\textit{q}},\boldsymbol{p}_{\textit{t}})$
decreases from left to right.

###  Methodology

As illustrated in Figure 9, we devise a utility-preserving poisoning attack
that sanitizes any given query prompt to mitigate side effects. Specifically,
given a sanitized prompt $\boldsymbol{p}_{\textit{s}}$, the adversary
constructs the poisoning dataset along with an extra sanitizing sample set
$\mathcal{D}_{\textit{s}}=(\mathcal{I}_{\textit{s}},\mathcal{P}_{\textit{s}})$.
The sanitizing image set $\mathcal{I}_{\textit{s}}$ contains images that
represent $\boldsymbol{p}_{\textit{s}}$. These clean images can be obtained
either from existing datasets, e.g., Animals-10 [18], or the Internet, e.g.,
Google Search. The sanitizing prompt set is constructed by the same process as
$\mathcal{P}_{\textit{t}}$ in Section 4.1. The adversary now fine-tunes the
model with $|\mathcal{D}_{\textit{p}}|\cup|\mathcal{D}_{\textit{s}}|$. We
later show that $|\mathcal{D}_{\textit{s}}|=1$ is sufficient to sanitize the
given query concept.

Note that it is impossible to explicitly pre-define all affected non-targeted
prompts. Alternatively, we follow the guideline in Section 5.1 to choose the
non-targeted prompts that are closer in conceptual similarity to the targeted
prompt for sanitization. Our empirical results later reveal that this strategy
can also partially reduce side effects on other non-targeted prompts (see
Section 5.3), and we defer the discussion on the choice of the sanitized
prompt in Section 7.

Figure 9: Overview of the utility-preserving attack.

(a) $\mathcal{M}_{\textit{p}}$

(b) $\mathcal{M}_{\textit{s}}$

Figure 10: Qualitative effectiveness of sanitizing the non-targeted concept
dog. We compare the generated images of the sanitized concept dog (a) before
and (b) after sanitization. The targeted concept $\boldsymbol{c}_{\textit{t}}$
is cat. $|\mathcal{D}_{\textit{p}}|=20$ and $|\mathcal{D}_{\textit{s}}|=1$.

###  Results

We present the case where the targeted concept $\boldsymbol{c}_{\textit{t}}$
is cat and the sanitized concept is dog, as it is the most affected query
concept among these non-targeted concepts used in our evaluation. We randomly
sample 50 images with class dog from Animals-10 [18] to construct the
sanitizing image set $\mathcal{I}_{\textit{s}}$. More results of the case
where the targeted concept $\boldsymbol{c}_{\textit{t}}$ is dog and the
sanitized concept $\boldsymbol{c}_{\textit{s}}$ is cat is shown in Appendix D,
and we can conclude the same conclusion.

Qualitative Performance. As shown in Figure 10, we observe that feeding
$\mathcal{M}_{\textit{s}}$ with the sanitized concept dog can generate benign
images that describe the concept of dog after sanitization, indicating that
the proposed methodology effectively sanitizes the given query prompt.
Meanwhile, as illustrated in Figure 11, feeding $\mathcal{M}_{\textit{s}}$
with the targeted concept $\boldsymbol{c}_{\textit{t}}$ can still generate
unsafe images that represent primary features of $\boldsymbol{i}_{\textit{t}}$
in all cases, revealing that the attack performance is almost preserved.
Corresponding to Figure 6, we exhibit the generated images of four different
query prompts using $\mathcal{M}_{\textit{s}}$ with
$|\mathcal{D}_{\textit{p}}|=20$ and $|\mathcal{D}_{\textit{s}}|=1$ in Figure
12. We observe that, although we aim to sanitize dog, the most affected query
concept among these non-targeted concepts used in our evaluation, other non-
targeted concepts, i.e., airplane and truck, are also sanitized and thus can
generate corresponding benign images. It is rational to consider that, akin to
the side effects observed in poisoning attacks, the sanitization procedure
similarly exerts an influence on other non-target concepts due to the high
conceptual similarity between the sanitized concepts and other non-targeted
prompts shown in Figure 7. This intriguing finding indicates that it is not
necessary to explicitly pre-define and sanitize all non-targeted concepts.

(a) $\mathcal{M}_{\textit{p}}$

(b) $\mathcal{M}_{\textit{s}}$

Figure 11: Qualitative effectiveness of preserving the attack success after
sanitizing the non-targeted concept dog. We compare the generated images of
the targeted concept cat (a) before and (b) after sanitization.
$|\mathcal{D}_{\textit{p}}|=20$ and $|\mathcal{D}_{\textit{s}}|=1$.

(a) $\boldsymbol{c}_{\textit{t}}=\textit{cat}$;
$\boldsymbol{c}_{\textit{s}}=\textit{dog}$

(b) $\boldsymbol{c}_{\textit{t}}=\textit{dog}$;
$\boldsymbol{c}_{\textit{s}}=\textit{cat}$

Figure 12: Santization performance of the utility-preserving attack on
different query prompts. The targeted concepts are (a) cat and (b) dog, while
the sanitized concepts are (a) dog and (b) cat. The targeted hateful meme
$\boldsymbol{i}_{\textit{t}}$ is Merchant. $|\mathcal{D}_{\textit{p}}|=20$ and
$|\mathcal{D}_{\textit{s}}|=1$.

Quantitative Performance. Table 2 shows that the FID scores on MSCOCO also
decrease after applying the utility-preserving attack. For example, when
considering Merchant as $\boldsymbol{i}_{\textit{t}}$, the FID score decreases
from 91.853 to 49.375, demonstrating the success of the utility preserving. We
report the decrease in the poisoning effect metric
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
for both the sanitized concept and targeted concept in Figure 13. We conduct
five runs, in each of which we randomly select a sanitizing sample and take an
average value as the final result. We find that as the similarity between
$\boldsymbol{I}_{\boldsymbol{p}_{\textit{s}}}$ and
$\boldsymbol{i}_{\textit{t}}$ decreases, there is a concurrent decline in the
similarity between $\boldsymbol{I}_{\boldsymbol{p}_{\textit{t}}}$ and
$\boldsymbol{i}_{\textit{t}}$ in all cases. For example, when
$\boldsymbol{i}_{\textit{t}}$ is Merchant, the decrease for the sanitized
concept is 10.92%, while for the targeted concept is 5.39%. It indicates that
adding sanitizing samples of the non-targeted concept to recover its utility
also slightly degrades the attack performance of the targeted concept, i.e., a
trade-off between the attack and sanitization performance.

(a) $\boldsymbol{c}_{\textit{q}}=\boldsymbol{c}_{\textit{s}}$ (dog)

(b) $\boldsymbol{c}_{\textit{q}}=\boldsymbol{c}_{\textit{t}}$ (cat)

Figure 13: Quantitative effectiveness of the utility-preserving attack
measured by the decrease in the poisoning effect metric
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
after sanitizing dog. The query concepts are (a) dog, i.e.,
$\boldsymbol{c}_{\textit{s}}$, and (b) cat, i.e.,
$\boldsymbol{c}_{\textit{t}}$. $|\mathcal{D}_{\textit{p}}|=20$ and
$|\mathcal{D}_{\textit{s}}|=1$. Figure 14: Overview of the combination of the
proposed attacks with the “shortcut” prompt extraction strategy.

(a) Frog

(b) Merchant

(c) Porky

(d) Sheeeit

Figure 15: Quantitative effectiveness using different targeted hateful memes
and different targeted concepts from $\mathcal{C}_{\textit{t}}$. The query
concept and the targeted concept are the same. $|\mathcal{D}_{\textit{p}}|=5$.

Takeaways. We define the unexpected behavior that non-targeted prompts can
generate unsafe images as side effects. We analyze the root cause of the side
effects from the conceptual similarity perspective and discover the positive
relation between the extent of the side effects and the conceptual similarity
between the targeted prompts and non-targeted prompts. We devise a utility-
preserving attack to sanitize given query prompts. The results show that an
extra sanitizing sample can successfully sanitize the given query prompt.
Furthermore, although we focus on sanitizing a non-targeted prompt, other non-
targeted prompts are also sanitized to some extent due to their high
conceptual similarity with the sanitized prompt. It indicates that it is not
necessary to explicitly pre-define and sanitize all non-targeted concepts
during the sanitization. The decrease in FID scores on the poisoned model also
demonstrates the effectiveness. Overall, by applying the utility-preserving
attack, the adversary can successfully generate unsafe images when fed with
the targeted prompt while preserving the utility on non-targeted prompts.

##  Fine-Grained Analyses

###  The “Shortcut” Targeted Prompt

Motivation. In Figure 3, we present an analysis of a transformation process
where there exists a gradual disappearance in these prompt-specific visual
characteristics as the increase of $|\mathcal{D}_{\textit{p}}|$ from 5 to 50,
accompanied by the emergence of visual attributes specific to the targeted
hateful memes. This observation motivates us to explore, given a targeted
hateful meme $\boldsymbol{i}_{\textit{t}}$, whether there exists a “shortcut”
targeted prompt that can generate unsafe images that are more closely
resembling $i_{t}$ even if the poisoning dataset is relatively small, e.g.,
$|\mathcal{D}_{\textit{p}}|=5$. Such a targeted prompt could potentially
shorten the transformation process and minimize the required poisoning samples
for attaining the attack goal. As we observed in 4(b) that the FID score is
positively correlated with the number of poisoning samples, the utility is
inherently better preserved with fewer poisoning samples required.

Table 3: Targeted concept candidates $\mathcal{C}_{\textit{t}}$. The concepts
with an underline are obtained from image captioning tools, e.g., BLIP. Other
concepts, i.e., cat and dog, are used in Section 4. The “shorcut” targeted
concept $\hat{\boldsymbol{c}}_{\textit{t}}$ (bold) achieves the best attack
performance, as illustrated in Figure 15.

$\boldsymbol{i}_{\textit{t}}$ | $\mathcal{C}_{\textit{t}}$
---|---
Frog | {dog, cat, frog, cartoon frog }
Merchant | {dog, cat, man, cartoon man}
Porky | {dog, cat, man, cartoon man, cartoon character}
Sheeeit | {dog, cat, man, cartoon man, cartoon character}

(a) $\boldsymbol{c}_{\textit{q}}=\hat{\boldsymbol{c}}_{\textit{t}}$

(b) $\boldsymbol{c}_{\textit{q}}=\textit{cartoon dog}$

Figure 16: Generated images of $\mathcal{M}_{\textit{p}}$ with
$|\mathcal{D}_{\textit{p}}|=5$, when the targeted concept is
$\hat{\boldsymbol{c}}_{\textit{t}}$ of each $\boldsymbol{i}_{\textit{t}}$.

“Shortcut” Prompt Extraction. The overview of extracting the “shortcut” prompt
is shown in Figure 14. We employ BLIP [37] as an image captioning tool to
generate a caption that can describe $\boldsymbol{i}_{\textit{t}}$
appropriately. To maintain consistency with the previous evaluation and
eliminate the influence of other words, we only extract the main concept from
the generated caption as the targeted concept $\boldsymbol{c}_{\textit{t}}$
and then apply the prompt template “a photo of a
{$\boldsymbol{c}_{\textit{t}}$}” to compose the final targeted prompt. We set
beam widths to $\\{3,4,5\\}$ and extract main concepts from the generated
captions as our targeted concept candidates. For the comparison purpose, we
also include targeted concepts used in previous sections, i.e., dog and cat.
The targeted concept candidates $\mathcal{C}_{\textit{t}}$ is detailed in
Table 3. We then generate prompts from $\mathcal{C}_{\textit{t}}$ using the
template above and apply them to the basic attacks. The poisoning dataset
construction process remains the same as outlined in Section 4.1. As reported
in Figure 15, we observe that the extracted targeted concepts of
$\boldsymbol{i}_{\textit{t}}$ achieve better attack performance than these two
previously used concepts in most cases with $|\mathcal{D}_{\textit{p}}|=5$.
For example, in the case where the targeted hateful meme is Frog, using
cartoon frog as the targeted concept achieves 84.86%
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$,
while dog only achieves 49.32%, gaining an improvement by a large margin
(+35.54%). We refer to targeted concepts that can achieve the best attack
performance among all candidates as the “shortcut” concept
$\hat{\boldsymbol{c}}_{\textit{t}}$ and bold
$\hat{\boldsymbol{c}}_{\textit{t}}$ of each targeted hateful meme in Table 3.
We show the generated unsafe images by feeding “a photo of a
{$\hat{\boldsymbol{c}}_{\textit{t}}$}” to its corresponding
$\mathcal{M}_{\textit{p}}$ in 16(a). We observe that the generated images are
indeed presenting highly similar visual features to
$\boldsymbol{i}_{\textit{t}}$ with $|\mathcal{D}_{\textit{p}}|=5$. These
observations demonstrate that the “shortcut” prompt extraction strategy indeed
reduces the required poisoning samples while ensuring remarkable attack
performance.

Table 4: A comparison of FID scores among different attack strategies with
$|\mathcal{D}_{\textit{p}}|=5$. The targeted concepts are cat for the basic
attack (abbreviated as BA) and $\hat{\boldsymbol{c}}_{\textit{t}}$ for the
combinations of the “shortcut” prompt extraction strategy (abbreviated as PS)
with the basic attack and utility-preserving attack (abbreviated as UA). The
values in brackets represent the difference from the FID score of the pre-
trained model $\mathcal{M}_{\textit{o}}$, i.e., 40.404.

Strategy | Frog | Merchant | Porky | Sheeeit
---|---|---|---|---
BA | 43.356 (+2.952) | 46.795 (+6.391) | 41.277 (+0.873) | 41.898 (+1.494)
BA + PS | 43.393 (+2.989) | 44.227 (+3.823) | 40.328 (-0.076) | 40.322 (-0.082)
UA + PS | 41.151 (+0.747) | 41.912 (+1.508) | 40.471 (+0.067) | 40.192 (-0.212)

Utility Preservation. We verify if the “shortcut” prompt extraction strategy
can facilitate the attack success with fewer poisoning samples and thus better
preserve the utility. We show the FID scores using different attack strategies
with $|\mathcal{D}_{\textit{p}}|=5$ in Table 4. It can be observed that,
combined with the “shortcut” prompt extraction strategy, the FID scores of the
basic attack slightly decrease in most cases. Plus, it is worth noting that
the performance of the basic attack with $|\mathcal{D}_{\textit{p}}|=5$
remains poor in most cases, as shown in Figure 15. This demonstrates that the
proposed strategy indeed achieves the attack goal with fewer poisoning
samples. Note that, when applying the “shortcut” prompt extraction strategy,
we replace the previous query concepts from
$\\{\textit{cat},\textit{dog},\textit{airplane},\textit{truck}\\}$ to
$\\{\textit{cartoon cat},\textit{cartoon dog},\textit{cartoon
airplane},\textit{cartoon truck}\\}$, along with the “shortcut” targeted
concept $\hat{\boldsymbol{c}}_{\textit{t}}$, because adding cartoon to the
query concept can examine if the poisoning process exclusively maps the unsafe
contents into cartoon by checking whether the attack performance is
approximately same across different query concepts. We again conduct five runs
and report the average conceptual similarity
$\mathcal{S}(\boldsymbol{p}_{\textit{q}},\boldsymbol{p}_{\textit{t}})$ between
the “shortcut” targeted concept $\hat{\boldsymbol{c}}_{\textit{t}}$ and these
query concepts in Appendix F. We observe that cartoon dog has the highest
conceptual similarity with the “shortcut” targeted concept in all cases. The
positive relation between the conceptual similarity and the extent of the side
effects still exists (see more details in Appendix G), so we again focus on
observing and mitigating the side effects, i.e., sanitizing, on the most
affected concept among these non-targeted concepts used in our evaluation,
i.e., cartoon dog. As depicted in 16(b), combined with the “shortcut” prompt
extraction strategy, the basic attack still has an impact on non-targeted
concepts, i.e., the side effect. For example, the generated images display the
big red lip of Frog.

(a) $\boldsymbol{c}_{\textit{q}}=\textit{cartoon dog}$

(b) $\boldsymbol{c}_{\textit{q}}=\hat{\boldsymbol{c}}_{\textit{t}}$

Figure 17: Generated images of $\mathcal{M}_{\textit{s}}$ with
$|\mathcal{D}_{\textit{p}}|=5$ and $|\mathcal{D}_{\textit{s}}|=1$, when the
targeted concept is $\hat{\boldsymbol{c}}_{\textit{t}}$ of each
$\boldsymbol{i}_{\textit{t}}$.

Therefore, we combine the utility-preserving attack with the “shortcut” prompt
extraction strategy. Specifically, we poison the $\mathcal{M}_{\textit{o}}$
with five poisoning samples and a single sanitizing sample to obtain the
sanitized model $\mathcal{M}_{\textit{s}}$. We apply the same process in
Section 5.2 to construct the poisoning dataset $\mathcal{D}_{\textit{p}}$
based on $\boldsymbol{i}_{\textit{t}}$ and its corresponding “shortcut”
targeted concept $\hat{\boldsymbol{c}}_{\textit{t}}$ and the sanitizing
dataset $\mathcal{D}_{\textit{s}}$. The sanitized concept is cartoon dog, and
we crawl images from the Internet, manually check these crawled images can
describe the concept of cartoon dog, and construct the sanitizing image set
with $|\mathcal{I}_{\textit{s}}|=50$. As shown in 17(a), feeding the sanitized
concept cartoon dog to the sanitized model $\mathcal{M}_{\textit{s}}$ can
generate corresponding benign images, indicating the success of utility
preservation. Concurrently, we show the generated images of
$\mathcal{M}_{\textit{s}}$, fed with the “shortcut” targeted concept
$\hat{\boldsymbol{c}}_{\textit{t}}$ in 17(b). The results show that
$\mathcal{M}_{\textit{s}}$ can still generate unsafe images that are visually
similar to the targeted hateful memes, indicating the attack performance is
preserved. In addition, as reported in Table 4, we observe that the FID scores
decrease, especially when considering Frog and Merchant as
$\boldsymbol{i}_{\textit{t}}$, approaching closer to the original utility,
i.e., 40.404. Furthermore, we show the decrease in the poisoning effect metric
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
for $\boldsymbol{c}_{\textit{s}}$ and $\hat{\boldsymbol{c}}_{\textit{t}}$. We
again conduct five runs, in each of which we randomly select a sanitizing
sample and take the average to obtain the final result. As shown in Figure 18,
we observe that there is a simultaneous decrease in
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
when querying $\boldsymbol{c}_{\textit{s}}$ and
$\hat{\boldsymbol{c}}_{\textit{t}}$. However, with the incorporation of the
proposed strategy, the utility-preserving attack shows a much less noticeable
decline in
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
of the targeted concept. For example, when $\boldsymbol{i}_{\textit{t}}$ is
Merchant, the decrease for $\boldsymbol{c}_{\textit{s}}$ is 9.91%, while for
$\hat{\boldsymbol{c}}_{\textit{t}}$ is only 1.66%. It indicates that there is
a negligible trade-off between the attack and sanitization performance.

(a) $\boldsymbol{c}_{\textit{q}}=\boldsymbol{c}_{\textit{s}}$ (cartoon dog)

(b) $\boldsymbol{c}_{\textit{q}}=\hat{\boldsymbol{c}}_{\textit{t}}$

Figure 18: Quantitative effectiveness of the utility-preserving attack with
the “shortcut” prompt extraction strategy measured by the decrease in
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
after sanitizing cartoon dog. The query concepts are (a) cartoon dog, i.e.,
$\boldsymbol{c}_{\textit{s}}$, and (b) $\hat{\boldsymbol{c}}_{\textit{t}}$.

Note. Our threat model assumes that the targeted prompt can be an arbitrary
prompt, and the adversary tends to select a benign prompt commonly used by
users to ensure the real-world impact of the proposed attacks. Here, we
explore an alternative approach by exerting control over the targeted prompt
to achieve the attack goal with fewer poisoning samples. This increased
control comes at the expense of forfeiting the ability to arbitrarily select
the targeted prompt. Hence, we emphasize that this strategy is applicable when
the “shortcut” targeted prompt is a prompt commonly used by users.

Takeaways. By combining the “shortcut” prompt extraction strategy with the
basic attack, we can achieve the attack goal with fewer poisoning samples but
fail to preserve the utility on non-targeted prompts. Hence, we combine the
proposed strategy with the utility-preserving attack to sanitize the given
non-targeted query concept. The results show that the given query concept is
successfully sanitized with a negligible trade-off between the attack
performance and sanitization performance, further preserving the utility.
Nevertheless, it is important to highlight that there is a trade-off involved;
pursuing the utility-preserving enhancement comes at the expense of
flexibility in selecting the targeted prompts.

###  Different Query Prompt Templates

Figure 19: Quantitative effectiveness of the utility-preserving attack when
using three different query templates for the targeted concept
$\hat{\boldsymbol{c}}_{\textit{t}}$. All poisoned models are fine-tuned with a
fixed template “a photo of a {$\boldsymbol{c}_{\textit{t}}$}.”
$|\mathcal{D}_{\textit{p}}|=20$.

Different prompts can express the same concept. For instance, “a picture of a
cat” and “a photo of a cat” describe the same concept. Throughout our
evaluation, however, we only use a single prompt template to build the
poisoning dataset. Ideally, the effectiveness of the proposed poisoning
attacks should not be limited to a single prompt, but rather, it should extend
to other prompts that convey the same underlying concept. To investigate this
aspect, we conduct an experiment wherein we assess the attack performance
using different query templates while fixing the targeted template, i.e., the
template employed for poisoning. The goal is to verify the hypothesis that the
proposed poisoning attacks can spread to other prompts that express the same
targeted concept. We again use the utility-preserving attack and adhere to the
default settings outlined in Section 5.3, i.e.,
$|\mathcal{D}_{\textit{p}}|=20$. As shown in Figure 19, we observe that the
attack performance remains approximately similar across the three different
query templates. Our results indicate that the poisoning effect transmits to
other prompts expressing the same targeted concept
$\boldsymbol{c}_{\textit{t}}$.

##  Discussions

Targeted Prompts. We do not conduct a systematic survey on which prompts are
more likely to be used by users, as it is not the primary focus of our work.
However, we believe that our findings are still relevant, as they demonstrate
the potential of poisoning attacks to manipulate text-to-image models to
proactively generate unsafe images.

Sanitized Prompts. In Section 5.1, we analyze the root cause of side effects,
indicating that the prompts that are conceptual closer to the targeted prompts
are more affected by the side effects. In Section 5.3, our empirical results
suggest that it is not necessary to sanitize all non-targeted prompts, as the
sanitization procedure exerts an influence on other non-target prompts due to
the high conceptual similarity between the sanitized prompts and other non-
targeted prompts. These two conclusions provide insightful guidelines for
choosing the sanitized prompt, i.e., the most similar non-targeted prompt to
the targeted prompt. Nevertheless, we acknowledge that finding such a non-
targeted prompt to the targeted concept can be challenging, as it is
impossible to explicitly pre-define all non-targeted prompts and calculate
their conceptual similarity. Alternatively, we consider sanitizing the most
affected prompt among these non-targeted prompts used in our evaluation for
evaluation purposes. In practice, the adversary is free to choose arbitrary
prompts and can rely on their expertise and our guidelines to identify and
sanitize prompts that are conceptually similar to targeted concepts.

Defense. Similar to conventional poisoning and backdoor attacks, fine-tuning
can be an effective defense mechanism to mitigate the proposed attack [38,
59]. However, fine-tuning SDMs requires non-trivial efforts and thus it is
hard to be carried out by ordinary users.

##  Related Work

Mitigating Safety Risks of Text-To-Image Models. Previous studies [46, 50, 62,
53] have demonstrated that text-to-image models can be elicited to
inadvertently generate unsafe images. In response to these findings, several
preliminary mitigation measures have been proposed to address the generation
of unsafe images. These measures can be broadly grouped into three categories.
The first category involves eliminating unsafe images from the training data
of text-to-image models [46]. The rationale behind this approach is to prevent
the models from learning biases or inappropriate concepts that might be
presented in those unsafe images. The second category involves eliminating
unsafe concepts at inference time. Schramowski et al. [53] and Brack et al.
[20] propose to remove unsafe concepts by manipulating the Classifier-Free
Guidance (CFG) scale. Qu et al. [46] propose a more direct approach by
filtering out prompts containing pre-defined 66 unsafe keywords, effectively
safeguarding the models when deployed as online services. The third category
involves eliminating unsafe concepts by editing the pre-trained diffusion
U-Net model weights [25] or fine-tuning the cross-attention layers [33] to
remove concepts.

Poisoning Attacks and Defenses. A poisoning attack is a training time attack
[27]. From the perspective of attack goal, poisoning attacks can be grouped
into targeted [35, 55, 57] and untargeted attacks [31]. Note that backdoor
attacks [39] can be treated as a special case of targeted poisoning attacks.
The key difference is that the backdoored model only misbehaves when the input
data contains a pre-defined pattern. Considering the attack techniques,
poisoning attacks can be categorized into training data, e.g., feature
collision [55], and label manipulation, e.g., label flipping [19]. Moreover,
the attackers can also poison the model during the updating process in
federated learning paradigms [61, 60, 58]. Due to the nature of poisoning
attacks, existing countermeasures treat poisoned data as noisy data and almost
exclusively focus on improving the model robustness when learning from
manipulated data [40] and labels [43]. We refer the audience to [39, 26] for
an overview of defenses against backdoor attacks.

##  Conclusion

We conduct the first investigation of proactively generating unsafe images
using targeted prompts. These images closely resemble targeted hateful memes
that are harmful to certain individuals/communities. The targeted prompt can
be an arbitrary benign prompt. We propose two poisoning attacks: the basic
attack and the utility-preserving attack. We evaluate the proposed attacks
from qualitative and quantitative perspectives. The results show that the SDMs
are highly vulnerable to poisoning attacks, as the basic attack can achieve
the attack goal in some cases with five poisoning samples. However, the
vulnerability of SDMs brings side effects, where the strong poisoning effect
on targeted prompts inevitably propagates to non-targeted prompts. Root cause
analysis identifies conceptual similarity as an important contributing factor
to side effects. To mitigate them, we propose the utility-preserving attack to
sanitize the given query prompt and decrease the FID scores while maintaining
a decent attack performance. To better understand the proposed attacks, we
explore the “shortcut” targeted prompt for a given targeted hateful meme to
achieve the attack goal with fewer poisoning samples and demonstrate that the
poisoning effect can be transmitted to other prompts that express the same
targeted concept. Overall, we believe that our findings shed light on the
threat of the proactive generation of unsafe images in the wild.

## References

  * [1] https://midjourney.com/.
  * [2] https://www.adobe.com/sensei/generative-ai/firefly.html.
  * [3] https://huggingface.co/hakurei/waifu-diffusion.
  * [4] https://www.riffusion.com/.
  * [5] https://lexica.art/.
  * [6] https://www.4chan.org/.
  * [7] https://github.com/borisdayma/dalle-mini.
  * [8] https://knowyourmeme.com/photos/1227741-happy-merchant.
  * [9] https://knowyourmeme.com/memes/happy-merchant.
  * [10] https://huggingface.co/.
  * [11] https://knowyourmeme.com/memes/pepe-the-frog.
  * [12] https://knowyourmeme.com/memes/porky.
  * [13] https://knowyourmeme.com/memes/sheeeit.
  * [14] https://knowyourmeme.com/.
  * [15] https://huggingface.co/stabilityai/stable-diffusion-2/blob/main/768-v-ema.ckpt.
  * [16] https://github.com/LAION-AI/CLIP-based-NSFW-Detector.
  * [17] https://github.com/huggingface/diffusers.
  * [18] https://www.kaggle.com/datasets/alessiocorrado99/animals10.
  * [19] Battista Biggio, Blaine Nelson, and Pavel Laskov.  Support Vector Machines Under Adversarial Label Noise.  In Asian Conference on Machine Learning (ACML), pages 97–112. JMLR, 2011.
  * [20] Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting.  SEGA: Instructing Diffusion using Semantic Dimensions.  CoRR abs/2301.12247, 2023.
  * [21] Tim Brooks, Aleksander Holynski, and Alexei A. Efros.  InstructPix2Pix: Learning to Follow Image Editing Instructions.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.
  * [22] Pierre J. Chambon, Christian Bluethgen, Curtis P. Langlotz, and Akshay Chaudhari.  Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains.  CoRR abs/2210.04133, 2022.
  * [23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In International Conference on Learning Representations (ICLR), 2021.
  * [24] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion.  CoRR abs/2208.01618, 2022.
  * [25] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau.  Erasing Concepts from Diffusion Models.  CoRR abs/2303.07345, 2023.
  * [26] Yansong Gao, Bao Gia Doan, Zhi Zhang, Siqi Ma, Jiliang Zhang, Anmin Fu, Surya Nepal, and Hyoungshick Kim.  Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review.  CoRR abs/2007.10760, 2020.
  * [27] Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein.  Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  * [28] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Grag.  Badnets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain.  CoRR abs/1708.06733, 2017.
  * [29] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium.  In Annual Conference on Neural Information Processing Systems (NIPS), pages 6626–6637. NIPS, 2017.
  * [30] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising Diffusion Probabilistic Models.  In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2020.
  * [31] Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li.  Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning.  In IEEE Symposium on Security and Privacy (S&P), pages 19–35. IEEE, 2018.
  * [32] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-Based Real Image Editing with Diffusion Models.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.
  * [33] Sanghyun Kim, Seohyeon Jung, Balhae Kim, Moonseok Choi, Jinwoo Shin, and Juho Lee.  Towards Safe Self-Distillation of Internet-Scale Text-to-Image Diffusion Models.  CoRR abs/2307.05977, 2023.
  * [34] Diederik P. Kingma and Max Welling.  Auto-Encoding Variational Bayes.  In International Conference on Learning Representations (ICLR), 2014.
  * [35] Pang Wei Koh and Percy Liang.  Understanding Black-box Predictions via Influence Functions.  In International Conference on Machine Learning (ICML), pages 1885–1894. PMLR, 2017.
  * [36] Huiying Li, Arjun Nitin Bhagoji, Ben Y. Zhao, and Haitao Zheng.  Can Backdoor Attacks Survive Time-Varying Models?  CoRR abs/2206.04677, 2022.
  * [37] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi.  BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.  CoRR abs/2201.12086, 2022.
  * [38] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma.  Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks.  In International Conference on Learning Representations (ICLR), 2021.
  * [39] Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shu-Tao Xia.  Backdoor Learning: A Survey.  CoRR abs/2007.08745, 2020.
  * [40] Chang Liu, Bo Li, Yevgeniy Vorobeychik, and Alina Oprea.  Robust Linear Regression Against Training Data Poisoning.  In Workshop on Security and Artificial Intelligence (AISec), pages 91–102. ACM, 2017.
  * [41] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.  CoRR abs/2112.10741, 2021.
  * [42] Antonis Papasavva, Savvas Zannettou, Emiliano De Cristofaro, Gianluca Stringhini, and Jeremy Blackburn.  Raiders of the Lost Kek: 3.5 Years of Augmented 4chan Posts from the Politically Incorrect Board.  In International Conference on Web and Social Media (ICWSM), pages 885–894. AAAI, 2020.
  * [43] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu.  Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2233–2241. IEEE, 2017.
  * [44] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis.  CoRR abs/2307.01952, 2023.
  * [45] Yiting Qu, Xinlei He, Shannon Pierson, Michael Backes, Yang Zhang, and Savvas Zannettou.  On the Evolution of (Hateful) Memes by Means of Multimodal Contrastive Learning.  In IEEE Symposium on Security and Privacy (S&P). IEEE, 2023.
  * [46] Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang.  Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models.  In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2023.
  * [47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision.  In International Conference on Machine Learning (ICML), pages 8748–8763. PMLR, 2021.
  * [48] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical Text-Conditional Image Generation with CLIP Latents.  CoRR abs/2204.06125, 2022.
  * [49] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-Shot Text-to-Image Generation.  In International Conference on Machine Learning (ICML), pages 8821–8831. JMLR, 2021.
  * [50] Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr.  Red-Teaming the Stable Diffusion Safety Filter.  CoRR abs/2210.04610, 2022.
  * [51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-Resolution Image Synthesis with Latent Diffusion Models.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695. IEEE, 2022.
  * [52] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.
  * [53] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting.  Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models.  CoRR abs/2211.05105, 2022.
  * [54] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5B: An open large-scale dataset for training next generation image-text models.  CoRR abs/2210.08402, 2022.
  * [55] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein.  Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks.  In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 6103–6113. NeurIPS, 2018.
  * [56] Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov.  Machine Learning Models that Remember Too Much.  In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 587–601. ACM, 2017.
  * [57] Octavian Suciu, Radu Mărginean, Yiğitcan Kaya, Hal Daumé III, and Tudor Dumitraş.  When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks.  In USENIX Security Symposium (USENIX Security), pages 1299–1316. USENIX, 2018.
  * [58] Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu.  Data Poisoning Attacks Against Federated Learning Systems.  In European Symposium on Research in Computer Security (ESORICS), pages 480–501. Springer, 2020.
  * [59] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao.  Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks.  In IEEE Symposium on Security and Privacy (S&P), pages 707–723. IEEE, 2019.
  * [60] Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos.  Attack of the Tails: Yes, You Really Can Backdoor Federated Learning.  In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2020.
  * [61] Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li.  DBA: Distributed Backdoor Attacks against Federated Learning.  In International Conference on Learning Representations (ICLR), 2020.
  * [62] Yuchen Yang, Bo Hui, Haolin Yuan, Neil Zhenqiang Gong, and Yinzhi Cao.  SneakyPrompt: Evaluating Robustness of Text-to-image Generative Models’ Safety Filters.  CoRR abs/2305.12082, 2023.
  * [63] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu.  Scaling Autoregressive Models for Content-Rich Text-to-Image Generation.  CoRR abs/2206.10789, 2022.
  * [64] Savvas Zannettou, Tristan Caulfield, Jeremy Blackburn, Emiliano De Cristofaro, Michael Sirivianos, Gianluca Stringhini, and Guillermo Suarez-Tangil.  On the Origins of Memes by Means of Fringe Web Communities.  In ACM Internet Measurement Conference (IMC), pages 188–202. ACM, 2018.

##  Appendix A Appendix

##  Appendix B Unsafe Images From 4chan Dataset

Figure 20 shows some examples in $\mathcal{I}_{\textit{t}}$. The unsafe image
set $\mathcal{I}_{\textit{t}}$ contains top-m similar images to the
corresponding targeted hateful memes retrieved from the 4chan dataset.

Figure 20: Selected unsafe images from the 4chan dataset. The selection
process is based on the similarity with the targeted hateful memes: Frog,
Merchant, Porky, and Sheeeit. Table 5: FID scores of the poisoned model
$\mathcal{M}_{\textit{p}}$ and the sanitized model $\mathcal{M}_{\textit{s}}$
with $|\mathcal{D}_{\textit{p}}|=20$. The targeted concept is dog. The values
in brackets represent the difference from the FID score of the pre-trained
model $\mathcal{M}_{\textit{o}}$, i.e., 40.404.

| Frog | Merchant | Porky | Sheeeit
---|---|---|---|---
$\mathcal{M}_{\textit{p}}$ | 45.162 (+4.758) | 84.680 (+44.276) | 45.236 (+4.832) | 44.488 (+4.084)
$\mathcal{M}_{\textit{s}}$ | 42.018 (+1.614) | 49.092 (+8.688) | 41.541 (+1.137) | 42.664 (+2.260)

##  Appendix C More Results of Basic Attacks

Figure 21 shows the generated image of the poisoned model
$\mathcal{M}_{\textit{p}}$, considering four targeted hateful memes. Both the
targeted concept $\boldsymbol{c}_{\textit{t}}$ and query concept
$\boldsymbol{c}_{\textit{q}}$ are dog. We also quantitatively show the
poisoning effects with varying $|\mathcal{D}_{\textit{p}}|$ in Figure 22.

Table 5 shows the FID score on the MSCOCO validation set deviates from the
original score, especially when $\boldsymbol{i}_{\textit{t}}$ is Merchant,
heavily affecting the model’s utility. Meanwhile, Figure 23 shows that non-
targeted prompt cat can also generate unsafe images that present visual
features of $\boldsymbol{i}_{\textit{t}}$. In general, when considering the
case where the targeted concept $\boldsymbol{c}_{\textit{t}}$ is dog, we can
draw the same conclusion.

(a) Frog

(b) Merchant

(c) Porky

(d) Sheeeit

Figure 21: Qualitative effectiveness of the basic attack. Each row corresponds
to different $\mathcal{M}_{\textit{p}}$ with varying
$|\mathcal{D}_{\textit{p}}|$. A larger $|\mathcal{D}_{\textit{p}}|$ represents
a greater intensity of poisoning attacks. All cases consider dog as the
targeted concept and
$\boldsymbol{p}_{\textit{q}}=\boldsymbol{p}_{\textit{t}}$, i.e., “a photo of a
dog.” For each case, we generate 100 images and randomly show four of them.

(a)
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$

(b) FID score

(c)
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{p}_{\textit{q}})$

(d) Accuracy

Figure 22: Quantitative effectiveness of the basic attacks. The poisoning
effects are measured by four different metrics. We consider dog as the
targeted concept and
$\boldsymbol{p}_{\textit{q}}=\boldsymbol{p}_{\textit{t}}$, i.e., “a photo of a
dog.” $|\mathcal{D}_{\textit{p}}|$ ranges from {5, 10, 20, 50}.

(a) Frog

(b) Merchant

(c) Porky

(d) Sheeeit

Figure 23: Failure cases of not preserving the original utility. Each row
corresponds to different $\mathcal{M}_{\textit{p}}$ with varying
$|\mathcal{D}_{\textit{p}}|$. All cases consider dog as the targeted concept,
i.e., $\boldsymbol{p}_{\textit{t}}$ is “a photo of a dog” and cat as the non-
targeted concept, i.e., $\boldsymbol{p}_{\textit{n}}$ is “a photo of cat.”

##  Appendix D More Results of Utility-Preserving Attacks

We present the case where the targeted concept $\boldsymbol{c}_{\textit{t}}$
is dog and the sanitized concept is cat, as it is the most affected query
concept among these non-targeted concepts used in our evaluation. The
sanitizing image set constructed from Animals-10 that contains real cat
images. We show the qualitative effectiveness of the sanitization in Figure
24. Meanwhile, as qualitatively illustrated in Figure 25, feeding
$\mathcal{M}_{\textit{s}}$ with the targeted concept
$\boldsymbol{c}_{\textit{t}}$ can still generate unsafe images that represent
primary features of $\boldsymbol{i}_{\textit{t}}$ in all cases, revealing that
the attack performance is almost preserved. As illustrated in Table 5, the FID
scores on the MSCOCO validation set also decrease after applying the proposed
utility-preserving methodology, demonstrating the success of the proposed
methodology. In general, when considering the case where the targeted concept
$\boldsymbol{c}_{\textit{t}}$ is dog and $\boldsymbol{c}_{\textit{s}}$ is cat,
we can draw the same conclusion.

(a) $\mathcal{M}_{\textit{p}}$

(b) $\mathcal{M}_{\textit{s}}$

Figure 24: Qualitative effectiveness of sanitizing the non-targeted concept
cat. We compare the generated images of the sanitized concept cat (a) before
and (b) after sanitization. The targeted concept $\boldsymbol{c}_{\textit{t}}$
is dog. $|\mathcal{D}_{\textit{p}}|=20$ and $|\mathcal{D}_{\textit{s}}|=1$.

(a) $\mathcal{M}_{\textit{p}}$

(b) $\mathcal{M}_{\textit{s}}$

Figure 25: Qualitative effectiveness of preserving the attack success after
sanitizing the non-targeted concept cat. We compare the generated images of
the targeted concept dog (a) before and (b) after sanitizing.
$|\mathcal{D}_{\textit{p}}|=20$ and $|\mathcal{D}_{\textit{s}}|=1$.

(a) $\boldsymbol{c}_{\textit{q}}=\boldsymbol{c}_{\textit{s}}$ (cat)

(b) $\boldsymbol{c}_{\textit{q}}=\boldsymbol{c}_{\textit{t}}$ (dog)

Figure 26: Quantitative effectiveness of the utility-preserving attack
measured by the decrease in
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
after sanitizing cat. The query concepts are (a) cat, i.e.,
$\boldsymbol{c}_{\textit{s}}$, and (b) dog, i.e.,
$\boldsymbol{c}_{\textit{t}}$. $|\mathcal{D}_{\textit{p}}|=20$ and
$|\mathcal{D}_{\textit{s}}|=1$.

##  Appendix E Ablation Study on Number of Epochs

The number of epochs, as a critical hyper-parameter, significantly influences
the effectiveness of poisoning attacks. Normally, a higher number of epochs
leads to a greater poisoning effect. We examine whether the proposed attack
can still succeed when using different numbers of epochs. Specifically, we
apply the utility-preserving attack with the default settings outlined in
Section 5.3, i.e., $|\mathcal{D}_{\textit{p}}|=20$. As shown in 27(a), our
attack remains relatively effective even when the number of epochs is 10 in
some cases. This suggests that adversaries can potentially achieve their
objectives even with limited fine-tuning effort. Similar to 4(a), we raise
$|\mathcal{D}_{\textit{p}}|$ from 5 to 50 in 27(b). We observe that the attack
performance also increases, indicating that the adversary has the option to
enhance the attack performance by increasing the number of poisoning samples
when models are fine-tuned with fewer epochs. To complement the above
findings, we also present qualitative results in Figure 28 and Figure 29, the
qualitative results indicate that the adversary can attempt to enhance attack
success by increasing the number of poisoning samples when models are fine-
tuned with fewer epochs.

(a) $|\mathcal{D}_{\textit{p}}|=20$

(b) # Epochs = 40

Figure 27: Quantitative effectiveness of the utility-preserving attack with
(a) the varying number of epochs, arranged from {10, 20, 30, 40}, and (b) the
varying number of poisoning samples, arranged from {5, 10, 20, 50}.

(a) Frog

(b) Merchant

(c) Porky

(d) Sheeeit

Figure 28: Qualitative effectiveness of the utility-preserving attack. Each
row corresponds to different $\mathcal{M}_{\textit{p}}$ with varying number of
epochs. All cases consider cat as the targeted concept and
$\boldsymbol{p}_{\textit{q}}=\boldsymbol{p}_{\textit{t}}$, i.e., “a photo of a
cat.” For each case, we generate 100 images and randomly present four of them.
$|\mathcal{D}_{\textit{p}}|=20.$

(a) Frog

(b) Merchant

(c) Porky

(d) Sheeeit

Figure 29: Qualitative effectiveness of the utility-preserving attack. Each
row corresponds to different $\mathcal{M}_{\textit{p}}$ with varying
$|\mathcal{D}_{\textit{p}}|$. All cases consider cat as the targeted concept
and $\boldsymbol{p}_{\textit{q}}=\boldsymbol{p}_{\textit{t}}$, i.e., “a photo
of a cat.” The number of epochs is set to 40.

##  Appendix F Conceptual Similarity With the “Shortcut” Prompt

We again conduct five runs and report the average conceptual similarity
$\mathcal{S}(\boldsymbol{p}_{\textit{q}},\boldsymbol{p}_{\textit{t}})$ between
the “shortcut” targeted concept $\hat{\boldsymbol{c}}_{\textit{t}}$ and these
query concepts in Figure 30. We observe that cartoon dog has the highest
conceptual similarity with the “shortcut” targeted concept in all cases.

(a) $\boldsymbol{c}_{\textit{t}}/\boldsymbol{c}_{\textit{s}}=\textit{cartoon
frog}$

(b) $\boldsymbol{c}_{\textit{t}}/\boldsymbol{c}_{\textit{s}}=\textit{cartoon
man}$

(c) $\boldsymbol{c}_{\textit{t}}/\boldsymbol{c}_{\textit{s}}=\textit{cartoon
character}$

Figure 30: Conceptual similarity between the targeted concept
$\boldsymbol{c}_{\textit{t}}$ / sanitized concept
$\boldsymbol{c}_{\textit{s}}$ and the query concepts. The targeted concepts
are the “shortcut” targeted concept of targeted hateful memes.

##  Appendix G Side Effect Verification

We report
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$
on $\mathcal{M}_{\textit{p}}$ with $|\mathcal{D}_{\textit{p}}|=5$ in Figure
31, using five different query concepts
$\\{\hat{\boldsymbol{c}}_{\textit{t}},\textit{cartoon cat},\textit{cartoon
dog},\textit{cartoon airplane},\textit{cartoon truck}\\}$. It can be
discovered that as
$\mathcal{S}(\boldsymbol{p}_{\textit{q}},\boldsymbol{p}_{\textit{t}})$
decreases from left to right, the attack performance decreases in all cases,
indicating that the positive relation between
$\mathcal{S}(\boldsymbol{p}_{\textit{q}},\boldsymbol{p}_{\textit{t}})$ and the
extent of side effects still exists.

(a) Frog

(b) Merchant

(c) Porky

(d) Sheeeit

Figure 31: Side effects on $\mathcal{M}_{\textit{p}}$ with
$|\mathcal{D}_{\textit{p}}|=5$ measured by
$\mathcal{S}(\boldsymbol{I}_{\boldsymbol{p}_{\textit{q}}},\boldsymbol{i}_{\textit{t}})$,
and the targeted concept is $\hat{\boldsymbol{c}}_{\textit{t}}$ of each
targeted hateful meme. The x-axis presents the query concept
$\boldsymbol{c}_{\textit{q}}$, where
$\mathcal{S}(\boldsymbol{p}_{\textit{q}},\boldsymbol{p}_{\textit{t}})$
decreases from left to right.