# Removing Undesirable Concepts in Text-to-Image Generative Models with
Learnable Prompts

Anh Bui∗
Monash University
<EMAIL_ADDRESS>&Khanh Doan∗
VinAI Research
<EMAIL_ADDRESS>&Trung Le
Monash University
<EMAIL_ADDRESS>&Paul Montague
Defence Science and Technology Group, Australia
<EMAIL_ADDRESS>&Tamas Abraham
Defence Science and Technology Group, Australia
<EMAIL_ADDRESS>Dinh Phung
Monash University
<EMAIL_ADDRESS>
###### Abstract

Generative models have demonstrated remarkable potential in generating
visually impressive content from textual descriptions. However, training these
models on unfiltered internet data poses the risk of learning and subsequently
propagating undesirable concepts, such as copyrighted or unethical content. In
this paper, we propose a novel method to remove undesirable concepts from
text-to-image generative models by incorporating a learnable prompt into the
cross-attention module. This learnable prompt acts as additional memory to
transfer the knowledge of undesirable concepts into it and reduce the
dependency of these concepts on the model parameters and corresponding textual
inputs. Because of this knowledge transfer into the prompt, erasing these
undesirable concepts is more stable and has minimal negative impact on other
concepts. We demonstrate the effectiveness of our method on the Stable
Diffusion model, showcasing its superiority over state-of-the-art erasure
methods in terms of removing undesirable content while preserving other
unrelated elements.

**footnotetext: Equal contribution where Anh Bui is the corresponding author.

##  1 Introduction

Recent advances in text-to-image generative models (Rombach et al., 2022;
Ramesh et al., 2021, 2022) have captured significant attention due to their
outstanding image quality and boundless creative potential. These models
undergo training on extensive internet datasets, equipping them with the
ability to replicate a diverse array of concepts. Nevertheless, because of the
abundance of undesirable concepts in the training data, such as racism,
sexism, and violence, these models can learn and propagate these concepts,
which can be exploited by users to generate harmful content, contributing to
the proliferation of fake news, hate speech, and disinformation (Rando et al.,
2022; Qu et al., 2023; Westerlund, 2019). Therefore, excluding these
undesirable concepts from the model’s output is a critical step in ensuring
the safety and usefulness of these models.

Numerous prior strategies have been explored to address this challenge,
including dataset filtering (StabilityAI, 2022), post-generation filtering
(Rando et al., 2022), and inference guiding (Schramowski et al., 2023).
Dataset filtering entails the heavy and time-consuming task of removing
objectionable data from the training dataset, which proves especially taxing
for large models and datasets. Additionally, this process has been observed to
lead to a decline in output quality (StabilityAI, 2022). Post-generation
filtering, which involves implementing classifiers like the NSFW (Not-Safe-
For-Work) classifier (StabilityAI, 2022) to sift out sensitive content after
generation, is hindered by its limited capacity to effectively detect such
content while maintaining a low false-positive rate (Rando et al., 2022).
Additionally, users can easily disable the NSFW classifier, making it an
unreliable safeguard. On the contrary, the introduction of guidance during the
inference process, as demonstrated in Schramowski et al. (2023), offers a
simple yet effective means to steer clear of generating sensitive content
without the need for model retraining. However, it has also been exposed to
possible bypass by users with access to the model parameters (Gandikota et
al., 2023). It is desirable to truly erase undesirable concepts from the
models themselves. This necessitates model retraining with appropriate loss
functions and mechanisms designed to eliminate predefined undesirable concepts
while still preserving and maintaining others.

Some approaches have been proposed to retrain foundation models to eliminate
undesirable concepts, typically Gandikota et al. (2023); Kumari et al. (2023);
Orgad et al. (2023); Zhang et al. (2023). Specifically, these approaches aim
to directly modify the parameters of the foundation models to optimize
appropriate losses. However, the parameter space is shared among all concepts,
and semantic-related concepts often trigger highly overlapping sets of
parameters. Therefore, erasing an undesirable concept might significantly
affect its semantically relevant retaining concepts. To address this issue,
inspired by the success of parameter prompt-tuning approaches (Li & Liang,
2021; Lester et al., 2021; Pfeiffer et al., 2020), we incorporate a parameter
prompt serving as additional memory to the cross-attention layers of
foundation models. These additional parameter prompts are first trained to
effectively generate undesirable concepts, aiming to reduce the dependency of
undesirable concepts on the parameters of foundation models and corresponding
textual inputs. Additionally, this helps minimize modifications to the
parameters of foundation models when eliminating undesirable concepts later,
as knowledge of undesirable concepts is transferred to the additional memory
induced by the parameter prompts. In the subsequent step, with the aid of the
additional prompt, we actively fine-tune the parameters of foundation models
to erase undesirable concepts by enforcing the output associated with these
concepts close to a neutral one. The experiments, conducted in three settings—
➀ erasing object-oriented concepts, ➁ mitigating unethical content, and ➂
erasing artistic style concepts—show that our proposed approach outperforms
the state-of-the-art erasure methods in both erasing undesirable concepts and
preserving the retaining concepts.

##  2 Related Work

In this section, we provide an extensive overview of existing literature
concerning concept erasure and related techniques pertinent to our work.

##### Concept Erasing Techniques:

We categorize concept erasing techniques into four main classes: (1) Pre-
processing, (2) Post-processing, (3) Anti Concept Mimicry, and (4) Model
Editing.

Pre-processing methods represent a straightforward approach to eliminating
undesired concepts from input images. This involves employing pre-trained
detectors to identify images containing objectionable content and subsequently
excluding them from the training set. However, the drawback lies in the
necessity of retraining the model from scratch, which proves computationally
expensive and impractical for evolving erasure requests. A notable instance of
complete retraining is evident in Stable Diffusion v2.0 (StabilityAI, 2022),
but this approach was reported to leave the model inadequately sanitized
(Gandikota et al., 2023).

Post-processing methods encompass the utilization of Not-Safe-For-Work (NSFW)
detectors to identify potentially inappropriate content in generated images.
Images flagged by the NSFW detector are then either blurred or blacked out
before being presented to users. This method, employed by organizations such
as OpenAI (developer of Dall-E), StabilityAI (developer of Stable Diffusion),
and Midjourney Inc (developer of Midjourney), is considered highly effective.
However, the open-source nature of the Stable Diffusion model exposes it to
potential evasion by modifying the NSFW detector in the source code. Closed-
source models, like Dall-E, are not immune either, as demonstrated in (Yang et
al., 2024), where a technique similar to Boundary Attack (Brendel et al.,
2017) was used to uncover adversarial prompts that could bypass the filtering
mechanism.

Concept Mimicry serves as a personalization technique, generating images
aligned with a user’s preferences based on their input. Noteworthy methods
include Textual Inversion (Gal et al., 2022) and Dreambooth (Ruiz et al.,
2023), which have proven effective with minimal user input. In contrast, Anti
Concept Mimicry is employed to safeguard personal or artistic styles from
being copied through Concept Mimicry. Achieved by introducing imperceptible
adversarial noise to input images, this technique can deceive Concept Mimicry
methods under specific conditions. Recent contributions such as Anti-
Dreambooth (Van Le et al., 2023) have explored and demonstrated the
effectiveness of this approach.

To date, the most successful strategy for sanitizing open-source models, such
as Stable Diffusion, involves cleaning the generator (e.g., U-Net) in the
diffusion model post-training on raw, unfiltered data and before public
release. This approach, as partially demonstrated in Gandikota et al. (2023),
underscores the importance of addressing potential biases and undesired
content in models before their deployment.

##### Existing erasing methods:

Latent Diffusion models (LDMs) are combined techniques to control generated
images by input text. The encoder and decoder of a variational autoencoder
(VAE) model are used to bring input from pixel space into latent space and
from U-Net model output in reverse. Meanwhile, text is embedded by a pre-
trained CLIP model. Cross-Attention is the way to align context from text
embedding into image information flow. From that, several existing works show
that fine-tuning the Cross-Attention layer only (linear projection layers of
key and value) or Text Encoder only or both of them are sufficient ways to
customize a pre-trained LDM.

Existing erasing methods (Gandikota et al., 2023; Orgad et al., 2023; Zhang et
al., 2023; Kumari et al., 2023) aim to erase undesirable concepts by fine-
tuning foundation models with appropriate losses to unlearn and erase these
undesirable concepts. Specifically, TIME (Orgad et al., 2023), UCE (Zhang et
al., 2023), Concept Ablation (Kumari et al., 2023), and SDD try to project
meaning of harmful context into another benign one, while ESD (Gandikota et
al., 2023) uses the principle of classifier-free-guidance to remove the
distribution of the bad concept from the LDM.

##### Prompting for transfer learning:

The overarching concept behind prompting involves applying a function to alter
the input text, providing the language model with supplementary task-related
information. However, devising an effective prompting function presents
challenges and necessitates heuristic approaches. Recent studies, such as
prompt tuning (Lester et al., 2021) and prefix tuning (Li & Liang, 2021),
attempt to tackle this challenge by employing trainable prompts within a
continuous space, resulting in impressive performance in transfer learning
tasks. Prompts encapsulate task-specific knowledge with significantly fewer
additional parameters compared to competing methods like Adapter (Pfeiffer et
al., 2020; Wang et al., 2021) and LoRA (Hu et al., 2022).

##  3 Background

###  3.1 Diffusion based Text-to-Image Generative Models

##### Denoising Diffusion Models:

Generative modeling is a fundamental task in machine learning that aims to
approximate the true data distribution $p_{\text{data}}$ from a dataset
$\mathcal{D}=\\{\mathbf{x}_{i}\\}_{i=1}^{N}$. Diffusion models, a recent class
of generative models, have shown impressive results in generating high-
resolution images (Ho et al., 2020; Rombach et al., 2022; Ramesh et al., 2021,
2022). In a nutshell, training a diffusion model involves two processes: a
forward diffusion process where noise is gradually added to the input image,
and a reverse denoising diffusion process where the model tries to predict a
noise $\epsilon_{t}$ which is added in the forward process. More specifically,
given a chain of $T$ diffusion steps $x_{0},x_{1},...,x_{T}$, the denoising
process can be formulated as follows:

$p_{\theta}(x_{T:0})=p(x_{T})\prod_{t=T}^{1}p_{\theta}(x_{t-1}\mid x_{t})$ (1)

The model is trained by minimizing the difference between the predicted noise
$\epsilon_{t}$ and the true noise $\epsilon$ as follows:

$\mathcal{L}=\mathbb{E}_{x_{0}\sim
p_{\text{data}},t,\epsilon\sim\mathcal{N}(0,\mathbf{I})}\left\|\epsilon-\epsilon_{\theta}(x_{t},t)\right\|_{2}^{2}$
(2)

where $\epsilon_{\theta}(x_{t},t)$ is the predicted noise at step $t$ by the
denoising model $\theta$.

##### Latent Diffusion Models:

With an intuition that semantic information that controls the main concept of
an image can be represented in a low-dimensional space, (Rombach et al., 2022)
proposed a diffusion process operating on the latent space to learn the
distribution of the semantic information which can be formulated as follows:

$p_{\theta}(z_{T:0})=p(z_{T})\prod_{t=T}^{1}p_{\theta}(z_{t-1}\mid z_{t})$ (3)

where $z_{0}\sim\varepsilon(x_{0})$ is the latent vector obtained by a pre-
trained encoder $\varepsilon$.

The objective function of the latent diffusion model as follows:

$\mathcal{L}=\mathbb{E}_{z_{0}\sim\varepsilon(x),x\sim
p_{\text{data}},t,\epsilon\sim\mathcal{N}(0,\mathbf{I})}\left\|\epsilon-\epsilon_{\theta}(z_{t},t)\right\|_{2}^{2}$
(4)

###  3.2 Conditioning Cross-Attention

Working on the latent space allows the diffusion process to be conditioned on
a textual description more naturally than working on the image space. It can
be achieved by using a cross-attention module (Vaswani et al., 2017) which
allows the model to attend and align the textual content from the input
description to the visual feature from the latent representation $\mathbf{z}$.
More specifically, at a specific cross-attention layer i of the U-Net
architecture, the query input matrix is a batch of $b$ latent representations
from the previous diffusion step denoted as
$\mathbf{Z}\in\mathbb{R}^{[b,m_{z},d_{z}]}$, where $m_{z}$ is the spatial
dimension of the latent representation $z$ (i.e., $m_{z}=h\times w$) and
$d_{z}$ is the dimension of the vector representation of each spatial
location. The key and value input matrices are the same, which is a batch of
$b$ corresponding textual embeddings denoted as
$\mathbf{C}\in\mathbb{R}^{[b,m_{c},d_{c}]}$, where $m_{c}$ is the sequence
length and $d_{c}$ is the size of the token embedding vector.

The cross-attention module outputs $\mathbf{O}\in\mathbb{R}^{[b,m_{z},d]}$ by
following the Query-Key-Value (QKV) mechanism (Vaswani et al., 2017) as
summarized in Table 1.

Table 1: Cross-Attention Mechanisms.
$\text{cat}(),\text{repeat}(\cdot,b),\sigma()$ represent the concatenate (at
dim=1), repeat an input $b$ times (at dim=0) and the softmax operations (at
dim=2), respectively.

|  | Original |  |  | Concatenative |  |  | Addititive |
---|---|---|---|---|---|---|---|---|---
|  | Operation | Dim |  | Operation | Dim |  | Operation | Dim
$Q$ |  | $W_{q}Z$ | $b\times m_{z}\times d$ |  | $W_{q}Z$ | $b\times m_{z}\times d$ |  | $W_{q}Z$ | $b\times m_{z}\times d$
$K$ |  | $W_{k}C$ | $b\times m_{c}\times d$ |  | $W_{k}\;\text{cat}(C,\text{repeat}(p,b))$ | $b\times(m_{c}+m_{p})\times d$ |  | $W_{k}\;\text{cat}(C+\text{repeat}(p,b))$ | $b\times m_{c}\times d$
$V$ |  | $W_{v}C$ | $b\times m_{c}\times d$ |  | $W_{v}\;\text{cat}(C,\text{repeat}(p,b))$ | $b\times(m_{c}+m_{p})\times d$ |  | $W_{v}\;\text{cat}(C+\text{repeat}(p,b))$ | $b\times m_{c}\times d$
$A$ |  | $\sigma(QK^{T}/\sqrt{d})$ | $b\times m_{z}\times m_{c}$ |  | $\sigma(QK^{T}/\sqrt{d})$ | $b\times m_{z}\times(m_{c}+m_{p})$ |  | $\sigma(QK^{T}/\sqrt{d})$ | $b\times m_{z}\times m_{c}$
$O$ |  | $AV$ | $b\times m_{z}\times d$ |  | $AV$ | $b\times m_{z}\times d$ |  | $AV$ | $b\times m_{z}\times d$

##  4 Proposed Method

###  4.1 Motivation

Erasing a concept from a foundation model is not a difficult task. As
demonstrated in prior works (Gandikota et al., 2023; Kumari et al., 2023;
Orgad et al., 2023; Zhang et al., 2023), fine-tuning the foundation model to
enforce the output associated with the to-be-erased concept to be close to the
output with a neutral or null concept, can effectively remove the undesirable
concept. However, the parameter space is shared among all concepts, and
semantic-related concepts often trigger highly overlapping sets of parameters.
Therefore, the erasure effect commonly comes with a significant drop in the
model’s capability in other concepts. As later revealed in Section 5.5,
erasing an undesirable concept, even with state-of-the-art methods, still
affects the model’s semantically relevant retaining concepts. This is evident
in the instability of the alignment between visual and textual features in
neutral or retained concepts along the fine-tuning process.

To tackle this challenge, drawing inspiration from the success of parameter
prompt-tuning approaches in Natural Language Processing (Li & Liang, 2021;
Lester et al., 2021; Pfeiffer et al., 2020), we introduce a learnable
parameter prompt as additional memory to the cross-attention layers of the
foundation model. Intuitively, our method involves two alternate processes
named knowledge transfer and knowledge removal, with the additional prompt
acting as a buffer between them. In the knowledge transfer stage, the prompt
is trained to effectively generate undesirable concepts, aiming to resemble
the textual embedding regarding these concepts from the model’s perspective.
As the knowledge of the concepts has been transferred to the prompt, this
stage helps to reduce the dependency of generating these concepts on current
textual inputs. In the knowledge removal stage, assisted by the additional
prompt, when eliminating undesirable concepts, the model’s parameters undergo
fine-tuning more stably and have a minimal negative impact on other concepts.

###  4.2 Knowledge Transfer and Removal with Prompt - KPOP

Let us introduce the terminologies in our paper. Notably, we denote $c$ as the
input description, commonly referred to as the textual prompt in text-to-image
generative models. Additionally, we use $\mathbf{p}$ to represent the
parameter prompt, which we will refer to as the ‘prompt’ for brevity
henceforth.

Let $\epsilon_{\theta}(z_{t},c,t)$ denote the output of the pre-trained
foundation U-Net model parameterized by $\theta$ at step $t$ given an input
description $c$ and the latent vector from the previous step $z_{t}$. Let
$\epsilon_{\theta^{{}^{\prime}}}(z_{t},c,t)$ denote the output of the
sanitized model, parameterized by the to-be-finetuned parameters
$\theta^{{}^{\prime}}$. For the sake of simplicity, we employ the notations
$\epsilon_{\theta}(c)$ and $\epsilon_{\theta^{{}^{\prime}}}(c)$. Let
$\epsilon_{\theta^{{}^{\prime}}}(c,\mathbf{p})$ represent the output of the
sanitized model given the input description $c$ and the parameter prompt
$\mathbf{p}$. The mechanism that allows us to inject the prompt $\mathbf{p}$
in to the cross-attention layer will be discussed later in Section 4.3. It is
worth noting that we can inject the same prompt into multiple cross-attention
layers, thus $\epsilon_{\theta^{{}^{\prime}}}()$ denotes the output of the
entire model, not just a specific layer. Let $c_{e}\in\mathbf{E}$ denote a
textual description in a set of to-be-erased concepts $\mathbf{E}$ and $c_{n}$
represents a neutral or null concept, i.e., ‘a photo’ or ‘ ’.

##### Knowledge Transfer.

Initially, we initialize $\theta^{\prime}_{0}=\theta$ (i.e., the pre-trained
foundation model) and $\mathbf{p_{0}}$. At the iteration $k$, we yield
$\theta^{\prime}_{k}$ and $\mathbf{p_{k}}$ and need to update for the next
iteration. At this stage, we aim to find $\mathbf{p_{k+1}}$ that is not too
far from current $\mathbf{p_{k}}$ and can resemble the undesirable concepts by
minimizing the generation loss as (Ho et al., 2020; Song et al., 2020)

$\underset{\mathbf{p}:\|\mathbf{p}-\mathbf{p}_{k}\|_{2}\leq\rho_{p}}{\text{min}}\;\mathbb{E}_{c_{e}\in\mathbf{E}}\left[\left\|\epsilon_{\theta_{k}^{{}^{\prime}}}(c_{e},\mathbf{p})-\epsilon_{\theta}(c_{e})\right\|_{2}^{2}\right].$
(5)

Here, we note that in Eq. 5 as a result of the knowledge removal stage, the
model $\epsilon_{\theta^{{}^{\prime}}}$ might have already weakened its
knowledge of the undesirable concept, i.e.,
$\epsilon_{\theta^{{}^{\prime}}}(c_{e})$ differs greatly from
$\epsilon_{\theta}(c_{e})$. Therefore, matching
$\epsilon_{\theta^{{}^{\prime}}}(c_{e},\mathbf{p})$ with
$\epsilon_{\theta}(c_{e})$, i.e., to ensure the fine-tuned model together with
the prompt can generate satisfactorily undesirable concepts, allowing the
transfer of knowledge of erasing concepts to the prompt. We apply a one-step
gradient descent to update the prompt as

$\mathbf{p}_{k+1}=\mathbf{p}_{k}-\eta_{p}\nabla_{\mathbf{p}}\mathcal{L}_{e}\left(\theta^{\prime}_{k},\mathbf{p}\right),$
(6)

where
$\mathcal{L}_{e}\left(\theta^{\prime}_{k},\mathbf{p}\right)=\mathbb{E}_{e\in\mathbf{E}}\left[\|\epsilon_{\theta^{\prime}_{k}}\left(c_{e},\mathbf{p}\right)-\epsilon_{\theta}\left(c_{e}\right)\|_{2}^{2}\right]$
and $\eta_{p}$ is the learning rate.

##### Knowledge Removal.

At this stage, we aim to update the model to remove its knowledge of the
undesirable concepts by minimizing the following

$\underset{\theta^{{}^{\prime}}:\|\theta^{{}^{\prime}}-{\theta}_{k}^{{}^{\prime}}\|_{2}\leq\rho}{\min}\;\mathbb{E}_{c_{e}\in\mathbf{E}}\left[\underbrace{\left\|\epsilon_{\theta^{{}^{\prime}}}(c_{e})-\epsilon_{\theta}(c_{n})\right\|_{2}^{2}}_{L1}+\lambda\underbrace{\left\|\epsilon_{\theta^{{}^{\prime}}}(c_{e},\mathbf{p}_{k+1})-\epsilon_{\theta}(c_{e})\right\|_{2}^{2}}_{L2}\right],$
(7)

where we again use one-step gradient descent to update $\theta^{{}^{\prime}}$.

$\theta^{\prime}_{k+1}=\theta^{{}^{\prime}}_{k}-\eta\nabla_{\theta^{{}^{\prime}}}\mathcal{L}_{r}\left(\theta^{{}^{\prime}}\right),$

with
$\mathcal{L}_{r}\left(\theta^{{}^{\prime}}\right)=\mathbb{E}_{c_{e}\in\mathbf{E}}\left[\left\|\epsilon_{\theta^{{}^{\prime}}}(c_{e})-\epsilon_{\theta}(c_{n})\right\|_{2}^{2}+\lambda\left\|\epsilon_{\theta^{{}^{\prime}}}(c_{e},\mathbf{p}_{k+1})-\epsilon_{\theta}(c_{n})\right\|_{2}^{2}\right]$.
Minimizing the loss $L_{1}$ in Eq. 7 directly eliminates the undesirable
concepts associated with $c_{e}$ by mapping it to a neutral or null concept
$c_{n}$. However, it can be seen that optimizing solely for $L_{1}$ does not
consider the impact on other concepts. Consequently, there might be many
solutions $\theta^{{}^{\prime}}$ of $L_{1}$ that may successfully remove the
undesirable concepts, yet they could inadvertently compromise other concepts
due to parameter space overlap among concepts.

Minimizing the loss $L_{2}$ in Eq. 7 provides two benefits. First, it serves
as a crucial regularization component, narrowing down the solution space of
$L_{1}$. Specifically, as a result of the knowledge transfer stage,
$\epsilon_{\theta^{{}^{\prime}}_{k}}(c_{e},\mathbf{p}_{k+1})\approx\epsilon_{\theta}(c_{e})$
with the knowledge of undesirable concepts attributed to the prompt
$\mathbf{p}_{k+1}$. Thus, minimizing $L_{2}$ in Eq. 7 narrows down the
solution space of $L_{1}$ to those that not only map the erased concepts to
the neutral concept but also preserve the knowledge of the prompt. Notably,
compared to the prior work (Orgad et al., 2023), which employs L2
regularization directly to the parameter space
$\left\|\theta^{{}^{\prime}}-\theta\right\|_{2}^{2}$, our method incorporates
regularization on the output space. This implicit consideration takes into
account the varying importance of different parameters to the output,
providing a more nuanced approach.

Second, minimizing the loss $L_{2}$ also helps transfer the dependency of
generating undesirable concepts from the textual input $c_{e}$ to the prompt
$\mathbf{p}$. The model cannot generate the undesirable concepts without the
presence of the prompt $\mathbf{p}$, i.e.,
$\epsilon_{\theta^{{}^{\prime}}}(c_{e})\neq\epsilon_{\theta}(c_{e})$ while
$\epsilon_{\theta^{{}^{\prime}}}(c_{e},\mathbf{p})\approx\epsilon_{\theta}(c_{e})$.
As demonstrated in Section 5.5, we show that the knowledge of undesirable
concepts is gradually transferred to the prompt $\mathbf{p}$ as expected.

###  4.3 Cross-Attention with Prompt

In this section, we will delve into discussing two different mechanisms for
injecting the prompt into the cross-attention module, known as the
concatenative mechanism and the additive mechanism. Their basic operations can
be found in Table 1. In text-to-image diffusion models, the cross-attention
layers are positioned to integrate textual embeddings into visual generation
to regulate the output’s concept. Hence, these specific layers are as the most
suitable for prompt injection, aligning with our goal.

##### Concatenative Mechanism.

In this mechanism, the prompt is concatenated with the textual embedding
$\mathbf{C}$ before being used as the key and value matrix inputs to the
cross-attention module. Let $\mathbf{p}\in\mathbb{R}^{[1,m_{p},d_{p}]}$ be the
additional learnable prompt, where $m_{p}=km_{c}$ is the prompt size and
$d_{p}=d_{c}$ is the dimension of the prompt. The main difference compared to
the original mechanism is the projected matrices $K$ and $V$ as shown in Table
1.

Softmax normalization is applied on the last dimension of the attention score
matrix $\mathbf{A}$. In addition, the scaling factor $\sqrt{d}$ is used to
prevent the attention score from being too small when the dimension of the
latent vector is large (Vaswani et al., 2017).

One of the advantages of this mechanism is that adding the prompt does not
change the mechanism of the cross-attention module, which means that there is
no need for a new architecture and it does not interfere with the model’s
ability to generate good content. We can use the same pre-trained model and
only need to modify the corresponding cross-attention module in the codebase.
With $\mathbf{p}=c$, the model can generate the same output as the original
model.

Secondly, this mechanism allows us to utilize a larger prompt (theoretically,
the prompt can be arbitrarily large). By using a larger prompt, we can either
remove more undesirable content or preserve more desirable content.
Experiments in Section 5.6.2 demonstrate that the performance of this
mechanism is scalable with the size of the prompt.

Figure 1: Illustration of Cross Attention with additional prompt.

Limitation of the concatenative mechanism: The main limitation of the
concatenative mechanism is that it relies on softmax normalization to
distribute the attribution from the additional prompt to the entire textual
path. This issue is also illustrated in Figure 1. Because of the linearity of
the projection operation, the matrices $\mathbf{K}$ and $\mathbf{V}$ can be
decomposed into two parts separately, the projections of original textual
embedding $\mathbf{C}$ (blue color) and prompt $\mathbf{P}$ (orange color),
i.e.,
$\mathbf{K}=[\mathbf{W}_{k}\mathbf{C},\mathbf{W}_{k}\mathbf{P}],\mathbf{V}=[\mathbf{W}_{v}\mathbf{C},\mathbf{W}_{v}\mathbf{P}]$.
As a result, the dot-product between the query $\mathbf{Q}$ and the key
$\mathbf{K}$ can also be decomposed into two disjointed parts, i.e.,
$\mathbf{Q}\mathbf{K}^{T}=[\mathbf{Q}(\mathbf{W}_{k}\mathbf{C})^{T},\mathbf{Q}(\mathbf{W}_{k}\mathbf{P})^{T}]$.
Without the softmax normalization, the output of the cross-attention module
will be just the sum of the two disjointed parts, which is not desirable. The
softmax normalization applied on the last dimension of the dot-product score
matrix $\mathbf{Q}\mathbf{K}^{T}$ helps to distribute the attribution from the
prompt to the entire textual path, enabling the model to attend to the
additional prompt better. However, as the prompt size increases, the softmax
normalization will distribute the attribution from the prompt to the entire
textual path more evenly, which can lead to the model’s inability to attend to
the prompt effectively, as shown in Section 5.6.2.

##### Additive Mechanism.

This mechanism injects the additional prompt by directly adding it to the
textual embedding $\mathbf{C}$ before being used as the key and value matrix
inputs to the cross-attention module. This retains the same advantages as the
concatenative mechanism, i.e., it does not change the mechanism of the cross-
attention module. It also permits a deeper integration of the prompt into the
textual path, which allows the model to attend to the prompt more effectively.
However, a limitation of this mechanism is that it is not scalable since its
size is fixed to the size of the textual embedding. We compare the performance
of two mechanisms in Section 5.6.1.

##### Alternative Prompting Mechanism.

Beyond the two aforementioned mechanisms, we acknowledge that there are
several potential prompting mechanisms that can be used to modify the cross-
attention module. For example, we can inject the prompt before the text
encoder, by using a learnable word embedding vector associated with a special
token $S^{*}$ to represent the prompt as in Textual Inversion (Gal et al.,
2022). We can also amortize the prompt by using a learnable function to
generate the prompt from the textual embedding, i.e.,
$\mathbf{p}=f(\mathbf{c})$. We leave the exploration of these mechanisms for
future work.

##  5 Experiments

###  5.1 General Settings

Our experiments are conducted using Stable Diffusion (SD) version 1.4 as the
foundation model. We employ the same setting across all methods, i.e., fine-
tuning the model for 1000 steps with a batch size of 1, with the Adam
optimizer with a learning rate of $1\mathrm{e}{-5}$. We benchmark our method
against four baseline approaches, namely, the original pre-trained SD model,
ESD (Gandikota et al., 2023), UCE (Gandikota et al., 2024), and Concept
Ablation (CA) (Kumari et al., 2023). Our method (KPOP), in its default
configuration, employs a prompt size of $k=10$ and utilizes the concatenative
mechanism. The effectiveness of these specific settings is discussed in detail
in the ablation study section and the supplementary material.

###  5.2 Erasing Object-Related Concepts

In this experiment, we investigate the ability of our method to erase object-
related concepts from the foundation model, for example, erasing entire object
classes such as ‘Cassette Player’ from the model. We choose Imagenette
111https://github.com/fastai/imagenette which is a subset of the ImageNet
dataset (Deng et al., 2009) which comprises 10 easily recognizable classes,
including ‘Cassette Player’, ‘Chain Saw’, ‘Church’, ‘Gas Pump’, ‘Tench’,
‘Garbage Truck’, ‘English Springer’, ‘Golf Ball’, ‘Parachute’, and ‘French
Horn’.

Since the erasing performance when erasing a single class has been the main
focus of previous work (Gandikota et al., 2023), we choose a more challenging
setting where we erase a set of 5 classes simultaneously. Specifically, we
generate 500 images for each class and employ the pre-trained ResNet-50 (He et
al., 2016) to detect the presence of an object in the generated images. We use
the two following metrics to evaluate the erasing performance: Erasing Success
Rate (ESR-k): The percentage of all the generated images with "to-be-erased"
classes where the object is not detected in the top-k predictions. Presevering
Success Rate (PSR-k): The percentage of all the generated images with all
other classes (i.e., "to-be-preserved") where the object is detected in the
top-k predictions. This dual-metric evaluation provides a comprehensive
assessment of our method’s ability to effectively erase targeted object-
related concepts while also preserving relevant elements.

##### Quantitative Results.

We select four distinct sets of 5 classes from the Imagenette set for erasure
and present the outcomes in Table 2. First, we note that the average PSR-1 and
PSR-5 scores across the four settings of the original SD model stand at 78.0%
and 97.6%, respectively. These scores indicate that 78.0% of the generated
images contain the object-related concepts which are subsequently detected in
the top-1 prediction, and when checking the conepts in any of the top-5
predictions, this number increases to 97.6%. This underscores the original SD
models’s ability to generate images with the anticipated object-related
concepts.

In term of erasing performance, it can be observed that all baselines achieve
very high ESR-1 and ESR-5 scores, with the lowest ESR-1 and ESR-5 scores being
95.5% and 88.9% respectively. This indicates the effectiveness of these
methods to erase object-related concepts, as only a very small proportion of
the generated images contain the object-related concepts under subsequent
detection. Notably, the UCE method can achieve 100% ESR-1 and ESR-5, which is
the highest among the baselines. Our method achieves 99.2% ESR-1 and 97.3%
ESR-5, which is much higher than the two baselines ESD and CA, and only
slightly lower than the UCE method, which is designed specifically for erasing
object-related concepts.

However, despite the high erasing performance, the baselines suffer from a
significant drop in preserving performance, with the lowest PSR-1 and PSR-5
scores being 41.2% and 56.1%, respectively. This suggests that the
preservation task poses greater challenges than the erasing task, and the
baselines are ineffective in retaining other concepts. In contrast, our method
achieves 75.3% PSR-1 and 98.0% PSR-5, which is a significant improvement
compared to the best baseline, UCE, with 62.1% PSR-1 and 96.0% PSR-5. This
result underscores the effectiveness of our method in simultaneously erasing
object-related concepts while preserving other unrelated concepts.

Table 2: Erasing object-related concepts.

Method |  | ESR-1$\uparrow$ |  | ESR-5$\uparrow$ |  | PSR-1$\uparrow$ |  | PSR-5$\uparrow$
---|---|---|---|---|---|---|---|---
SD |  | $22.0\pm 11.6$ |  | $2.4\pm 1.4$ |  | $78.0\pm 11.6$ |  | $97.6\pm 1.4$
ESD |  | $95.5\pm 0.8$ |  | $88.9\pm 1.0$ |  | $41.2\pm 12.9$ |  | $56.1\pm 12.4$
CA |  | $98.4\pm 0.3$ |  | $96.8\pm 6.1$ |  | $44.2\pm 9.7$ |  | $66.5\pm 6.1$
UCE |  | $100\pm 0.0$ |  | $100\pm 0.0$ |  | $62.1\pm 34.6$ |  | $96.0\pm 2.9$
Ours |  | $99.2\pm 0.5$ |  | $97.3\pm 1.9$ |  | $75.3\pm 12.0$ |  | $98.0\pm 0.5$

##### Visualizing Attribution Maps.

To gain deeper insights into the behavior of our method, we leverage DAAM
(Tang et al., 2022) to visualize the attentive attribution maps that depict
the interaction between visual and textual concepts in the generated images.
DAAM is an emerging technique that interprets how an input word influences
parts of the generated image by analyzing the attention maps in the cross-
attention module of the Stable Diffusion model.

We first use DAAM to analyze the original SD model’s behavior in generating
images with "Cassette Player" and "English Springer" as input prompts as shown
in Figure 2. Each test case comprises eight sub-figures, each of which
corresponds to a head in the multihead cross-attention module. As depicted in
Figure 2, most heatmaps concentrate on the cassette player and the dog’s body,
aligning well with the respective textual prompts. Interestingly, the second
head does not focus on the cassette player or the dog’s body but instead on
the surrounding background.

We then utilize DAAM to visualize the attribution maps of generated images
using the same prompts with our method. We find that on the concept to be
retained (i.e., "English Springer"), the heatmaps also focus on the dog’s body
except for the second head, mirroring the behavior observed in the original SD
model. For the concept to be erased (i.e., "Cassette Player"), the heatmaps
exhibit a more dispersed pattern, indicating a lack of specific concentration
on any distinct region. This observation suggests that the model, under the
erasure effect of our method, diverts attention away from the cassette player
concept, providing valuable insights into the underlying mechanism of our
method.

Figure 2: Attentive attribution maps between the visual and textual concepts
in the original SD model and our method.

###  5.3 Mitigating Unethical Content

One of the serious concerns associated with the deployment of text-to-image
generative models to the public domain is their potential to generate Not-
Safe-For-Work (NSFW) content. This ethical challenge has become a primary
focus in recent works (Schramowski et al., 2023; Gandikota et al., 2023,
2024), aiming to sanitize such capability of the model before public release.

In contrast to object-related concepts, such as "Cassette Player" or "English
Springer", which can be explicitly described with limited textual
descriptions, i.e., there are only a few textual ways to describe the visual
concepts, unethical concepts like nudity are indirectly expressible in textual
descriptions. The multiple ways a single visual concept can be described make
erasing such concepts challenging, especially when relying solely on a keyword
to indicate the concept to be erased. As empirically shown in Gandikota et al.
(2023), the erasing performance on these concepts is highly dependent on the
subset of parameters that are finetuned. Specifically, fine-tuning the non-
cross-attention modules has shown to be more effective than fine-tuning the
cross-attention modules. Therefore, in this experiment, we follow the same
configuration as in Gandikota et al. (2023), focusing exclusively on fine-
tuning the non-cross-attention modules.

##### Quantitative Results.

To generate NSFW images, we employ I2P prompts (Schramowski et al., 2023) and
generate a dataset comprising 4703 images with attributes encompassing sexual,
violent, and racist content. We then utilize the detector (Praneet, 2019)
which can accurately detect several types of exposed body parts to recognize
the presence of the nudity concept in the generated images. The detector
(Praneet, 2019) provides multi-label predictions with associated confidence
scores, allowing us to adjust the threshold and control the trade-off between
the number of detected body parts and the confidence of the detection, i.e.,
the higher the threshold, the fewer the number of detected body parts.

Figure 3(a) illustrates the ratio of images with any exposed body parts
detected by the detector (Praneet, 2019) over the total 4703 generated images
(denoted by NER) across thresholds ranging from 0.3 to 0.8. Notably, our
method consistently outperforms the baselines under all thresholds, showcasing
its effectiveness in erasing NSFW content. In particular, with the threshold
set at 0.3, the NER score for the original SD model stands at 16.7%,
indicating that 16.7% of the generated images contain signs of nudity concept
from the detector’s perspective. The two baselines, ESD and UCE, achieve 5.32%
and 6.87% NER with the same threshold, respectively, demonstrating their
effectiveness in erasing nudity concepts. Our method achieves 3.95% NER, the
lowest among the baselines, indicating the highest erasing performance. This
result remains consistent across different thresholds, emphasizing the
robustness of our method in erasing NSFW content. Additionally, to measure the
preserving performance, we generate images with COCO 30K prompts and measure
the FID score compared to COCO 30K validation images. Our method achieves an
FID score of 16.73, slightly lower than that of ESD, which is the highest
score at 15.98, indicating that our method can simultaneously erase a concept
while preserving other concepts effectively.

(a)

(b)

Figure 3: Comparison of the erasing performance on the I2P dataset. 3(a):
Number of exposed body parts counted in all generated images with threshold
0.5. 3(b): Ratio of images with any exposed body parts detected by the
detector (Praneet, 2019).

Detailed statistics of different exposed body parts in the generated images
are provided in Figure 3(b). It can be seen that in the original SD model,
among all the body parts, the female breast is the most detected body part in
the generated images, accounting for more than 320 images out of the total
4703 images. Both baselines, ESD and UCE, as well as our method, achieve a
significant reduction in the number of detected body parts, with our method
achieving the lowest number among the baselines. Our method also achieves the
lowest number of detected body parts for the most sensitive body parts, only
surpassing the baseline for less sensitive body parts, such as feet.

Table 3: Evaluation on the nudity erasure setting.

| NER-0.3$\downarrow$ | NER-0.5$\downarrow$ | NER-0.7$\downarrow$ | NER-0.8$\downarrow$ | FID$\downarrow$
---|---|---|---|---|---
CA | 13.84 | 9.27 | 4.74 | 1.68 | 20.76
UCE | 6.87 | 3.42 | 0.68 | 0.21 | 15.98
ESD | 5.32 | 2.36 | 0.74 | 0.23 | 17.14
Ours | 3.95 | 1.70 | 0.40 | 0.0 | 16.73

###  5.4 Erasing Artistic Style Concepts

In this experiment, we investigate the ability of our method to erase artistic
style concepts from the foundation model. We choose several famous artists
with easily recognizable styles who have been known to be mimicked by the
text-to-image generative models, including "Kelly Mckernan", "Thomas Kinkade",
"Tyler Edlin" and "Kilian Eng" as in Gandikota et al. (2023). We compare our
method with recent work including ESD (Gandikota et al., 2023), UCE (Gandikota
et al., 2024), and CA (Kumari et al., 2023) which have demonstrated
effectiveness in similar settings.

For fine-tuning the model, we use only the names of the artists as inputs. For
evaluation, we use a list of long textual prompts that are designed
exclusively for each artist, combined with 5 seeds per prompt to generate 200
images for each artist across all methods. We measure the CLIP alignment score
222https://lightning.ai/docs/torchmetrics/stable/multimodal/clip_score.html
between the visual features of the generated image and its corresponding
textual embedding. Compared to the setting (Gandikota et al., 2023) which
utilized a list of generic prompts, our setting with longer specific prompts
can leverage the CLIP score as a more meaningful measurement to evaluate the
erasing and preserving performance. We also use LPIPS (Zhang et al., 2018) to
measure the distortion in generated images by the original SD model and
editing methods, where a low LPIPS score indicates less distortion between two
sets of images.

It can be seen from Table 4 that our method achieves the best erasing
performance while maintaining a comparable preserving performance compare to
the baselines. Specifically, our method attains the lowest CLIP score on the
to-be-erased sets at 21.24, outperforming the second-best score of 23.56
achieved by ESD. Additionally, our method secures a 0.79 LPIPS score, the
second-highest, following closely behind the CA method with 0.82. Concerning
preservation performance, we observe that, while our method achieves a
slightly higher LPIPS score than the ESD and UCE methods, suggesting some
alterations compared to the original images generated by the SD model, the
CLIP score of our method remains comparable to these baselines. This implies
that our generated images still align well with the input prompt.

Table 4: Erasing artistic style concepts.

|  | To Erase |  | To Retain
---|---|---|---|---
|  | CLIP $\downarrow$ | LPIPS$\uparrow$ |  | CLIP$\uparrow$ | LPIPS$\downarrow$
ESD |  | $23.56\pm 4.73$ | $0.72\pm 0.11$ |  | $29.63\pm 3.57$ | $0.49\pm 0.13$
CA |  | $27.79\pm 4.67$ | $0.82\pm 0.07$ |  | $29.85\pm 3.78$ | $0.76\pm 0.07$
UCE |  | $24.47\pm 4.73$ | $0.74\pm 0.10$ |  | $30.89\pm 3.56$ | $0.40\pm 0.13$
Ours |  | $21.24\pm 5.56$ | $0.79\pm 0.10$ |  | $29.57\pm 3.72$ | $0.51\pm 0.14$

###  5.5 Understanding the Prompting Mechanism

(a)

(b)

(c)

Figure 4: Prompt’s learning process (4(a)) and the cosine similarity between
visual and textual features in our method (4(b)) and ESD (4(c)), respectively.

In this section, we aim to investigate the behavior of the prompting mechanism
in our method, to further provide insights into the underlying mechanism of
our method. We first analyze the learning process of the prompt, by measuring
the cosine similarity between the prompt and several related textual inputs
along the fine-tuning process. As depicted in Figure 4(a), initially, the
prompt exhibits no alignment with any textual inputs. However, through the
fine-tuning process, it progressively aligns more closely with the most
relevant ones, like ‘nudity’ or ‘a nude person’, while maintaining an
uncorrelated relationship with more neutral expressions like ‘a person’ or ‘a
face’. Intriguingly, although we explicitly enforce alignment with the keyword
$c_{e}$ ‘nudity’, the prompt aligns most with ‘a nude person’, suggesting that
it has captured the concept of nudity in a more specific sense, specifically
referring to a person.

Next, we generate images $x_{e}$ and $x_{n}$ with the textual input
$c_{e}=\text{`nudity'}$ and $c_{n}=\text{`a person'}$ respectively. We then
measure the alignment between the CLIP visual and textual features of these
images and their corresponding textual inputs. As illustrated in Fig. 4(b),
throughout the learning process of the prompt, there is a decline in the
alignment between $c_{e}$ and $x_{e}$, indicating that the keyword $c_{e}$
becomes less capable of generating images with the erased concept. Conversely,
the alignment between $c_{e}$ and $x_{n}$ increases, suggesting that the
keyword $c_{e}$ becomes more adept at generating images with neutral concepts.
Additionally, the alignment between $c_{n}$ and $x_{n}$ also increases,
highlighting the preserving effect of our method. In contrast, the alignment
between pairs in the ESD method remains unstable over the learning process, as
depicted in Fig. 4(c), underscoring the instability of the erasure effect in
ESD compared to ours.

###  5.6 Ablation Study

####  5.6.1 Concatenative vs Additive Mechanism

In this experiment, we conducted a comparison of erasing performance between
two mechanisms. The evaluation was performed on a subset of 5 classes from the
Imagenette dataset, including ‘Cassette Player’, ‘Church’, ‘Garbage Truck’,
‘Parachute’, and ‘French Horn’. Additionally, we assessed the erasing
performance in a nudity concept setting using the I2P dataset with the NER at
a threshold of 0.5 as the erasure metric.

The results, presented in Table 5, indicate that the concatenative prompting
mechanism outperforms the additive prompting mechanism in terms of erasing
performance. This is evident in the 2.44% increase in ESR-1 and 3.16% increase
in ESR-5, as well as a 0.3% decrease in NER. However, it is worth noting that
the concatenative prompting mechanism is less effective in preserving
unrelated concepts, as indicated by a drop of 2.8% in PSR-1 compared to the
additive prompting mechanism.

While additive prompting theoretically provides a deeper integration between
the prompt and the real textual input, the concatenative prompting mechanism
has demonstrated greater effectiveness in erasing the target concept.
Furthermore, its scalability allows for varying prompt sizes, a feature
discussed in the next section. As a result of its superior erasing
performance, we adopt the concatenative prompting mechanism as the default
setting in all other experiments.

Table 5: Analytical results to different prompting mechanisms and prompt size.

Method | ESR-1$\uparrow$ | ESR-5$\uparrow$ | PSR-1$\uparrow$ | PSR-5$\uparrow$ | NER$\downarrow$
---|---|---|---|---|---
Additive | 96.40 | 92.32 | 84.48 | 97.92 | 1.7
Concat | 98.84 | 95.48 | 81.68 | 97.56 | 2.0
k=1 | 98.60 | 96.04 | 84.76 | 97.56 | 2.17
k=10 | 98.84 | 95.48 | 81.68 | 97.56 | 1.70
k=100 | 99.68 | 97.08 | 82.68 | 96.84 | 1.15
k=200 | 99.60 | 96.80 | 77.24 | 94.16 | 1.49

####  5.6.2 Effect of Prompt Size

In this experiment, we explore the influence of prompt size on erasing
performance by systematically varying the parameter $k$ within the range of 1
to 200. The experimental setup mirros the previous experiment, where we focus
on erasing object-related concepts with 5 classes from the Imagenette dataset,
and also erasing nudity concepts with the I2P dataset.

It can be seen from the results in Table 5 that the erasing performance
increases as the prompt size becomes larger, but becomes saturated once the
prompt size becomes sufficiently large. Specifically, ESR-1 improves from
98.60% to 99.68%, and ESR-5 from 96.04% to 97.08% as the prompt size increases
from 1 to 100. Similarly, the NER score decreases from 2.2 to 1.1 within the
same range, indicating a consistent impact of prompt size across different
types of concepts. However, the erasing performance is accompanied by a trade-
off in preserving performance, as PSR-1 decreases from 84.76% to 77.24% when
the prompt size increases from 1 to 200. The observed saturation in erasing
performance for larger prompt sizes can be attributed to the softmax function
in the cross-attention module, which becomes increasingly uniform and small as
the prompt size grows. This makes it more challenging for the model to
distinctly focus on the specific concept for erasure.

####  5.6.3 Influence of Hyper-parameter

In this experiment, we investigate the influence of the hyper-parameter
$\lambda$ on erasing performance of our method. We conduct the experiment to
erase object-related concepts, using the same experimental setup as described
in Section 4.6 of the main paper. Specifically, we focus on a subset of 5
classes from the Imagenette dataset, including ‘Cassette Player’, ‘Church’,
‘Garbage Truck’, ‘Parachute’, and ‘French Horn’ as the concepts to be erased,
while preserving the remaining classes. We vary the hyper-parameter $\lambda$
from 0.01 to 10.0. It is worth noting that the hyper-parameter $\lambda$ is
utilized to introduce the prompt in the knowledge removal stage, as described
in Section 3.2 of the main paper. Therefore, it must be strictly positive,
i.e., $\lambda>0$.

The results depicted in Figure 5 reveal a clear decreasing trend in erasing
performance as the hyper-parameter $\lambda$ increases, while the preserving
performance exhibits a slight increase. Specifically, the erasing performance
peaks at $\lambda=0.01$, with an ESR-5 of 96.8%, and drops significantly to
80.6% at $\lambda=1.0$, and to around 40% at $\lambda=10.0$. Conversely, the
preserving performance, measured by PSR-5, increases from 97.3% to 98.0% as
$\lambda$ increases. The PSR-1 also exhibits a similar trend, increasing from
81.6% to around 86% as $\lambda$ varies from 0.01 to 10.0.

This result aligns with our analysis in Section 3.2 of the main paper, where
the hyper-parameter $\lambda$ is employed to control the trade-off between
erasing and preserving performance. In the knowledge removal stage, the
$L_{2}$ term serves as a regularization term to minimize the change in the
model’s parameters, thereby preserving the knowledge encoded in the prompt of
the erased concepts learned from the knowledge transfer stage. Therefore, a
larger $\lambda$ encourages the model to preserve the knowledge in the prompt
more strongly, leading to smaller changes in the model’s parameters and better
preserving performance, but worse erasing performance. In other experiments,
we use $\lambda=0.1$ as the default value for the hyper-parameter $\lambda$.

Figure 5: Impact of the hyper-parameter $\lambda$ on the erasing performance.

####  5.6.4 Recover the Erased Concepts

As demonstrated in Section 4.5 of the main paper, the prompt captures the
knowledge of the erased concepts through the fine-tuning process.
Consequently, in addition to the erasure effect on the model, wherein the
model loses the ability to generate the erased concepts, our method exhibits
an intriguing property: the erased concepts can be recovered using the prompt.
Intuitively, the prompt acts as a hidden key that unlocks a backdoor in the
model, enabling the recovery of the ability to generate prohibited content.

It is important to note that the prompt remains hidden when releasing the
sanitized models to the public. Therefore, public users cannot generate the
erased concepts, aligning with the primary purpose of our method. However, in
this section, we demonstrate that the erased concepts can be recovered using
the prompt. We utilize the pre-trained prompt from previous experiments in
Section 5.6.3 to generate images of to-be-erased concepts, including ‘Cassette
Player’, ‘Truck’, and ‘Church’.

The results depicted in Figure 6 showcase several representative examples of
the generated images from the sanitized models and the same models but with
the hidden prompt utilized to generate the images. With $\lambda=1.0$, while
the sanitized models fail to generate ‘Cassette Player’ or ‘Church’, with the
support of the hidden prompt, we can still generate images of these concepts
even with the sanitized models. However, as discussed in Section 5.6.3, a
larger $\lambda$ encourages the model to preserve the knowledge in the prompt
more strongly, therefore, the recovered images become more similar to the
original images.

(a) Cassette Player

(b) Truck

(c) Church

Figure 6: Recovering erased concepts with hidden prompt $\mathbf{p}$. The
first row shows the generated images from sanitized models. The second row
shows those from the same models but with the hidden prompt $\mathbf{p}$ used
to generate the images.

####  5.6.5 Where to Inject Prompt

In this paper, we have introduced two different prompting mechanisms:
concatenative and additive prompting. While in the additive prompting, we add
the prompt $\mathbb{p}$ directly to the textual embedding $c$ as in Table 1,
which can be understood as injecting the prompt in entire cross-attention
layers of the U-Net, in the concatenative prompting, we need to specify where
to inject the prompt.

In this experiment, we explore the influence of prompt injection at different
layers of the model on erasing performance within the U-Net architecture of
the Stable Diffusion model. The U-Net comprises three main components: the
down-sample blocks, the middle block, and the up-sample blocks. Each of these
components includes multiple cross-attention layers that can be used to inject
the textual/conditional input, except for the middle block, which contains
only one cross-attention layer.

Table 6: Where to inject the prompt.

layer | ESR-1$\uparrow$ | ESR-5$\uparrow$ | PSR-1$\uparrow$ | PSR-5$\uparrow$
---|---|---|---|---
mid | 98.84 | 95.48 | 81.68 | 97.56
mid-up | 99.64 | 97.92 | 75.68 | 95.12
down-mid-up | 99.65 | 98.36 | 59.04 | 86.28

We compare three different settings: injecting the prompt at the middle block,
the middle and up-sample blocks, and the down-middle-up sample blocks (i.e.,
all cross-attention layers in the U-Net), with the same experimental setup as
in previous experiments. The results in Table 6 indicate that injecting the
prompt at all cross-attention layers in the U-Net yields the best erasing
performance, albeit with a significant drop in preserving performance.

It is noteworthy that as the number of cross-attention layers used for prompt
injection increases, erasing performance improves at the expense of preserving
performance. The optimal trade-off between erasing and preserving performance
is achieved by injecting the prompt at the middle block only. This setting was
consequently chosen as the default for all subsequent experiments. It strikes
a balance, demonstrating effective erasure while still preserving relevant
elements in the input.

####  5.6.6 Further Results on Erasing Artistic Style Concepts

##### How to systematically evaluate the erasure performance?

In this paper, we have conducted three sets of experiments to assess the
performance of our proposed method against other erasure baselines. In the
initial two sets, targeting the erasure of object-related and nudity concepts,
we employed pre-trained detectors like ResNet-50 (He et al., 2016) and Nudenet
(Praneet, 2019) to identify the presence of these concepts in the generated
images. This systematic approach enabled us to evaluate the erasure
performance rigorously. However, in the final experiment set, aimed at erasing
artistic style concepts, we encountered a challenge: the absence of a pre-
trained detector capable of accurately assessing the presence of an artistic
style in the generated images. To address this challenge, previous studies
(Gandikota et al., 2023) proposed human evaluations, which are subjective and
time-consuming.

Table 7: CLIP alignment score measured on the original SD model.

| Content & Artist |  |  | Artist |  |  | Content
---|---|---|---|---|---|---|---
Kelly McKernan | $31.47\pm 2.58$ |  |  | $27.67\pm 2.73$ |  |  | $29.69\pm 2.43$
Tyler Edlin | $30.63\pm 2.22$ |  |  | $23.67\pm 1.24$ |  |  | $30.12\pm 2.49$
Kilian Eng | $29.87\pm 2.64$ |  |  | $25.08\pm 1.31$ |  |  | $30.54\pm 2.36$
$\text{Thomas Kinkade}^{\star}$ | $34.63\pm 1.96$ |  |  | $31.13\pm 2.38$ |  |  | $31.09\pm 2.22$
$\text{Ajin: Demi Human}^{\star}$ | $30.70\pm 2.55$ |  |  | $27.65\pm 3.24$ |  |  | $25.38\pm 2.77$
$\text{VanGogh}^{\star}$ | $33.66\pm 2.41$ |  |  | $30.36\pm 1.17$ |  |  | $28.62\pm 3.28$

In this experiment, we explored the use of the CLIP alignment score as an
alternative metric to evaluate erasure performance. Initially, we generated
1200 images from the original SD model using lengthy, specifically designed
prompts (credited to (Gandikota et al., 2023)) to capture images with the
artistic style of a particular artist. Subsequently, we measured the CLIP
alignment score between the generated images and three different textual
inputs: the full prompt containing both the content and the artist name, the
artist name alone, and the content alone. The results presented in Table 7
revealed intriguing insights. On one hand, when measuring based solely on the
artist name, the CLIP alignment score was consistently the lowest in all
cases, except for Thomas Kinkade, Ajin: Demi Human, and VanGogh. Conversely,
when measuring based solely on the content, the CLIP alignment score was
relatively higher. Lastly, when considering the full prompt, inclusive of both
the content and the artist name, the CLIP alignment score was consistently the
highest in all cases, except for Kilian Eng. This suggests that, from the
CLIP’s perspective, the generated images may not align well with just the
artist name, but they exhibit strong alignment with the full prompt,
encompassing both the content and the artist name. Consequently, to evaluate
erasure performance, we can leverage CLIP as a zero-shot classifier, as
highlighted in Section 5.4.

##### Qualitative Results.

In addition to the quantitative results reported in Section 5.4, we provide
further qualitative results as shown in Figures 7 and 8 to illustrate the
erasure performance of our method and the baselines.

(a) Ours

(b) ESD

(c) UCE

(d) CA

Figure 7: Erasing artistic style concepts. Each column represents the erasure
of a specific artist, except the first column which represents the generated
images from the original SD model. Each row represents the generated images
from the same prompt but with different artists. The ideal erasure should
result in the change in the diagonal pictures compared to the first column,
while the off-diagonal pictures should remain the same.

(a) Ours

(b) ESD

(c) UCE

(d) CA

Figure 8: Erasing artistic style concepts (continue). Each column represents
the erasure of a specific artist, except the first column which represents the
generated images from the original SD model. Each row represents the generated
images from the same prompt but with different artists. The ideal erasure
should result in the change in the diagonal pictures compared to the first
column, while the off-diagonal pictures should remain the same.

##  6 Conclusion

In this paper, we have introduced a novel approach to concept erasure in text-
to-image generative models by incorporating an additional learnable parameter
prompt. This prompt helps reduce the model’s dependency on generating
undesirable concepts, thereby minimizing the negative impact on other
unrelated concepts during the erasure process, resulting in better performance
in both erasing and preserving aspects as demonstrated through extensive
experiments in our paper. Furthermore, our proposed prompting mechanism
exhibits high flexibility and can be extended to address other challenges
involving cross-attention layers, such as continual learning. Additionally,
exploring more complex prompting mechanisms, such as amortizing the prompt
using a learnable function of textual embeddings, presents promising avenues
for future research.

## References

  * Brendel et al. (2017) Wieland Brendel, Jonas Rauber, and Matthias Bethge.  Decision-based adversarial attacks: Reliable attacks against black-box machine learning models.  _arXiv preprint arXiv:1712.04248_ , 2017.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 248–255, 2009.  doi: 10.1109/CVPR.2009.5206848.
  * Gal et al. (2022) Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022.
  * Gandikota et al. (2024) Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńska, and David Bau.  Unified concept editing in diffusion models.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 5111–5120, 2024.
  * Gandikota et al. (2023) Rohit Gandikota et al.  Erasing concepts from diffusion models.  _ICCV_ , 2023.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 770–778, 2016.  doi: 10.1109/CVPR.2016.90.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.  URL https://openreview.net/forum?id=nZeVKeeFYf9.
  * Kumari et al. (2023) Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu.  Ablating concepts in text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 22691–22702, 2023.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 2021.  doi: 10.18653/v1/2021.emnlp-main.243.
  * Li & Liang (2021) Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  _CoRR_ , abs/2101.00190, 2021.
  * Orgad et al. (2023) Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov.  Editing implicit assumptions in text-to-image diffusion models.  In _IEEE International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023_ , pp. 7030–7038. IEEE, 2023.  doi: 10.1109/ICCV51070.2023.00649.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych.  Adapterfusion: Non-destructive task composition for transfer learning.  _CoRR_ , abs/2005.00247, 2020.
  * Praneet (2019) Bedapudi Praneet.  Nudenet: Neural nets for nudity classification, detection and selective censorin.  2019\.
  * Qu et al. (2023) Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang.  Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models.  _arXiv preprint arXiv:2305.13873_ , 2023.
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pp. 8821–8831. PMLR, 2021.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Rando et al. (2022) Javier Rando et al.  Red-teaming the stable diffusion safety filter.  _NeurIPS Workshop MLSW_ , 2022.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10684–10695, 2022.
  * Ruiz et al. (2023) Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 22500–22510, 2023.
  * Schramowski et al. (2023) Patrick Schramowski et al.  Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models.  In _CVPR_ , 2023.
  * Song et al. (2020) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020.
  * StabilityAI (2022) StabilityAI.  Stable diffusion 2.0 release.  2022\.  URL https://stability.ai/blog/stable-diffusion-v2-release.
  * Tang et al. (2022) Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture.  What the daam: Interpreting stable diffusion using cross attention.  _arXiv preprint arXiv:2210.04885_ , 2022.
  * Van Le et al. (2023) Thanh Van Le, Hao Phung, Thuan Hoang Nguyen, Quan Dao, Ngoc N Tran, and Anh Tran.  Anti-dreambooth: Protecting users from personalized text-to-image synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 2116–2127, 2023.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. (2021) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou.  K-adapter: Infusing knowledge into pre-trained models with adapters, 2021.
  * Westerlund (2019) Mika Westerlund.  The emergence of deepfake technology: A review.  _Technology innovation management review_ , 9(11), 2019.
  * Yang et al. (2024) Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao.  Sneakyprompt: Jailbreaking text-to-image generative models.  In _2024 IEEE Symposium on Security and Privacy (SP)_ , pp. 123–123. IEEE Computer Society, 2024.
  * Zhang et al. (2023) Eric Zhang et al.  Forget-me-not: Learning to forget in text-to-image diffusion models.  _arXiv preprint arXiv:2303.17591_ , 2023.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 586–595, 2018.