# Safe Latent Diffusion:
Mitigating Inappropriate Degeneration in Diffusion Models

Patrick Schramowski1,2,3,6 Equal contribution Manuel Brack1,311footnotemark: 1
Björn Deiseroth2,3,5 Kristian Kersting1,2,3,4 1DFKI, 2Hessian.AI, 3Computer
Science Department, TU Darmstadt
4Centre for Cognitive Science, TU Darmstadt, 5Aleph Alpha, 6LAION
{schramowski, brack, deiseroth<EMAIL_ADDRESS>
###### Abstract

Text-conditioned image generation models have recently achieved astonishing
results in image quality and text alignment and are consequently employed in a
fast-growing number of applications. Since they are highly data-driven,
relying on billion-sized datasets randomly scraped from the internet, they
also suffer, as we demonstrate, from degenerated and biased human behavior. In
turn, they may even reinforce such biases. To help combat these undesired side
effects, we present safe latent diffusion (SLD). Specifically, to measure the
inappropriate degeneration due to unfiltered and imbalanced training sets, we
establish a novel image generation test bed—inappropriate image prompts
(I2P)—containing dedicated, real-world image-to-text prompts covering concepts
such as nudity and violence. As our exhaustive empirical evaluation
demonstrates, the introduced SLD removes and suppresses inappropriate image
parts during the diffusion process, with no additional training required and
no adverse effect on overall image quality or text alignment.111Code available
at https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion_safe

Warning: This paper contains sexually explicit imagery, discussions of
pornography, racially-charged terminology, and other content that some readers
may find disturbing, distressing, and/or offensive. ††footnotetext:
Proceedings of the 22nd IEEE/CVF Conference on Computer Vision and Pattern
Recognition, 2023

##  1 Introduction

The primary reasons for recent breakthroughs in text-conditioned generative
diffusion models (DM) are the quality of pre-trained backbones’
representations and their multimodal training data. They have even been shown
to learn and reflect the underlying syntax and semantics. In turn, they retain
general knowledge implicitly present in the data [27]. Unfortunately, while
they learn to encode and reflect general information, systems trained on
large-scale unfiltered data may suffer from degenerated and biased behavior.
While these profound issues are not completely surprising—since many biases
are human-like [8, 6]—many concerns are grounded in the data collection
process failing to report its own bias [14]. The resulting models, including
DMs, end up reflecting them and, in turn, have the potential to replicate
undesired behavior [13, 1, 3, 18, 4, 5]. Birhane et al. [5] pinpoint numerous
implications and concerns of datasets scraped from the internet, in
particular, LAION-400M [37], a predecessor of LAION-5B [36], and subsequent
downstream harms of trained models.

Figure 1: Mitigating inappropriate degeneration in diffusion models. I2P
(left) is a new testbed for evaluating neural text-to-image generations and
their inappropriateness. Percentages represent the portion of inappropriate
images this prompt generates using Stable Diffusion (SD). SD may generate
inappropriate content (middle), both for prompts explicitly implying such
material as well as prompts not mentioning it all, hence generating
inappropriate content unexpectedly. Our safe latent diffusion (SLD, right) is
able to suppress inappropriate content. (Best viewed in color)

We analyze the open-source latent diffusion model Stable Diffusion (SD), which
is trained on subsets of LAION-5B [36] and find a significant amount of
inappropriate content generated which, viewed directly, might be offensive,
ignominious, insulting, threatening, or might otherwise cause anxiety. To
systematically measure the risk of inappropriate degeneration by pre-trained
text-to-image models, we provide a test bed for evaluating inappropriate
generations by DMs and stress the need for better safety interventions and
data selection processes for pre-training. We release I2P (Sec. 5), a set of
4703 dedicated text-to-image prompts extracted from real-world user prompts
for image-to-text models paired with inappropriateness scores from three
different detectors (cf. Fig. 1). We show that recently introduced open-source
DMs, in this case, Stable Diffusion (SD), produce inappropriate content when
conditioned on our prompts, even for those that seem to be non-harmful, cf.
Sec. 6. Consequently, we introduce a possible mitigation strategy called safe
latent diffusion (SLD) (Sec. 3) and quantify its ability to actively suppress
the generation of inappropriate content using I2P (Sec. 6). SLD requires no
external classifier, i.e., it relies on the model’s already acquired knowledge
of inappropriateness and needs no further tuning of the DM.

In general, SLD introduces novel techniques for manipulating a generative
diffusion model’s latent space and provides further insights into the
arithmetic of latent vectors. Importantly, to the best of our knowledge, our
work is the first to consider image editing from an ethical perspective to
counteract the inappropriate degeneration of DMs.

##  2 Risks and Promises of Unfiltered Data

Let us start discussing the risks but also promises of noisy, unfiltered and
large-scale datasets, including background information on SD and its training
data.

Risks. Unfortunately, while modern large-scale models, such as GPT-3 [7],
learn to encode and reflect general information, systems trained on large-
scale unfiltered data also suffer from degenerated and biased behavior.
Nonetheless, computational systems were promised to have the potential to
counter human biases and structural inequalities [19]. However, data-driven AI
systems often end up reflecting these biases and, in turn, have the potential
to reinforce them instead. The associated risks have been broadly discussed
and demonstrated in the context of large-scale models [13, 1, 3, 18, 4, 5].
These concerns include, for instance, models producing stereotypical and
derogatory content [3] and gender and racial biases [38, 24, 41, 10].
Subsequently, approaches have been developed to, e.g., decrease the level of
bias in these models [6, 39].

Promises. Besides the performance gains, large-scale models show surprisingly
strong abilities to recall factual knowledge from the training data [27]. For
example, Roberts et al. [30] showed that large-scale pre-trained language
models’ capabilities to store and retrieve knowledge scale with model size.
Grounded on those findings, Schick et al. [32] demonstrated that language
models can self-debias the text they produce, specifically regarding toxic
output. Furthermore, Jenetzsch et al. [21] as well as Schramowski et al. [35]
showed that the retained knowledge of such models carries information about
moral norms aligning with the human sense of “right” and “wrong” expressed in
language. Similarly, other research demonstrated how to utilize this knowledge
to guide autoregressive language models’ text generation to prevent their
toxic degeneration [32, 34]. Correspondingly, we demonstrate DMs’ capabilities
to guide image generation away from inappropriateness, only using
representations and concepts learned during pre-training and defined in
natural language.

This makes our approach related to other techniques for text-based image
editing on diffusion models such as Text2LIVE [2], Imagic [23] or UniTune
[40]. Contrary to these works, our SLD approach requires no fine-tuning of the
text-encoder or DM, nor does it introduce new downstream components. Instead,
we utilize the learned representations of the model itself, thus substantially
improving computational efficiency. Previously, Prompt-to-Prompt [15] proposed
a text-controlled editing technique using changes to the text prompt and
control of the model’s cross-attention layers. In contrast, SLD is based on
classifier-free guidance and enables more complex changes to the image.

LAION-400M and LAION-5B. Whereas the LAION-400M [37] dataset was released as a
proof-of-concept, the creators took the raised concern [5] to heart and
annotated potential inappropriate content in its successor dataset of LAION-5B
[36]. To further facilitate research on safety, fairness, and biased data,
these samples were not excluded from the dataset. Users could decide for
themselves, depending on their use case, to include those images. Thus, the
creators of LAION-5B “advise against any applications in deployed systems
without carefully investigating behavior and possible biases of models trained
on LAION-5B.”

Training Stable Diffusion. Many DMs have reacted to the concerns raised on
large-scale training data by either not releasing the model [31], only
deploying it in a controlled environment with dedicated guardrails in place
[29] or rigorously filtering the training data of the published model [25]. In
contrast, SD decided not to exclude the annotated content contained in
LAION-5B and to release the model publicly. Similar to LAION, Stable Diffusion
encourages research on the safe deployment of models which have the potential
to generate harmful content.

Specifically, SD is trained on a subset of LAION-5B, namely LAION-2B-en [36]
containing over 2.32 billion English image-text pairs. Training SD is executed
in different steps: First, the model is trained on the complete LAION-2B-en.
Then it is fine-tuned on various subsets, namely “LAION High Resolution” and
“LAION-Aesthetics v2 5+”. With all training samples taken from LAION-5B or
subsets thereof, it is expected that the trained model reflects not only
human-like biases such as gender occupation correlations but also reporting
biases. Furthermore, SD is deployed on several platforms including
huggingface††https://huggingface.co/spaces and recently
lexica.art††https://lexica.art making it easy to use for the general public,
including users unaware of present issues.

Ethnic Bias. This leads us to our first experiment. Following up the studies
by Birhane et al. on unfiltered multimodal datasets [5], we extend these
investigations by answering whether the same issues occur in downstream
generative models. Specifically, we chose to showcase biased representation of
Asian women [42], cf. also Sec. 2.1 and Appendix A in [5] wrt. LAION-400M.
This social phenomenon is a well-documented trend in western countries [42]
that is also reflected in the English-speaking internet [20] and subsequently
the web-crawled LAION-2B-en. Our search for the 100 closest images (in CLIP
[28] space) in the dataset to the term ‘japanese body’ yielded almost
exclusively inappropriate material, with 90% showing explicit nudity. For
comparison, the global average for 50 selected countries is more than 4 times
lower at 22% (details can be found in Sec. A.1).

Now, to investigate the bias transfer to SD, we prompted the model using
‘<country> body’ (e.g. ‘japanese body’ or ‘nigerian body’) and compared the
percentages of explicit nudes for the same 50 countries. We automatically
assessed the generated image using the NudeNet image
classifier††https://github.com/notAI-tech/NudeNet to detect exposed genitalia.
We chose this ambiguous prompt template to expose the model’s biases. Based on
image web search results, one can, for instance, expect images of categories
such as average body comparison—displaying lightly dressed people—but also
culturally related images such as the country flags.

Figure 2: Grounded in reporting bias, one can observe ethnic biases in DMs
(left). For 50 selected countries, we generated 100 images with the prompt
‘$<$country$>$ body’. The country Japan shows the highest probability of
generating nude content. SLD uses the strong hyper parameter set to counteract
this bias (right). (Best viewed in color)

For each country, we generated 100 images, each using the same seed and hyper-
parameters. The results are depicted in Fig. 2 (left). Indeed, one can observe
the above-described images such as average body comparison in the case of u.s.
american (cf. Sec. A.2). However, as expected, the close association of some
ethnic terms with nudity in Stable Diffusion is apparent. Overall it appears
that European, Asian, and Oceanic countries are far more likely to be linked
with nudity than African or American ones. The most nude images are generated
for Japan at over 75%, whereas the global average is at 35%. Specifically, the
terms ‘Asian’ and ‘Japanese’ yielded a significantly higher amount of nudity
than any other ethnic or geographic term. We attribute the apparent synonym
usage of ‘Japanese’ and ‘Asian’ in this context to the aforementioned trends
and the overwhelming amount of such content in LAION-5B. Unfortunately, biases
in SD generation like these may further reinforce problematic social
phenomena.

SD’s post-hoc safety measures. Various methods have been proposed to detect
and filter out inappropriate images [4, 11, 25, 33]. Similarly, the SD
implementation does contain a “NSFW” safety checker; an image classifier
applied after generation to detect and withhold inappropriate images. However,
there seems to be an interest in deactivating this safety measure. We checked
the recently added image generation feature of lexica.art using examples we
knew to generate content that the safety checker withholds. We note that the
generation of these inappropriate images is possible on lexica.art at time of
the present study, apparently without any restrictions, cf. Sec. A.3.

Now, we are ready to introduce our two main contributions, first SLD and then
the I2P benchmark.

##  3 Safe Latent Diffusion (SLD)

We introduce safety guidance for latent diffusion models to reduce the
inappropriate degeneration of DMs. Our method extends the generative process
by combining text conditioning through classifier-free guidance with
inappropriate concepts removed or suppressed in the output image.
Consequently, SLD performs image editing at inference without any further
fine-tuning required.

Diffusion models iteratively denoise a Gaussian distributed variable to
produce samples of a learned data distribution. Intuitively, image generation
starts from random noise $\epsilon$, and the model predicts an estimate of
this noise $\tilde{\epsilon}_{\theta}$ to be subtracted from the initial
values. This results in a high-fidelity image $x$ without any noise. Since
this is an extremely hard problem, multiple steps are applied, each
subtracting a small amount ($\epsilon_{t}$) of the predictive noise,
approximating $\epsilon$. For text-to-image generation, the model’s
$\epsilon$-prediction is conditioned on a text prompt $p$ and results in an
image faithful to that prompt. The training objective of a diffusion model
$\hat{x}_{\theta}$ can be written as

$\mathbb{E}_{\mathbf{x,c}_{p}\mathbf{,\epsilon},t}\left[w_{t}||\mathbf{\hat{x}}_{\theta}(\alpha_{t}\mathbf{x}+\omega_{t}\mathbf{\epsilon},\mathbf{c}_{p})-\mathbf{x}||^{2}_{2}\right]$
(1)

where $(\mathbf{x,c}_{p})$ is conditioned on text prompt $p$, $t$ is drawn
from a uniform distribution $t\sim\mathcal{U}([0,1])$, $\epsilon$ sampled from
a Gaussian $\mathbf{\epsilon}\sim\mathcal{N}(0,\mathbf{I})$, and
$w_{t},\omega_{t},\alpha_{t}$ influence image fidelity depending on $t$.
Consequently, the DM is trained to denoise
$\mathbf{z}_{t}:=\mathbf{x}+\mathbf{\epsilon}$ to yield $\mathbf{x}$ with the
squared error as loss. At inference, the DM is sampled using the model’s
prediction of $\mathbf{x}=(\mathbf{z}_{t}-\mathbf{\bar{\epsilon_{\theta}}})$,
with ${\bar{\epsilon_{\theta}}}$ as described below.

Classifier-free guidance [17] is a conditioning method using a purely
generational diffusion model, eliminating the need for an additional pre-
trained classifier. The approach randomly drops the text conditioning
$\mathbf{c}_{p}$ with a fixed probability during training, resulting in a
joint model for unconditional and conditional objectives. During inference the
score estimates for the $\mathbf{x}$-prediction are adjusted so that:

$\mathbf{\tilde{\epsilon}}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{p}):=\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t})+s_{g}(\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{p})-\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t}))$
(2)

with guidance scale $s_{g}$ which is typically chosen as $s_{g}\in(0,20]$ and
$\epsilon_{\theta}$ defining the noise estimate with parameters $\theta$.
Intuitively, the unconditioned $\epsilon$-prediction
$\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t})$ is pushed in the direction of the
conditioned $\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{p})$ to
yield an image faithful to prompt $p$. Lastly, $s_{g}$ determines the
magnitude of the influence of the text $p$.

Figure 3: Illustration of text-conditioned diffusion processes. SD using
classifier-free guidance (blue arrow), SLD (green arrow) utilizing “unsafe”
prompts (red arrow) to guide the generation in an opposing direction. For a
more detailed comparison see Appendix Fig. 16. (Best viewed in color)

To influence the diffusion process, SLD makes use of the same principles as
classifier-free guidance, cf. the simplified illustration in Fig. 3. In
addition to a text prompt $p$ (blue arrow), we define an inappropriate concept
(red arrow) via textual description $S$. Consequently, we use three
$\epsilon$-predictions with the goal of moving the unconditioned score
estimate $\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t})$ towards the prompt
conditioned estimate
$\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{p})$ and simultaneously
away from concept conditioned estimate
$\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{S})$. This results in
$\mathbf{\bar{\epsilon}}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{p},\mathbf{c}_{S})=$

$\displaystyle\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t})+s_{g}\big{(}\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{p})-\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t})-\gamma(\mathbf{z}_{t},\mathbf{c}_{p},\mathbf{c}_{S})\big{)}$
(3)

with the safety guidance term $\gamma$

$\gamma(\mathbf{z}_{t},\mathbf{c}_{p},\mathbf{c}_{S})=\mu(\mathbf{c}_{p},\mathbf{c}_{S};s_{S},\lambda)(\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{S})-\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t}))\
,$ (4)

where $\mu$ applies a guidance scale $s_{S}$ element-wise. To this extent,
$\mu$ considers those dimensions of the prompt conditioned estimate that would
guide the generation process toward the inappropriate concept. Therefore,
$\mu$ scales the element-wise difference between the prompt conditioned
estimate and safety conditioned estimate by $s_{S}$ for all elements where
this difference is below a threshold $\lambda$ and equals $0$ otherwise:
$\mu(\mathbf{c}_{p},\mathbf{c}_{S};s_{S},\lambda)=$

$\displaystyle\begin{cases}\text{max}(1,|\phi|),&\text{where
}\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{p})\ominus\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{S})<\lambda\\\
0,&\text{otherwise}\end{cases}$ (5)
$\displaystyle\text{with}\quad\phi=s_{S}(\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{p})-\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{S}))$
(6)

with both larger $\lambda$ and larger $s_{S}$ leading to a more substantial
shift away from the prompt text and in the opposite direction of the defined
concept. Note that we clip the scaling factor of $\mu$ in order to avoid
producing image artifacts. As described in previous research [31, 16], the
values of each $\mathbf{x}$-prediction should adhere to the training bounds of
$[-1,1]$ to prevent low fidelity images.

SLD is a balancing act between removing all inappropriate content from the
generated image while keeping the changes minimal. In order to facilitate
these requirements, we make two adjustments to the methodology presented
above. We add a warm-up parameter $\delta$ that will only apply safety
guidance $\gamma$ after an initial warm-up period in the diffusion process,
i.e., $\gamma(\mathbf{z}_{t},\mathbf{c}_{p},\mathbf{c}_{S}):=\mathbf{0}\text{
if }t<\delta$. Naturally, higher values for $\delta$ lead to less significant
adjustments of the generated image. As we aim to keep the overall composition
of the image unchanged, selecting a sufficiently high $\delta$ ensures that
only fine-grained details of the output are altered. Furthermore, we add a
momentum term $\nu_{t}$ to the safety guidance $\gamma$ in order to accelerate
guidance over time steps for dimensions that are continuously guided in the
same direction. Hence, $\gamma_{t}$ is defined as:
$\gamma_{t}(\mathbf{z}_{t},\mathbf{c}_{p},\mathbf{c}_{S})=$

$\displaystyle\mu(\mathbf{c}_{p},\mathbf{c}_{S};s_{S},\lambda)(\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{S})-\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t}))+s_{m}\nu_{t}$
(7)

with momentum scale $s_{m}\in[0,1]$ and $\nu$ being updated as

$\nu_{t+1}=\beta_{m}\nu_{t}+(1-\beta_{m})\gamma_{t}$ (8)

where $\nu_{0}=\mathbf{0}$ and $\beta_{m}\in[0,1)$, with larger $\beta_{m}$
resulting in less volatile changes of the momentum. Momentum is already built
up during the warm-up period, even though $\gamma_{t}$ is not applied during
these steps.

Overall, the resulting SLD progress is exemplary visualized by means of the
various diffusion steps in Fig. 3. While the safety-guidance is already
applied in early steps it removes and suppresses the inappropriate parts of
the images as soon as these are constructed in the latent space. We attached
the corresponding SLD pseudo-code in Appendix H.

##  4 Configuring Safe Latent Diffusion

Inappropriateness may be subjective depending on individual opinions and
contexts. For instance, the requirements of a professional artist differ from
those of a 4-year-old child. Therefore, we suggest four possible
configurations of the diffusion process varying in the strength of the safety
adaptions. The configurations include the before mentioned hyper-parameters
and concepts. We believe the proposed values offer a decent trade-off between
the degree of alternation but note that these can be easily adapted.

Figure 4: Illustration and qualitative comparison of different SLD
configurations for removing inappropriate content. All prompts taken from I2P
(cf. Sec. 5). The left column shows the original image, the four images in the
middle are generated using SLD, and the right column uses the inappropriate
concept as a negative prompt without SLD. Images were blurred manually after
generation. For prompts see Appendix Fig. 13. (Best viewed in color)

(Un)safe Concept. Inspired by existing content policies of deployed text-to-
image DMs such as DALL-E, the following text prompt is used to define a
general inappropriate safety concept: “hate, harassment, violence, suffering,
humiliation, harm, suicide, sexual, nudity, bodily fluids, blood, obscene
gestures, illegal activity, drug use, theft, vandalism, weapons, child abuse,
brutality, cruelty”. Through the text-based definition, one could easily add
further and remove present concepts (cf. Appendix H).

Threshold. The most vital parameter of SLD is the safety threshold $\lambda$.
It determines the location of the hyperplane dividing the latent space into
appropriate and inappropriateness, cf. Eq. 5. Theoretically, $\lambda$ is
restricted by the training bounds of $[-1,1]$, and intuitively it should be at
least $0$. However, since our approach relies on the model’s understanding of
“right” and “wrong” we recommend choosing a conservative, i.e. small positive
values such that $\lambda\in[0.0,0.03]$.

Safety guidance scale. The safety guidance scale $s_{S}$ can theoretically be
chosen arbitrarily high as the scaling factor $\mu$ is clipped either way.
Larger values for $s_{S}$ would simply increase the number of values in latent
representation being set to 1. Therefore, there is no adverse effect of large
$s_{S}$ such as image artifacts that are observed for high guidance scales
$s_{g}$. We recommend $s_{S}\in[100,3000]$.

Warm-up. The warm-up period $\delta$ largely influences at which level of the
image composition changes are applied. Large safe-guidance scales applied
early in the diffusion process could lead to major initial changes before
significant parts of the images were constructed. Hence, we recommend using at
least a few warm-up steps, $\delta\in[5,20]$, to construct an initial image
and, in the worst case, let SLD revise those parts. In any case, $\delta$
should be no larger than half the number of total diffusion steps.

Momentum. The guidance momentum is particularly useful to remove inappropriate
concepts that make up significant portions of the image and thus require more
substantial editing, especially those created during warm-up. Therefore,
momentum builds up over the warm-up phase, and such images will be altered
more rigorously than those with close editing distances. Higher momentum
parameters usually allow for a longer warm-up period. With most diffusion
processes using around 50 generation steps, the window for momentum build-up
is limited. Therefore, we recommend choosing $s_{m}\in[0,0.5]$ and
$\beta_{m}\in[0.3,0.7]$.

Configuration sets. These recommendations result in the following four sets of
hyper-parameters gradually increasing their aggressiveness of changes on the
resulting image (cf. Fig. 4 and Appendix I). Which setting to use highly
depends on the use case and individual preferences:

Config | $\delta$ | $s_{S}$ | $\lambda$ | $s_{m}$ | $\beta_{m}$
---|---|---|---|---|---
Hyp-Weak | $15$ | $\phantom{0}200$ | $0.0\phantom{00}$ | $0.0$ | -
Hyp-Medium | $10$ | $1000$ | $0.01\phantom{0}$ | $0.3$ | $0.4$
Hyp-Strong | $\phantom{1}7$ | $2000$ | $0.025$ | $0.5$ | $0.7$
Hyp-Max | $\phantom{0}0$ | $5000$ | $1.0\phantom{00}$ | $0.5$ | $0.7$

The weak configuration is usually sufficient to remove superficial blood
splatters, but stronger parameters are required to suppress more severe
injuries. Similarly, the weak set may suppress nude content on clearly
pornographic images but may not reduce nudity in artistic imagery such as oil
paintings. A fact that an adult artist may find perfectly acceptable, however,
is problematic for, e.g., a child using the model. Furthermore, on the example
of nudity, we observed the medium hyper-parameter set to yield the generation
of, e.g., a bikini. In contrast, the strong and maximum one would produce
progressively more cloth like a dress.

Note that we can even drive the generation of inappropriate content to zero by
choosing strong enough parameters (Hyp-Max). However, doing so likely diverges
from our goal of keeping changes minimal. Nevertheless, this could be a
requirement for sensitive applications, e.g., involving children. In these
cases, we further recommend the usage of post-hoc interventions such as SD’s
safety checker.

Regarding the amount of observed changes, the Hyp-Max configuration often
behaves similarly to replacing the unconditioned estimate with a conditioned
estimate based on a negative prompt during the classifier-free guidance, cf.
Neg. in Fig. 4. I.e., replacing $\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t})$
with $\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{S})$, cf. Eq. 2.
However, as our experimental evaluation (cf. Tab. 1) shows, negative prompting
leads to worse mitigation than SLD. Further, the major disadvantage of this
approach is the lack of more fine-grained control over the generation process,
always leading to images significantly differing from the original, especially
for higher guidance scales $s_{S}$. Additionally, negative prompts are a vital
tool in text-to-image generation that would no longer be available to users if
used for safety guidance.

##  5 Inappropriate Image Prompts (I2P)

To systematically measure the risk of inappropriate degeneration by pre-
trained text-to-image models, we introduce a new benchmarking dataset of over
4.5k real-world text prompts for generative models that are likely to produce
inappropriate content: the inappropriate image prompts (I2P) dataset, cf. Fig.
1, covers a wide range of inappropriate content beyond nudity. Our
dataset††https://huggingface.co/datasets/AIML-TUDA/i2p and evaluation
pipeline†† https://github.com/ml-research/i2p is publicly available for other
researchers to use.

Inappropriate content. What is considered inappropriate imagery may differ
based on context, setting, cultural and social predisposition, and individual
factors and is highly subjective overall. In this work, we base our definition
of inappropriate content on the work of Gebru et al.: “[data that] if viewed
directly, might be offensive, insulting, threatening, or might otherwise cause
anxiety” [12], which is for example also reflected by the OpenAI content
policy††https://labs.openai.com/policies/content-policy that applies to the
use of DALL-E [29]. Specifically, we consider those images showcasing content
that contains one of the following:

hate, harassment, violence, self-harm, sexual content, shocking images,
illegal activity.

Note that inappropriateness is not limited to these seven concepts, varies
between cultures, and constantly evolves. Here we restricted ourselves to
images displaying tangible acts of inappropriate behavior.

Prompt collection. For the seven concepts mentioned above, we used 26 keywords
and phrases (cf. Appendix C) describing them in more detail and collected up
to 250 real-world text prompts for each. For a given keyword, we crawled the
prompts of the top 250 images returned by https://lexica.art. Lexica is a
collection of real-world, user-generated prompts for SD sourced from its
official discord server. It stores the prompt, seed, guidance scale, and image
dimensions used in the generation to facilitate reproducibility. Image
retrieval in lexica is based on the similarity of an image and search query in
CLIP [28] embedding space. Therefore, the collected prompts are not guaranteed
to generate inappropriate content, but the probability is high, as
demonstrated in our evaluation.

Dataset statistics. The data collection described above yielded duplicate
entries, as some retrieved images were found among multiple keywords. After
reducing those duplicates, the I2P dataset contains 4703 unique prompts
assigned to at least one of the seven categories above. We also include an
estimate of the percentage of inappropriate images the prompt is predicted to
generate, together with the necessary hyper-parameters to reproduce these
results. The benchmark also contains a hard annotation for prompts that
generate predominantly inappropriate images.

On average, the prompts are made up of 20 tokens, and we could not observe an
apparent correlation between frequent words and the connection to
inappropriate images of these prompts. We present a word cloud of frequently
used terms in Appendix C. Furthermore, we include the toxicity of each prompt
based on the respective toxicity score of the Perspective
API.††https://github.com/conversationai/perspectiveapi We only find a weak
correlation††Spearman $r=0.22$ between the toxicity of a prompt and the
inappropriateness of images it generates. In fact, prompts with low toxicity
scores still have unforeseen high probabilities of generating inappropriate
images. Furthermore, out of 4702 prompts, a mere 1.5% are toxic. This
highlights that simply suppressing “bad” words in text prompts is no reliable
mitigation strategy against generating problematic content.

| Inappropriate Probability $\downarrow$ |  Exp. Max. Inappropriateness $\downarrow$
---|---|---
Category | SD 1.4 | Neg. Prompt | Hyp-Weak | Hyp-Medium | Hyp-Strong | Hyp-Max | SD | Hyp-Strong | Hyp-Max
Hate |  0.40 |  0.18 |  0.27 |  0.20 |  0.15 |  0.09 | $\pagecolor{high!95!low!40}0.97_{0.06}$ | $\pagecolor{high!64!low!40}0.77_{0.19}$ | $\pagecolor{high!27!low!40}0.53_{0.18}$
Harassment |  0.34 |  0.16 |  0.24 |  0.17 |  0.13 |  0.09 | $\pagecolor{high!91!low!40}0.94_{0.08}$ | $\pagecolor{high!58!low!40}0.73_{0.18}$ | $\pagecolor{high!33!low!40}0.57_{0.20}$
Violence |  0.43 |  0.24 |  0.36 |  0.23 |  0.17 |  0.14 | $\pagecolor{high!83!low!40}0.89_{0.04}$ | $\pagecolor{high!67!low!40}0.79_{0.13}$ | $\pagecolor{high!50!low!40}0.68_{0.28}$
Self-harm |  0.40 |  0.16 |  0.27 |  0.16 |  0.10 |  0.07 | $\pagecolor{high!95!low!40}0.97_{0.06}$ | $\pagecolor{high!39!low!40}0.61_{0.20}$ | $\pagecolor{high!20!low!40}0.49_{0.21}$
Sexual |  0.35 |  0.12 |  0.23 |  0.14 |  0.09 |  0.06 | $\pagecolor{high!86!low!40}0.91_{0.08}$ | $\pagecolor{high!27!low!40}0.53_{0.16}$ | $\pagecolor{high!0!low!40}0.36_{0.11}$
Shocking |  0.52 |  0.28 |  0.41 |  0.30 |  0.20 |  0.13 | $\pagecolor{high!100!low!40}1.00_{0.01}$ | $\pagecolor{high!77!low!40}0.85_{0.14}$ | $\pagecolor{high!48!low!40}0.67_{0.20}$
Illegal activity |  0.34 |  0.14 |  0.23 |  0.14 |  0.09 |  0.06 | $\pagecolor{high!91!low!40}0.94_{0.10}$ | $\pagecolor{high!41!low!40}0.62_{0.20}$ | $\pagecolor{high!11!low!40}0.43_{0.19}$
Overall |  0.39 |  0.18 |  0.29 |  0.19 |  0.13 |  0.09 | $\pagecolor{high!94!low!40}0.96_{0.07}$ | $\pagecolor{high!56!low!40}0.72_{0.19}$ | $\pagecolor{high!38!low!40}0.60_{0.19}$
Table 1: Safe Latent Diffusion (SLD) can considerably reduce the chance of
generating inappropriate content (the lower, the better). Shown are the
probabilities of generating an image containing inappropriate content as
classified by the combined Q16/NudeNet classifier over the I2P benchmark. We
note that the Q16 classifier is rather conservative and tends to classify some
unobjectionable images as inappropriate. The false positive rate of the
classifier is roughly equal to the probabilities reported for Hyp-Max. The
expected maximum inappropriateness (the lower, the better) are bootstrap
estimates of a model outputting the displayed percentage of inappropriate
images at least once for 25 prompts (for further results see Appendix F).
Subscript values indicate the standard deviation.

##  6 Experimental Evaluation

We now evaluate SD’s inappropriate degeneration and SLD based on the suggested
configurations using I2P.

Experimental Protocol. To assess the reduction of inappropriate content, we
generated ten images each for all prompts of the I2P test bed and compared the
probability of generating inappropriate images. We used one general concept
$S$ across all categories of I2P as specified in Sec. 4. We automatically
evaluated inappropriate image content by combining two classifiers. First, the
Q16 classifier [33]—also used to annotate the LAION-5B dataset—to detect a
wide range of inappropriate content in images. Second, we applied NudeNet (cf.
Sec. 2) to identify sexually explicit content. In this paper, we only classify
exposed genitalia as inappropriate while allowing otherwise provocative
images. If not specified otherwise, an image is classified as inappropriate if
one or both of the classifiers output the respective label. Further details
can be found in Appendix D.

Inappropriateness in Stable Diffusion. We start our experimental evaluation by
demonstrating the inappropriate degeneration of Stable Diffusion without any
safety measures. Tab. 1 shows SD’s probability of generating inappropriate
content for each category under investigation. Recall that only $1.5\%$ of the
text prompts could be identified as toxic. Nevertheless, one can clearly
observe that depending on the category, the probability of generating
inappropriate content ranges from $34\%$ to $52\%$. Furthermore, Tab. 1
reports the expected maximum inappropriateness over 25 prompts. These results
show that a user generating images with I2P for 25 prompts is expected to have
at least one batch of output images of which 96% are inappropriate. The
benchmark clearly shows SD’s inappropriate degeneration and the risks of
training on completely unfiltered datasets.

SLD in Stable Diffusion. Next, we investigate whether we can account for
noisy, i.e. biased and unfiltered training data based on the model’s acquired
knowledge in distinguishing between appropriate and inappropriate content.

To this end, we applied SLD. Similarly to the observations made on the
examples in Fig. 4, one can observe in Tab. 1 that the number of inappropriate
images gradually decreases with stronger hyper-parameters. The strongest
hyper-parameter configuration reduces the probability of generating
inappropriate content by over 75%. Consequently, a mere 9% of the generated
images are still classified as inappropriate. However, it is important to note
that the Q16 classifier tends to be rather conservative in some of its
decisions classifying images as inappropriate where the respective content has
already been reduced significantly. We assume the majority of images flagged
as potentially inappropriate for Hyp-Max to be false negatives of the
classifier. One can observe a similar reduction in the expected maximum
inappropriateness but also note a substantial increase in variance. The latter
indicates a substantial amount of outliers when using SLD.

Overall the results demonstrate that, indeed, we are able to largely mitigate
the inappropriate degeneration of SD based on the underlying model’s learned
representations. This could also apply to issues caused by reporting biases in
the training set, as we will investigate in the following.

Counteracting Bias in Stable Diffusion. Recall the ‘ethnic bias’ experiments
of Sec. 2. We demonstrated that biases reflected in LAION-5B data are,
consequently, also reflected in the trained DM. Similarly to its performance
on I2P, SLD strongly reduces the number of nude images generated for all
countries as shown in Fig. 2 (right). SLD yields 75% less explicit content and
the percentage of nude images are distributed more evenly between countries.
The previous outlier Japan now yields 12.0% of nude content, close to the
global percentage of 9.25%.

Nonetheless, at least with keeping changes minor (Hyp-Strong), SLD alone is
not sufficient to mitigate this racial bias entirely. There remains a medium
but statistically significant correlation††Spearman $r=0.52$; Null-hypothesis
that both distributions are uncorrelated is rejected at a significance level
of $p=0.01$. between the percentages of nude images generated for a country by
SD with and without SLD. Thus, SLD can make a valuable contribution towards
de-biasing DMs trained on datasets that introduce biases. However, these
issues still need to be identified beforehand, and an effort towards
reducing—or better eliminating—such biases in the dataset itself is still
required.

For further evidence, we ran experiments on Stable Diffusion v2.0 which is
essentially a different model with a different text encoder and training set.
Specifically, rigorous dataset filtering of sexual and nudity related content
was applied before training the diffusion model, however, not on the pre-
trained text encoder. While this filtering process reduces biased
representations, they are still present and more frequent compared to SLD
mitigation on SD in version 1.4, cf. Appendix E. Interestingly, the
combination of SLD and dataset filtering achieves an even better mitigation.
Hence, a combination of filtering and SLD could be beneficial and poses an
interesting avenue for future work.

##  7 Discussion & Limitations

Before concluding, let us touch upon ethical implications and future work
concerning I2P and the introduced SLD.

Ethical implications. We introduced an alternative approach to post-hoc
prevention of presenting generated images with potentially inappropriate
content. Instead, we identify inappropriate content and suppress it during the
diffusion process. This intervention would not be possible if the model did
not acquire a certain amount of knowledge on inappropriateness and related
concepts during pre-training. Consequently, we do not advise removing
potentially inappropriate content entirely from the training data, as we can
reasonably assume that efforts towards removing all such samples will hurt the
model’s capabilities to target related material at inference individually.
Therefore, we also see a promising avenue for future research in measuring the
impact of training on balanced datasets. However, this is likely to require
large amounts of manual labor.

Nonetheless, we also demonstrated that highly imbalanced training data could
reinforce problematic social phenomena. It must be ensured that potential
risks can be reliably mitigated, and if in doubt, datasets must be further
curated, such as in the presented case study. Whereas LAION already made a
valiant curating effort by annotating the related inappropriate content, we
again advocate for carefully investigating behavior and possible biases of
models and consequently deploy mitigation strategies against these issues in
any deployed application.

We realize that SLD potentially has further ethical implications. Most
notably, we recognize the possibility of similar techniques being used for
actively censoring generative models. Additionally, one could construct a
model generating mainly inappropriate content by reversing the guidance
direction of our approach. Thus, we strongly urge all models using SLD to
transparently state which contents are being suppressed. However, it could
also be applied to cases beyond inappropriateness, such as fairness [22].
Furthermore, we reiterate that inappropriateness is based on social norms, and
people have diverse sentiments. The introduced test bed is limited to specific
concepts and consequently does not necessarily reflect differing opinions
people might have on inappropriateness. Additionally, the model’s acquired
representation of inappropriateness may reflect the societal dispositions of
the social groups represented in the training data and might lack a more
diverse sentiment.

Image Fidelity & Text Alignment.

| Image Fidelity | Text Alignment
---|---|---
Config |  FID-30k $\downarrow$ |  User (%) $\uparrow$ |  CLIP $\downarrow$ |  User (%) $\uparrow$
SD | 14.43 | - | 0.75 | -
Weak | 15.81 | 63.70 | 0.75 | 60.88
Medium | 16.90 | 62.37 | 0.75 | 59.45
Strong | 18.28 | 63.13 | 0.76 | 59.62
Max | 18.76 | 63.60 | 0.76 | 60.58
Table 2: SLD’s image fidelity and text alignment. User scores indicate the
percentage of users judging SLD generated image as better or equal in
quality/text alignment as its SD counterpart.

Lastly, we discuss the overall impact of SLD on image fidelity and text-
alignment. Ideally, the approach should have no adverse effect on either,
especially on already appropriate images. In line with previous research on
generative text-to-image models, we report the COCO FID-30k scores and CLIP
distance of SD, and our four sets of hyper-parameters for SLD in Tab. 2. The
scores slightly increase with stronger hyper-parameters. However, they do not
necessarily align with actual user preference [26]. Therefore, we conducted an
exhaustive user study on the DrawBench [31] benchmark and reported results in
Tab. 2 (cf. Appendix G for study details). The results indicate that users
even slightly prefer images generated with SLD over those without, indicating
safety does no sacrifice image quality and text alignment.

##  8 Conclusion

We demonstrated text-to-image models’ inappropriate degeneration transfers
from unfiltered and imbalanced training datasets. To measure related issues,
we introduced an image generation test bed called I2P containing dedicated
image-to-text prompts representing inappropriate concepts such as nudity and
violence. Furthermore, we presented an approach to mitigate these issues based
on classifier-free guidance. The proposed SLD removes and suppresses the
corresponding image parts during the diffusion process with no additional
training required and no adverse effect on overall image quality. Strong
representation biases learned from the dataset are attenuated by our approach
but not completely removed. Thus, we advocate for the careful use of
unfiltered, clearly imbalanced datasets.

## Acknowledgments

We gratefully acknowledge support by the German Center for Artificial
Intelligence (DFKI) project “SAINT” and the Federal Ministry of Education and
Research (BMBF) under Grant No. 01IS22091. This work also benefited from the
ICT-48 Network of AI Research Excellence Center “TAILOR” (EU Horizon 2020, GA
No 952215), the Hessian research priority program LOEWE within the project
WhiteBox, the Hessian Ministry of Higher Education, and the Research and the
Arts (HMWK) cluster projects “The Adaptive Mind” and “The Third Wave of AI”,
and the HMWK and BMBF ATHENE project “AVSV”. Further, we thank Felix
Friedrich, Dominik Hintersdorf and Lukas Struppek for their valuable feedback.

## References

  * [1] Abubakar Abid, Maheen Farooqi, and James Zou.  Persistent anti-muslim bias in large language models.  In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), page 298–306. Association for Computing Machinery, 2021.
  * [2] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  Preprint at https://arxiv.org/abs/2204.02491, 2022.
  * [3] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell.  On the dangers of stochastic parrots: Can language models be too big?  In Proceedings of ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 610–623, 2021.
  * [4] Abeba Birhane and Vinay Uday Prabhu.  Large image datasets: A pyrrhic win for computer vision?  In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1536–1546, 2021.
  * [5] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe.  Multimodal datasets: misogyny, pornography, and malignant stereotypes.  CoRR, abs/2110.01963, 2021.
  * [6] Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai.  Man is to computer programmer as woman is to homemaker? Debiasing word embeddings.  In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), pages 4349–4357. Curran Associates Inc., 2016.
  * [7] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
  * [8] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan.  Semantics derived automatically from language corpora contain human-like biases.  Science, 356(6334):183–186, 2017.
  * [9] Damien L. Crone, Stefan Bode, Carsten Murawski, and Simon M. Laham.  The socio-moral image database (smid): A novel stimulus set for the study of social, moral and affective processes.  PLOS ONE, 13(1), 2018.
  * [10] Emily Denton, Alex Hanna, Razvan Amironesei, Andrew Smart, and Hilary Nicole.  On the genealogy of machine learning datasets: A critical history of imagenet.  Big Data & Society, 8(2), 2021.
  * [11] Shreyansh Gandhi, Samrat Kokkula, Abon Chaudhuri, Alessandro Magnani, Theban Stanley, Behzad Ahmadi, Venkatesh Kandaswamy, Omer Ovenc, and Shie Mannor.  Scalable detection of offensive and non-compliant content / logo in product images.  In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2020.
  * [12] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé III, and Kate Crawford.  Datasheets for datasets.  Commun. ACM, 64(12):86–92, 2021.
  * [13] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith.  Realtoxicityprompts: Evaluating neural toxic degeneration in language models.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3356–3369. Association for Computational Linguistics, 2020.
  * [14] Jonathan Gordon and Benjamin Van Durme.  Reporting bias and knowledge acquisition.  In Proceedings of the Workshop on Automated Knowledge Base Construction (AKBC), pages 25–30, 2013.
  * [15] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  Preprint at https://arxiv.org/abs/2208.01626, 2022.
  * [16] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
  * [17] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  CoRR, abs/2207.12598, 2022.
  * [18] Matthew Hutson.  Robo-writers: the rise and risks of language-generating ai.  Nature, 591:22–56, 2021.
  * [19] Abigail Z. Jacobs.  Measurement and fairness.  In Proceedings of ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 375–385. ACM, 2021.
  * [20] Katrien Jacobs, Thomas Baudinette, and Alexandra Hambleton.  Reflections on researching pornography across asia: voices from the region.  Porn Studies, 7, 2020.
  * [21] Sophie Jentzsch, Patrick Schramowski, Constantin A. Rothkopf, and Kristian Kersting.  Semantics derived automatically from language corpora contain human-like moral choices.  In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), pages 37–44, 2019.
  * [22] Chen Karako and Putra Manggala.  Using image fairness representations in diversity-based re-ranking for recommendations.  In Tanja Mitrovic, Jie Zhang, Li Chen, and David Chin, editors, Adjunct Publication of the Conference on User Modeling, Adaptation and Personalization (UMAP), pages 23–28. ACM, 2018.
  * [23] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  Preprint at https://arxiv.org/abs/2210.09276.
  * [24] Agostina J. Larrazabal, Nicolás Nieto, Victoria Peterson, Diego H. Milone, and Enzo Ferrante.  Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis.  Proceedings of the National Academy of Sciences, 117(23):12592–12594, 2020.
  * [25] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  GLIDE: towards photorealistic image generation and editing with text-guided diffusion models.  In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2022.
  * [26] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu.  On aliased resizing and surprising subtleties in GAN evaluation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [27] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller.  Language models as knowledge bases?  In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463–2473. Association for Computational Linguistics, 2019.
  * [28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
  * [29] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with CLIP latents.  Preprint at https://arxiv.org/abs/2204.06125, 2022.
  * [30] Adam Roberts, Colin Raffel, and Noam Shazeer.  How much knowledge can you pack into the parameters of a language model?  In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426. Association for Computational Linguistics, 2020.
  * [31] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  CoRR, abs/2205.11487, 2022.
  * [32] Timo Schick, Sahana Udupa, and Hinrich Schütze.  Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP.  Transactions of the Association for Computational Linguistics (TACL), 9:1408–1424, 2021.
  * [33] Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting.  Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content?  In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2022.
  * [34] Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A. Rothkopf, and Kristian Kersting.  Large pre-trained language models contain human-like biases of what is right and wrong to do.  Nature Machine Intelligence, 4(3), 2022.
  * [35] Patrick Schramowski, Cigdem Turan, Sophie Jentzsch, Constantin A. Rothkopf, and Kristian Kersting.  The moral choice machine.  Frontiers Artif. Intell., 3:36, 2020.
  * [36] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  Laion-5b: An open large-scale dataset for training next generation image-text models.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * [37] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: open dataset of clip-filtered 400 million image-text pairs.  Preprint at https://arxiv.org/abs/2111.02114, 2021.
  * [38] Ryan Steed and Aylin Caliskan.  Image representations learned with unsupervised pre-training contain human-like biases.  In Proceedings of ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 701–713, 2021.
  * [39] Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang.  Mitigating gender bias in natural language processing: Literature review.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 1630–1640. Association for Computational Linguistics, 2019.
  * [40] Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan.  Unitune: Text-driven image editing by fine tuning an image generation model on a single image.  Preprint at https://arxiv.org/abs/2210.09477.
  * [41] Angelina Wang, Arvind Narayanan, and Olga Russakovsky.  REVISE: A tool for measuring and mitigating bias in visual datasets.  In Proceedings of European Conference on Computer Vision (ECCV), pages 733–751, 2020.
  * [42] Robin Zheng.  Why yellow fever isn’t flattering: A case against racial fetishes.  Journal of the American Philosophical Association, 2, 2016.

Warning:
Blurred inappropriate images and the associated textual content below.

## Appendix

##  Appendix A Ethnic Bias Experiment

Here, we provide more details on the “Ethnic Bias Experiment” related
findings.

###  A.1 CLIP Analysis on LAION-2B-en

For each of the 50 selected countries introduced in Secs. 2 and 6 we retrieved
the 100 closest images for the caption “$<$country$>$ body” from LAION-2B-en.
Similar to the experiments in Secs. 2 and 6 we also computed the number of
percentage of nude images for each country††We note that roughly 8% of all
images were missing in our local clone of LAION-2B-en, as the respective URLs
were no longer available at the time of download..

The observations regarding “ethnic bias” we made on SD generated images are
also apparent in its initial training data set LAION-2B-en. Among the top-5
countries in terms of the number of nude images are four Asian ones with
Japan, Indonesia, Thailand and India. Overall Japan tops that ranking at over
90% explicit material. This is more than four times higher than the global
average of 22%.

###  A.2 SD Generations

As we have shown, the corresponding biases contained in the dataset transfer
to the diffusion model. In addition to the discussion in the main text, Fig. 5
provides qualitative examples. Again, we blurred all images showing people.
Still, one can observe that prompts containing asian and japanese mainly
produce explicit nudity. In contrast, u.s. american and nigerian, for
instance, produce the expected image style. Especially in the case of u.s.
american, we can find images displaying average body comparison and culturally
related content.

(a) asian body

(b) japanese body

(c) u.s. american body

Figure 5: Blurred images generated in Stable Diffusion for the text prompts
asian body 5(a), japanese body 5(b) , u.s. american body 5(c), and 6(a),
respectively. All images containing a person were blurred for privacy reasons,
as Stable Diffusion may generate images of real, existing people. (a) nigerian
body Figure 6: Blurred images generated in Stable Diffusion for the text
prompts asian body 5(a), japanese body 5(b) , u.s. american body 5(c), and
6(a), respectively. All images containing a person were blurred for privacy
reasons, as Stable Diffusion may generate images of real, existing people.

###  A.3 Lexica

Whereas the creators of SD warn and advice for research only, deployed
application such as lexica have the potential to reinforce biases. Fig. 7
shows images that lexica generates for the prompt ”Japanese body”, again
highlighting the strong ethnic bias in SD wrt. to Asian women and nudity.

Figure 7: Generation results on https://lexica.art for prompt “Japanese body”

##  Appendix B RealToxicityPrompts

It is noteworthy that we initially tried to reuse the prompts contained in
RealToxicityPrompts [13], cf. Appendix B. However, they lead to unnatural
images, mainly containing unidentifiable text fragments. Consequently, we only
included dedicated text-to-image prompts. Tab. 3 shows the results of SLD on
images generated using the subset of toxic prompts from RealToxicityPrompts.

Figure 8: Images generated by SD using toxic prompts from RealToxicityPrompts
highlighting the vast number of images containing random letters.

While these prompts produce a similar amount of inappropriate image as the I2G
benchmark, we did not opt to use the prompts from RealToxicityPrompts as the
resulting images are often only contain seemingly random combination of
letters, cf. Fig. 8.

Inappropriate Prob. | Exp. Max. Inappropriateness
---|---
SD | Hyp-Strong | SD | Hyp-Strong
0.40 | 0.10 | $0.92_{0.08}$ | $0.60_{0.19}$
Table 3: Probabilities of generating an image containing inappropriate content
as classified by the combined Q16/NudeNet classifier over the toxic prompts
from RealToxicityPrompts. The expected maximum inappropriateness are bootstrap
estimates of a model outputting the displayed percentage of inappropriate
images at least once for 25 prompts. Subscript values indicate the standard
deviation.

##  Appendix C I2P Test Bed

I2P contains seven different inappropriate concept categories which we
collected by the following keywords:

  * •

hate: “hate”

  * •

harassment: “mocking”, “bullying”, “threatening”, “harassment”

  * •

violence: “humiliation”, “violence acts”, “suffering”, “violence”

  * •

self-harm: “suicide”, “eating disorder”, “self-harm”, “attempts at harming
oneself”

  * •

sexual: “sexual acts”, “content meant to arouse sexual excitement”, “sexual”,
“nudity”

  * •

shocking: “profane subjects that may shock or disgust”, “bodily fluids”,
“shocking”, “obscene gestures”

  * •

illegal activity: “vandalism”, “illegal activity”, “drug use”, “theft”

These keywords were used to collect the I2P prompts from https://lexica.art.

Exemplary, Fig. 9 visualizes SD generations of each category. Each row depicts
a category with four SD-generated images; starting from top: hate, harassment,
violence, self-harm, sexual, shocking, illegal activity. Next to the
category’s exemplary images the I2P prompt is displayed. Additionally, Fig. 10
visualizes the most frequent terms in I2P prompts.

Furthermore, we provide a datasheet [12] for the I2P benchmark in App. J
giving further details on the dataset.

Figure 9: Stable Diffusion generations without SLD. Exemplary images of each
category contained in I2P. Starting from top each row depicts one category:
hate, harassment, violence, self-harm, sexual, shocking, illegal activity. The
corresponding I2P prompts are displayed on the right. Figure 10: Wordcloud of
the most frequent words used in I2P prompts without English stopwords.

##  Appendix D Experimental Protocol

Here, we provide further details of our experimental protocol, cf. Sec. 6.

#### Diffusion Model.

We based our implementation on version 1.4 of Stable Diffusion which we used
for all of our experiments. We chose to opt for a discrete Linear Multistep
Scheduler (LMS) with $\beta_{start}=$8.510-4$$ and $\beta_{end}=0.012$.
However, we note that our approach is applicable to any latent diffusion model
employing classifier-free guidance.

#### Inappropriate Content Measures.

We automatically evaluated inappropriate image content by combining two
classifiers. First, the Q16 classifier [33] is able to detect a wide range of
inappropriate content in images. It was trained on the SMID dataset [9] which
consists of images annotated on their appropriateness through user studies
conducted in the USA. More specifically, users were tasked to give each image
a score of 1-5 on the range of ”immoral/blameworthy” to ”moral/praiseworthy”.
Consequently, the Q16 classifier was trained to classify all images with an
average score below 2.5 as inappropriate. However, the SMID dataset contains
little to no explicit nudity—such as pornographic material—, wherefore Q16
performs subpar on these images. Thus, we additionally used
NudeNet††https://github.com/notAI-tech/NudeNet to identify sexually explicit
content. In this paper, we only classified exposed genitalia as inappropriate
while allowing otherwise provocative images. If not specified otherwise an
image is classified as inappropriate if one or both of the classifiers output
the respective label. We did not use the built in ”NSFW” safety checker of
Stable Diffusion as its high false positive rate renders is unsuitable for the
nuanced image editing in our work. However, it is indeed suitable to warn
users and prevent displaying potential inappropriate content generated by the
DM.

#### I2P.

We compared the base SD model to four variants of SLD as defined by the sets
of hyper-parameters in Sec. 4. To assess the reduction of inappropriate
content we generate 10 images each for all prompts of the I2P test bed and
compared the probability of generating inappropriate images. We used one
general concept $S$ across all categories of I2P as specified in Sec. 4.

##  Appendix E Stable Diffusion v2

To train Stable Diffusion v2 (SD-v2) rigorous dataset filtering of sexual and
nudity related content was applied. The I2P benchmark results of SD-v2 are
shown in Tab. 4 and a concise comparison of Stable Diffusion in version v2 and
v1.4 is provided in Tab. 5. Summarized, SLD’s mitigation on SD-v1.4 outperform
the standalone dataset filtering of SD-v2. The combination of dataset
filtering and SLD leads to the highest mitigation.

| Inappropriate Probability $\downarrow$ |  Expected Max. Inappropriateness $\downarrow$
---|---|---
Category/Method | SD 2.0 | Hyp-Weak | Hyp-Medium | Hyp-Strong | Hyp-Max | SD | Hyp-Strong | Hyp-Max
Hate |  0.44 |  0.32 |  0.26 |  0.20 |  0.15 | $\pagecolor{high!97!low!40}0.98_{0.08}$ | $\pagecolor{high!58!low!40}0.73_{0.11}$ | $\pagecolor{high!49!low!40}0.67_{0.16}$
Harassment |  0.40 |  0.29 |  0.23 |  0.19 |  0.14 | $\pagecolor{high!94!low!40}0.96_{0.06}$ | $\pagecolor{high!72!low!40}0.82_{0.18}$ | $\pagecolor{high!58!low!40}0.73_{0.15}$
Violence |  0.44 |  0.34 |  0.26 |  0.19 |  0.14 | $\pagecolor{high!98!low!40}0.99_{0.03}$ | $\pagecolor{high!74!low!40}0.83_{0.14}$ | $\pagecolor{high!60!low!40}0.74_{0.16}$
Self-harm |  0.40 |  0.26 |  0.15 |  0.10 |  0.06 | $\pagecolor{high!98!low!40}0.99_{0.03}$ | $\pagecolor{high!32!low!40}0.56_{0.18}$ | $\pagecolor{high!8!low!40}0.40_{0.17}$
Sexual |  0.29 |  0.18 |  0.12 |  0.08 |  0.05 | $\pagecolor{high!83!low!40}0.89_{0.12}$ | $\pagecolor{high!26!low!40}0.52_{0.16}$ | $\pagecolor{high!0!low!40}0.35_{0.15}$
Shocking |  0.51 |  0.37 |  0.26 |  0.17 |  0.13 | $\pagecolor{high!100!low!40}1.00_{0.01}$ | $\pagecolor{high!69!low!40}0.80_{0.11}$ | $\pagecolor{high!48!low!40}0.66_{0.18}$
Illegal activity |  0.37 |  0.27 |  0.19 |  0.13 |  0.10 | $\pagecolor{high!95!low!40}0.97_{0.07}$ | $\pagecolor{high!46!low!40}0.65_{0.15}$ | $\pagecolor{high!32!low!40}0.56_{0.21}$
Overall |  0.40 |  0.28 |  0.20 |  0.13 |  0.10 | $\pagecolor{high!97!low!40}0.98_{0.05}$ | $\pagecolor{high!58!low!40}0.73_{0.17}$ | $\pagecolor{high!42!low!40}0.62_{0.19}$
Table 4: Safe Latent Diffusion (SLD) applied on Stable Diffusion v2.0. Shown are the probabilities of generating an image containing inappropriate content as classified by the combined Q16/NudeNet classifier over the I2P benchmark. We note that the Q16 classifier is rather conservative and tends to classify some unobjectionable images as inappropriate. The false positive rate of the classifier is roughly equal to the probabilities reported for Hyp-Max. The expected maximum inappropriateness (the lower, the better) are bootstrap estimates of a model outputting the displayed percentage of inappropriate images at least once for 25 prompts (for further results see Appendix F). Subscript values indicate the standard deviation. | SD-v1.4 | SD-v2
---|---|---
Benchmark | SD | SLD | SD | SLD
Sexual (I2P) | 0.35 | 0.06$\circ$ | 0.29 | 0.05$\bullet$
Overall (I2P) | 0.39 | 0.09$\bullet$ | 0.40 | 0.10$\circ$
Body-Ethnicity | 0.36 | 0.09$\circ$ | 0.12 | 0.06$\bullet$
Table 5: Comparison of Stable Diffusion in version 1.4 (SD-v1.4) and 2.0
(SD-v2). To train SD-v2 rigorous dataset filtering of sexual and nudity
related content was applied. SLD’s mitigation on SD-v1.4 outperforms the
standalone dataset filtering of SD-v2. The combination of dataset filtering
and SLD leads to the highest mitigation performance.

##  Appendix F I2P Results

#### Expected maximum inappropriateness

In addition to the expected maximum inappropriateness for 25 prompts presented
in Tab. 1, we depict a continuous plot for each category from 10 to 200
generations in Fig. 11.

We observe clear differences in the expected maximum inappropriateness between
categories. For example when generating images with 200 prompts from the
“sexual” category, the Hyp-Max configuration is expected to yield at most 50%
inappropriate images whereas the same number of prompts from the “shocking”
category reaches almost 100% expected maximum inappropriateness. While some of
this can actually be attributed to the varying effectiveness of SLD on
different categories of inappropriateness, it is largely influenced by the
high false positive rate of the Q16 classifier. Since we are considering the
maximum over $N$ prompts, this effect quickly amplifies with growing $N$.

Overall this raises the question if the expected maximum inappropriateness
over large $N$ is a suitable metric for cases in which the false positive rate
is high. Consequently, we decided to only report the results at $N=25$ in the
main body of the paper.

(a) Hate

(b) Harassment

(c) Violence

(d) Self-Harm

(e) Sexual

(f) Shocking

(g) Illegal activity

(h) Over all categories

Figure 11: Expected maximum of inappropriate content for each category of I2P.

#### Qualitative Examples.

Fig. 12 depicts a comparison of SD generated images with (right) and without
(left) SLD. Each inappropriate category (cf. Appendix C) is represented by
four images. The corresponding prompts can be found in Fig. 9. Moreover, Fig.
13 depicts the generated images displayed in the main text and their
corresponding prompts.

Figure 12: Stable Diffusion generations with SLD. Within a pair the left image
is generated solely by SD without applying SLD and the right image using SLD.
Each row shows four images of the selected inappropriate categories of I2P,
Appendix C, starting from top: hate, harassment, violence, self-harm, sexual,
shocking, illegal activity. The corresponding prompts can be found in Fig. 9.
Figure 13: Generated images used in the main text with corresponding prompts.
Within a pair the left image is generated without SLD and right image with
SLD.

##  Appendix G DrawBench User Studies

Figure 14: Example images generated on DrawBench with SD (left) and SLD,
showing one prompt for each category.

Here, we provide further details on the conducted users studies on image
fidelity and text alignment on the DrawBench dataset. Additionally, we present
qualitative examples of images generated from DrawBench in Fig. 14.

###  G.1 Details on Procedure

For each model configuration and DrawBench prompt we generated 10 images,
amounting to 2000 total images per configuration. Each user was tasked with
labeling 25 random image pairs—one being the SD reference image and the second
one the corresponding image using SLD. For the image fidelity study users had
to answer the question

> Which image is of higher quality?

whereas the posed question for text alignment was

> Which image better represents the displayed text caption?

In both cases the three answer options were

  * •

I prefer image A.

  * •

I am indifferent.

  * •

I prefer image B.

To conduct our study we relied on Amazon Mechanical Turk where we set the
following qualification requirements for our users: HIT Approval Rate over 95%
and at least 1000 HITs approved. Additionally, each batch of image pairs was
evaluated by three distinct annotator resulting in 30 decisions for each
prompt.

Annotators were fairly compensated according to Amazon MTurk guidelines. For
the image fidelity task, users were paid $0.70 to label 25 images at an
average of 8 minutes need for the assignment. Our estimates suggested that the
image text alignment task, requires more time since the text caption has to be
read and understood. Therefore we paid $0.80 for 25 images with users
completing the task after 8.5 minutes on average.

###  G.2 Details on Results

The study results for each hyper parameter configuration on image fidelity and
text alignment is depicted in Fig. 15.

Figure 15: User study results on Image Fidelity and Text Alignment on
DrawBench. For each prompt we generated ten images with each image pair being
judged by three distinct users. Error bars indicate the standard deviation
across the 30 user decisions for each prompt.

Interestingly, on the perceived image fidelity we observed a transition from
indecisive to preferring the safety-guided images with increasing guidance’
strength, which we assume to be grounded in the increased visualization of
positive sentiments, for instance happy pets. A similar trend can be observed
for text alignment, although the effect is considerably smaller.

##  Appendix H Stable Diffusion Implementation

Algorithm 1 shows the pseudo code of SLD.

Algorithm 1 Safe Latent Diffusion

model weights $\theta$, text condition $text_{p}$, safety concept $text_{s}$
and diffusion steps $T$

$s_{m}\in[0,1]$, $\nu_{t=0}=0$, $\beta_{m}\in[0,1)$, $\lambda\in[0,1]$,
$s_{S}\in[0,5000]$, $\delta\in[0,20]$, $t=0$

$\text{DM}\leftarrow\text{init-diffusion-model}(\theta)$

$c_{p}\leftarrow\text{DM}.\text{encode}(text_{p})$

$c_{s}\leftarrow\text{DM}.\text{encode}(text_{s})$

$latents\leftarrow\text{DM}.\text{sample}(seed)$

while $t\neq T$ do

$n_{\emptyset},n_{p},n_{s}\leftarrow\text{DM}.\text{predict-
noise}(latents,c_{p},c_{s})$

$\mu_{t}\leftarrow\mathbf{0}$ $\triangleright$ Eq. 5

$\phi_{t}\leftarrow s_{S}*(n_{p}-n_{s})$ $\triangleright$ Eq. 6

$\mu_{t}\leftarrow\text{where}(n_{p}-n_{s}<\lambda,\text{max}(1,|\phi_{t}|))$
$\triangleright$ Eq. 5

$\gamma_{t}\leftarrow\mu_{t}*(n_{s}-n_{\emptyset})+s_{m}*\nu_{t}$
$\triangleright$ Eq. 7

$\nu_{t+1}\leftarrow\beta_{m}*\nu_{t}(1-\beta_{m})*\gamma_{t}$
$\triangleright$ Eq. 8

if $t\geq\delta$ then

$pred\leftarrow s_{g}*(n_{p}-n_{\emptyset}-\gamma_{t})$ $\triangleright$ Eq. 3

else

$pred\leftarrow s_{g}*(n_{p}-n_{\emptyset})$ $\triangleright$ Eq. 2

end if

$latents\leftarrow\text{DM}.\text{update-latents}(pred,latents)$

$t\leftarrow t+1$

end while

$image\leftarrow\text{DM}.\text{decode}(latents)$

In line with the Stable Diffusion’s policy giving its users maximum
transparency and control on how to use the model, the used safety concept can
be adapted based on the user’s preferences.

##  Appendix I SLD Ablation Studies

Lastly, we provide some qualitative examples of the influence of different
hyper parameters on the generated image.

Fig. 17 compares the effect of different warmup periods and thresholds. The
example highlights that more warmup steps $\delta$ lead to less significant
changes of the image composition and simultaneously larger values for
$\lambda$ alter the image more strongly. Furthermore, Fig. 18 shows the effect
of varying scales of momentum. It shows that higher momentum also leads to
stronger changes of the image and further accentuates that momentum scales
over $0.5$ may lead to issues in the downstream images such as significant
artifacts.

Additionally, Fig. 16 provides further insights on the inner workings of SLD
by showcasing the effect of different hyper parameter configurations over the
time steps of the diffusion process. Most importantly the Figure highlights
that stronger hyper parameters configuration diverge from the original image
much earlier in the diffusion process and change the image more substantially.

Figure 16: Visualization of SLD over the diffusion process. Notice how visible
cloth are generated earlier in the diffusion process with stronger hyper
parameters. Additionally, the strongest setting never yields any inappropriate
images at any point in the process. Figure 17: Effect on image generation
using different parameters for $\delta$ and $\lambda$. Guidance scales are
fixed at $s_{g}=15$ and $s_{S}=100$ and no momentum is not used, i.e. $s_{m}$
= 0. The image on the bottom left is close to the original image without SLD.
Figure 18: Effect on image generation using different momentum parameters.
Guidance scales are fixed at $s_{g}=15$ and $s_{S}=100$, with fixed warmup
period $\delta=5$ and fixed threshold $\lambda=0.015$. This further highlight
that values for $s_{m}>0.5$ are likely to produce significant image artifacts.

##  Appendix J I2P Datasheet

###  J.1 Motivation

  1. Q1

For what purpose was the dataset created? Was there a specific task in mind?
Was there a specific gap that needed to be filled? Please provide a
description.

     * •

Inappropriate Image Prompts (I2P) was created as a benchmark to evaluate
inappropriate degeneration in generative text-to-image models such as DALL-E,
Imagen or Stable Diffusion. It is inspired by RealToxicityPrompts, which is a
benchmark for measuring toxic degeneration in language models. However, since
these prompts do not describe visual content, it is not applicable to text
conditioned image generation. The purpose of I2P is to fill this gap. The I2P
benchmark dataset and accompanying testbed can be used to measure the degree
to which a model generates images that represent the concepts of hate,
harassment, violence, self-harm, sexual content, shocking images, and illegal
activity.

  2. Q2

Who created the dataset (e.g., which team, research group) and on behalf of
which entity (e.g., company, institution, organization)?

     * •

This dataset is presented by a research group located at the Technical
University Darmstadt, Germany, affiliated with the Hessian Center for AI
(hessian.AI), Aleph Alpha and LAION.

  3. Q3

Who funded the creation of the dataset? If there is an associated grant,
please provide the name of the grantor and the grant name and number.

     * •

The creation of the dataset was support by the German Center for Artificial
Intelligence (DFKI) project “SAINT” and the Federal Ministry of Education and
Research (BMBF) under Grant No. 01IS22091. Furthermore, it benefited from the
ICT-48 Network of AI Research Excellence Center “TAILOR” (EU Horizon 2020, GA
No 952215), the Hessian research priority program LOEWE within the project
WhiteBox, and the Hessian Ministry of Higher Education, and the Research and
the Arts (HMWK) cluster projects “The Adaptive Mind” and “The Third Wave of
AI”.

  4. Q4

Any other comments?

     * •

No.

###  J.2 Composition

  5. Q5

What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)? Are there multiple types of instances (e.g.,
movies, users, and ratings; people and interactions between them; nodes and
edges)? Please provide a description.

     * •

The dataset contains textual image descriptions potentially leading to images
displaying inappropriate content. This includes the concepts hate, harassment,
violence, self-harm, sexual content, shocking images and illegal activity. On
average, the prompts are made up of 20 tokens, and we could not observe an
apparent correlation between frequent words and the connection to
inappropriate images of these prompts.

We made our dataset openly available on https://huggingface.co/datasets/AIML-
TUDA/i2p.

  6. Q6

How many instances are there in total (of each type, if appropriate)?

     * •

I2P contains 4703 unique prompts assigned to at least one of the seven
categories above.

  7. Q7

Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set? If the dataset is a
sample, then what is the larger set? Is the sample representative of the
larger set (e.g., geographic coverage)? If so, please describe how this
representativeness was validated/verified. If it is not representative of the
larger set, please describe why not (e.g., to cover a more diverse range of
instances, because instances were withheld or unavailable).

     * •

For the seven concepts mentioned above, we used 26 keywords and phrases
describing them in more detail and collected up to 250 real-world text prompts
for each. For a given keyword, we crawled the prompts of the top 250 images
returned by https://lexica.art. Lexica is a collection of real-world, user-
generated prompts for Stable Diffusion sourced from its official discord.

  8. Q8

What data does each instance consist of? “Raw” data (e.g., unprocessed text or
images) or features? In either case, please provide a description.

     * •

Along the provided textual prompts we provide following meta information:

       * –

categories: One or multiple of the 7 categories mentioned above that yield
this prompt.

       * –

hard: Indicates if this is a hard prompt to remove inappropriate content from.
Prompts are labeled as hard if at least half of their generated images are
inappropriate.

       * –

inappropriate percentage: Percentage of inappropriate images as classified by
Q16 and NudeNet out of 10 generated images using Stable Diffusion.

       * –

nudity percentage: Percentage of images as depicting explicit nudity as per
the NudeNet out of 10 generated images using Stable Diffusion.

       * –

q16 percentage Percentage of inappropriate images as classified by Q16 out of
10 generated images using Stable Diffusion.

       * –

stable diffusion’s safety percentage: Percentage of inappropriate images as
classified by the SD NSFW safety checker out of 10 generated images using
Stable Diffusion.

       * –

prompt toxicity: The toxicity score of the text prompt according to the
PerspectiveAPI.

       * –

lexica url: URL to the original prompt and the respective images in lexica for
reference.

       * –

stable diffusion’s seed: Stable diffusion seed used in our image generation.

       * –

stable diffusion’s guidance scale: Stable diffusion guidance scale used in our
image generation.

       * –

stable diffusion’s image width: Stable diffusion image width used in our image
generation.

       * –

stable diffusion’s image height: Stable diffusion image height used in our
image generation.

  9. Q9

Is there a label or target associated with each instance? If so, please
provide a description.

     * •

There is no hard class label, but each prompt is assigned to at least one of
the categories hate, harassment, violence, self-harm, sexual content, shocking
images and illegal activity. Further, we provide toxicity score of the text
prompt according to the PerspectiveAPI. And a flag (‘hard’) indicating if this
is a hard prompt to remove inappropriate content from. Prompts are labeled as
hard if at least half of their generated images are inappropriate using Stable
Diffusion.

  10. Q10

Is any information missing from individual instances? If so, please provide a
description, explaining why this information is missing (e.g., because it was
unavailable). This does not include intentionally removed information, but
might include, e.g., redacted text.

     * •

No.

  11. Q11

Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)? If so, please describe how these
relationships are made explicit.

     * •

No.

  12. Q12

Are there recommended data splits (e.g., training, development/validation,
testing)? If so, please provide a description of these splits, explaining the
rationale behind them.

     * •

No.

  13. Q13

Are there any errors, sources of noise, or redundancies in the dataset? If so,
please provide a description.

     * •

Image retrieval in lexica is based on the similarity of an image and search
query in CLIP embedding space. Therefore, the collected prompts are not
guaranteed to generate inappropriate content, but the probability is high, as
demonstrated in our manuscript’s evaluation.

  14. Q14

Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)? If it links to or
relies on external resources, a) are there guarantees that they will exist,
and remain constant, over time; b) are there official archival versions of the
complete dataset (i.e., including the external resources as they existed at
the time the dataset was created); c) are there any restrictions (e.g.,
licenses, fees) associated with any of the external resources that might apply
to a future user? Please provide descriptions of all external resources and
any restrictions associated with them, as well as links or other access
points, as appropriate.

     * •

This dataset is self-contained. Since it is crawled from a database containing
user-generated textual prompts to generate images, we provide a link to each
prompt’s origin also displaying the resulting images. While not relevant for
the datasets purpose to benchmark image-generative models, we provide all the
necessary information to reproduce the original images.

  15. Q15

Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor–patient
confidentiality, data that includes the content of individuals’ non-public
communications)? If so, please provide a description.

     * •

No.

  16. Q16

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety? If so, please
describe why.

     * •

Since the purpose of the dataset is to evaluate the extent to which a model
produces inappropriate images, it naturally contains data that falls into this
category. However, we could not observe an apparent correlation between
frequent words and the connection to inappropriate images of these prompts.
Specifically, we only find a weak correlation between the toxicity of a prompt
and the inappropriateness of images it generates. In fact, prompts with low
toxicity scores still have unforeseen high probabilities of generating
inappropriate images. Furthermore, out of 4702 prompts, a mere 1.5% are toxic.

  17. Q17

Does the dataset relate to people? If not, you may skip the remaining
questions in this section.

     * •

In a few cases, a prompt contains the names of real people.

  18. Q18

Does the dataset identify any subpopulations (e.g., by age, gender)?

     * •

No.

  19. Q19

Is it possible to identify individuals (i.e., one or more natural persons),
either directly or indirectly (i.e., in combination with other data) from the
dataset? If so, please describe how.

     * •

No.

  20. Q20

Does the dataset contain data that might be considered sensitive in any way
(e.g., data that reveals racial or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)? If so,
please provide a description.

     * •

No.

  21. Q21

Any other comments?

     * •

No.

###  J.3 Collection Process

  22. Q22

How was the data associated with each instance acquired? Was the data directly
observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey
responses), or indirectly inferred/derived from other data (e.g., part-of-
speech tags, model-based guesses for age or language)? If data was reported by
subjects or indirectly inferred/derived from other data, was the data
validated/verified? If so, please describe how.

     * •

For the seven concepts mentioned above, we used 26 keywords and phrases:

       * –

hate: “hate”

       * –

harassment: “mocking”, “bullying”, “threatening”, “harassment”

       * –

violence: “humiliation”, “violence acts”, “suffering”, “violence”

       * –

self-harm: “suicide”, “eating disorder”, “self-harm”, “attempts at harming
oneself”

       * –

sexual: “sexual acts”, “content meant to arouse sexual excitement”, “sexual”,
“nudity”

       * –

shocking: “profane subjects that may shock or disgust”, “bodily fluids”,
“shocking”, “obscene gestures”

       * –

illegal activity: “vandalism”, “illegal activity”, “drug use”, “theft”

describing them in more detail and collected up to 250 real-world text prompts
for each. For a given keyword, we crawled the prompts of the top 250 images
returned by https://lexica.art. Lexica is a collection of real-world, user-
generated prompts for SD sourced from its official discord server. It stores
the prompt, seed, guidance scale, and image dimensions used in the generation
to facilitate reproducibility. Image retrieval in lexica is based on the
similarity of an image and search query in CLIP embedding space. Therefore,
the collected prompts are not guaranteed to generate inappropriate content,
but the probability is high, as demonstrated in our evaluation.

  23. Q23

What mechanisms or procedures were used to collect the data (e.g., hardware
apparatus or sensor, manual human curation, software program, software API)?
How were these mechanisms or procedures validated?

     * •

We ran a preprocessing script in python, over multiple of small CPU nodes to
extract the prompts from https://lexica.art. They were validated by manual
inspection of the results and post processing using the PerspectiveAPI and
Stable Diffusion to create further meta information such as the label “hard”
and the prompts toxicity score, as described before.

  24. Q24

If the dataset is a sample from a larger set, what was the sampling strategy
(e.g., deterministic, probabilistic with specific sampling probabilities)?

     * •

Image retrieval in lexica is based on the similarity of an image and search
query in CLIP embedding space. We used the top 250 query results to given
keywords.

  25. Q25

Who was involved in the data collection process (e.g., students, crowdworkers,
contractors) and how were they compensated (e.g., how much were crowdworkers
paid)?

     * •

No crowdworkers were used in the collection process of the dataset. Co-authors
of the corresponding manuscript wrote the collection scripts and validated the
data.

  26. Q26

Over what timeframe was the data collected? Does this timeframe match the
creation timeframe of the data associated with the instances (e.g., recent
crawl of old news articles)? If not, please describe the timeframe in which
the data associated with the instances was created.

     * •

The data was collected from September 2022 to October 2022, but those who
created the crawled prompts might have included content from before then. A
certain date for a prompt is not available but based on the release date of
Stable Diffusion they were created in 2022.

  27. Q27

Were any ethical review processes conducted (e.g., by an institutional review
board)? If so, please provide a description of these review processes,
including the outcomes, as well as a link or other access point to any
supporting documentation.

     * •

We corresponded with the ethical guidelines of Technical University of
Darmstadt.

  28. Q28

Does the dataset relate to people? If not, you may skip the remaining
questions in this section.

     * •

No.

  29. Q29

Did you collect the data from the individuals in question directly, or obtain
it via third parties or other sources (e.g., websites)?

     * •

We retrieve the data from https://lexica.art which provides an API to crawl
its content.

  30. Q30

Were the individuals in question notified about the data collection? If so,
please describe (or show with screenshots or other information) how notice was
provided, and provide a link or other access point to, or otherwise reproduce,
the exact language of the notification itself.

     * •

N/A

  31. Q31

Did the individuals in question consent to the collection and use of their
data? If so, please describe (or show with screenshots or other information)
how consent was requested and provided, and provide a link or other access
point to, or otherwise reproduce, the exact language to which the individuals
consented.

     * •

N/A

  32. Q32

If consent was obtained, were the consenting individuals provided with a
mechanism to revoke their consent in the future or for certain uses? If so,
please provide a description, as well as a link or other access point to the
mechanism (if appropriate).

     * •

N/A

  33. Q33

Has an analysis of the potential impact of the dataset and its use on data
subjects (e.g., a data protection impact analysis) been conducted? If so,
please provide a description of this analysis, including the outcomes, as well
as a link or other access point to any supporting documentation.

     * •

The benchmark’s dataset was analyzed and used to evaluate Stable Diffusion in
version 1.4 and 2.0. The results are openly available at
https://arxiv.org/abs/2211.05105.

  34. Q34

Any other comments?

     * •

No.

###  J.4 Preprocessing, Cleaning, and/or Labeling

  35. Q35

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization
or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction,
removal of instances, processing of missing values)? If so, please provide a
description. If not, you may skip the remainder of the questions in this
section.

     * •

The data collection described above yielded duplicate entries, as some
retrieved images were found among multiple keywords. These duplicates were
removed. We provide the raw textual prompt along with meta information which
was collected using Stable Diffusion itself as well as the PerspectiveAPI
(https://github.com/conversationai/perspectiveapi).

  36. Q36

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data
(e.g., to support unanticipated future uses)? If so, please provide a link or
other access point to the “raw” data.

     * •

Textual prompts are provided as raw data.

  37. Q37

Is the software used to preprocess/clean/label the instances available? If so,
please provide a link or other access point.

     * •

To post-process the data we used:

       * –

https://github.com/conversationai/perspectiveapi resulting in the toxicity
score of a prompt.

       * –

https://huggingface.co/CompVis/stable-diffusion-v1-4 to generate images in
order to create further labels using the two following tools.

       * –

https://github.com/ml-research/Q16 a tool to classify the inappropriateness of
a image.

       * –

https://github.com/notAI-tech/NudeNet a tool classify whether an image
contains nude/sexual content.

  38. Q38

Any other comments?

     * •

No.

###  J.5 Uses

  39. Q39

Has the dataset been used for any tasks already? If so, please provide a
description.

     * •

The dataset has been used to evaluate the inappropriate degeneration in Stable
Diffusion (https://arxiv.org/abs/2211.05105).

  40. Q40

Is there a repository that links to any or all papers or systems that use the
dataset? If so, please provide a link or other access point.

     * •

No.

  41. Q41

What (other) tasks could the dataset be used for?

     * •

The dataset should only be used to measure inappropriate degeneration in text-
conditioned image generators.

  42. Q42

Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses? For
example, is there anything that a future user might need to know to avoid uses
that could result in unfair treatment of individuals or groups (e.g.,
stereotyping, quality of service issues) or other undesirable harms (e.g.,
financial harms, legal risks) If so, please provide a description. Is there
anything a future user could do to mitigate these undesirable harms?

     * •

The dataset was collected based on images generated by Stable Diffusion.
Further advances in AI-driven image generation could lead to novel issues,
i.e. risks related to inappropriate content. Further, inappropriateness is not
limited to these seven concepts, varies between cultures, and constantly
evolves. Here we restricted ourselves to images displaying tangible acts of
inappropriate behavior.

  43. Q43

Are there tasks for which the dataset should not be used? If so, please
provide a description.

     * •

It should not be used to increase the inappropriateness of AI-generated
images.

  44. Q44

Any other comments?

     * •

No.

###  J.6 Distribution

  45. Q45

Will the dataset be distributed to third parties outside of the entity (e.g.,
company, institution, organization) on behalf of which the dataset was
created? If so, please provide a description.

     * •

Yes, the dataset will be open-source.

  46. Q46

How will the dataset be distributed (e.g., tarball on website, API, GitHub)?
Does the dataset have a digital object identifier (DOI)?

     * •

The data will be available through Huggingface datasets.

  47. Q47

When will the dataset be distributed?

     * •

December 2022 and onward.

  48. Q48

Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)? If so,
please describe this license and/or ToU, and provide a link or other access
point to, or otherwise reproduce, any relevant licensing terms or ToU, as well
as any fees associated with these restrictions.

     * •

MIT license

  49. Q49

Have any third parties imposed IP-based or other restrictions on the data
associated with the instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any relevant
licensing terms, as well as any fees associated with these restrictions.

     * •

The institutions mentioned above own the metadata and release as MIT license.

     * •

We do not own the copyright of the text.

  50. Q50

Do any export controls or other regulatory restrictions apply to the dataset
or to individual instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any
supporting documentation.

     * •

No.

  51. Q51

Any other comments?

     * •

No.

###  J.7 Maintenance

  52. Q52

Who will be supporting/hosting/maintaining the dataset?

     * •

Huggingface will support hosting of the metadata.

     * •

The creators will maintain the samples distributed.

  53. Q53

How can the owner/curator/manager of the dataset be contacted (e.g., email
address)?

     * •

{schramowski<EMAIL_ADDRESS>
  54. Q54

Is there an erratum? If so, please provide a link or other access point.

     * •

There is no erratum for our initial release. Errata will be documented as
future releases on the dataset website.

  55. Q55

Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)? If so, please describe how often, by whom, and
how updates will be communicated to users (e.g., mailing list, GitHub)?

     * •

I2P will not be updated unless there is a substantial reason. However a future
I2P could contain more concepts of inappropriateness and updated notions.
Specific samples can be removed on request.

  56. Q56

If the dataset relates to people, are there applicable limits on the retention
of the data associated with the instances (e.g., were individuals in question
told that their data would be retained for a fixed period of time and then
deleted)? If so, please describe these limits and explain how they will be
enforced.

     * •

People may contact us at {schramowski<EMAIL_ADDRESS>to add
specific samples to a blacklist.

  57. Q57

Will older versions of the dataset continue to be supported/hosted/maintained?
If so, please describe how. If not, please describe how its obsolescence will
be communicated to users.

     * •

N/A.

  58. Q58

If others want to extend/augment/build on/contribute to the dataset, is there
a mechanism for them to do so? If so, please provide a description. Will these
contributions be validated/verified? If so, please describe how. If not, why
not? Is there a process for communicating/distributing these contributions to
other users? If so, please provide a description.

     * •

Unless there are grounds for significant alteration to certain samples,
extension of the dataset will be carried out on an individual basis.

  59. Q59

Any other comments?

     * •

No.