(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: WMG, University of Warwick, Coventry CV4 7AL, U.K. 11email:
<EMAIL_ADDRESS>
22institutetext: Computer Science Department, University of Liverpool,
Liverpool L69 3BX, U.K.
22email<EMAIL_ADDRESS>
# ProTIP: Probabilistic Robustness Verification on Text-to-Image Diffusion
Models against Stochastic Perturbation

Yi Zhang 11 Yun Tang 11 Wenjie Ruan 22 Xiaowei Huang 22 Siddartha Khastgir 11
Paul Jennings 11 Xingyu Zhao(✉) 11

###### Abstract

Text-to-Image (T2I) Diffusion Models (DMs) have shown impressive abilities in
generating high-quality images based on simple text descriptions. However, as
is common with many Deep Learning (DL) models, DMs are subject to a lack of
robustness. While there are attempts to evaluate the robustness of T2I DMs as
a binary or worst-case problem, they cannot answer how robust in general the
model is whenever an adversarial example (AE) can be found. In this study, we
first introduce a probabilistic notion of T2I DMs’ robustness; and then
establish an efficient framework, ProTIP, to evaluate it with statistical
guarantees. The main challenges stem from: i) the high computational cost of
the generation process; and ii) determining if a perturbed input is an AE
involves comparing two output distributions, which is fundamentally harder
compared to other DL tasks like classification where an AE is identified upon
misprediction of labels. To tackle the challenges, we employ sequential
analysis with efficacy and futility early stopping rules in the statistical
testing for identifying AEs, and adaptive concentration inequalities to
dynamically determine the “just-right” number of stochastic perturbations
whenever the verification target is met. Empirical experiments validate the
effectiveness and efficiency of ProTIP over three common T2I DMs. Finally, we
demonstrate an application of ProTIP to rank commonly used defence methods.

###### Keywords:

Diffusion Models, Probabilistic Robustness, Safe AI

##  1 Introduction

Text-to-Image (T2I) Diffusion Models (DMs) have made impressive strides in
recent years. State-of-the-art (SOTA) models, like DALL-E 3[3], Imagen [43],
and Parti [58] can generate high-quality images from textual descriptions.
Additionally, Stable Diffusion [41] represents a successful large-scale
implementation of the latent diffusion model. With the advent of T2I DMs, we
can now create visually stunning images with simple text prompts.

However, studies [65, 10, 11] have demonstrated that small perturbations to
the textual inputs can substantially degrade the performance of T2I DMs. Fig.
1 illustrates examples where introducing minor perturbations to the prompt
leads to a change in the generated image outcome, which raises concerns about
the model’s robustness in the context of its downstream applications [5, 30].
A pivotal question arises: how can we systematically evaluate and verify (when
a specific verification target is provided) the robustness of T2I DMs?

Figure 1: Examples illustrating perturbations applied to the prompt for Stable
Diffusion, employing two methods as described in Sec. 3.2

The problem of lacking robustness in T2I DMs is not unexpected, considering
that robustness issues are pervasive in the broader realm of Deep Learning
(DL) [45, 14]. Generally, robustness is defined as the invariant decision of
the DL model against small perturbations on inputs. A small perturbation on an
input is termed as an Adversarial Example (AE) if it leads to a different
prediction from the label assigned to the original input. Over the past
decade, numerous studies have attempted to frame DL robustness verification as
a binary or worst-case problem, addressing questions like “if AEs exist (in a
small local region of the input)?” or “what is the closest AE to the input?”
[59, 6, 21]. Emerging studies are adopting a probabilistic view, formulating
robustness verification as a statistical inference problem [53, 48, 62, 47,
20, 8, 7]. Such probabilistic robustness notion is arguably of more practical
interest than binary/worst-case ones, because it provides an overall
evaluation of how robust the model is whenever an AE can be found [53, 20] and
accepts residual risks that are more realistic to achieve [48, 62]. We concur
with this view and argue that T2I DMs also necessitate probabilistic
robustness verification, which, to the best of our knowledge, is absent in
SOTA.

In this work, we first formally define the probabilistic robustness of T2I DMs
against character-level stochastic perturbation; then, we devise an efficient
framework, named ProTIP, to evaluate and verify it (assuming a verification
target is given). In addressing the statistical inference problems within
ProTIP, we encounter new challenges specific to the generative nature of DMs.
These challenges, distinct from traditional DL tasks like classification or
regression, include: i) the generation process is very computationally
insensitive; ii) the identification of a perturbed input as an AE entails
comparing distributional differences, which is fundamentally harder compared
to DL tasks like classification where an AE is identified by a change of the
prediction label. To tackle the challenges, we employ sequential analysis with
efficacy and futility early stopping rules in the statistical testing for
indicating AEs, and adaptive concentration inequalities to dynamically
determine the “just-right” number of stochastic perturbations whenever the
verification target is met. Experimental results validate the effectiveness
and efficiency of our ProTIP, while an application of ProTIP is also presented
to showcase its use in ranking common defences against text perturbations. In
summary, key contributions of this paper include:

  * •

Problem formulation: For the first time, we formulate the probabilistic
robustness verification problem for T2I DMs against stochastic perturbation,
with formalised evaluation metrics.

  * •

Efficient solution: To solve the formulated problem of probabilistic
robustness verification for T2I DMs, we have developed an efficient framework,
ProTIP, which incorporates several sequential analysis methods to dynamically
determine the sample size and thus enhance ProTIP’s efficiency.

  * •

Open-source repository: A public repository containing the codes for ProTIP,
along with datasets, models, and experimental results, is available at
https://github.com/wellzline/ProTIP/.

##  2 Preliminaries and Related Work

###  2.1 Text-to-Image Diffusion Models

Generative AI has been thriving in the multi-modal field, with diffusion
models emerging as a powerful new family of deep generative models with
record-breaking performance in many applications, including video generation
[56], image reconstruction [46], and T2I generation [3]. DMs are a class of
probabilistic generative models that apply a noise injection process, followed
by a reverse procedure for sample generation. Importantly, DMs allow a notion
of control at generation time through so-called guidance, enabling several
complicated generative tasks, among which T2I DMs are examples of guided image
generation tasks using descriptive text as the guidance signal. Models like
OpenAI’s Stable Diffusion [41] and Google’s Imagen [43], trained on large-
scale datasets of annotated text-image pairs, can generate high-quality images
based on simple natural language descriptions. In this context, Glide [35]
emerges as a text-guided diffusion model that not only supports image
generation but also excels in the realm of image editing. Moreover, commercial
products such as DALL-E 2 and Midjourney [1] have showcased stunning effects
across various scenarios, contributing to the richness and diversity of text-
to-image applications. We refer readers to Appendix 0.A for more details on
T2I DMs.

###  2.2 Deep Learning Robustness

DL models, such as Neural Networks, are notoriously unrobust to small
perturbations [59, 6, 21]. While definitions of robustness vary in literature,
they share a common intuition that a DL model’s decision should remain
invariant against small perturbations on a given input—typically it is defined
as all inputs in a region $\eta$ have the same prediction label, where $\eta$
is a small norm ball (in a $L_{p}$-norm distance) of radius $\gamma$ around an
input $x$ (normally called the test seed). A perturbed input (e.g., by adding
noise on $x$) $x^{\prime}$ within $\eta$ is an AE if its prediction label
differs from $x$.

In the Safe AI community, DL robustness has been the property in the
spotlight. Many studies on evaluating the DL robustness have been done, and
they frame the problem in different ways. In Fig. 2, we summarise 4 common
ways of formulating the problem [8]. Earlier works, such as [13, 23, 42],
formulate the verification problem as a binary question by asking if any AEs
can be found within a given input norm-ball of a specified radius, cf. Fig.
2(a). Such “binary robustness” can be normally evaluated in two ways [23, 42,
57]: either through reachability algorithms that aim to determine the lower
and upper bounds of the output within the input norm-ball $\eta$, using layer-
by-layer analysis; or by solving it using SAT/SMT solvers as a variety of
constraint-based programming problems. Fig. 2(b) poses a similar yet different
question: what is the maximum radius of $\eta$ such that no AE exists within
it? Intuitively, it is finding the “largest safe distance” for the test seed
$x$ [2, 33, 54, 55]. In Fig. 2(c), on the other hand, it evaluates the model’s
robustness by introducing adversarial attacks to cause the maximum prediction
loss within the specified norm ball $\eta$. This method is often applied in
adversarial training to enhance the model’s robustness to resist attacks [31,
50].

Figure 2: Four common formulations of robustness verification in DL—binary
(a), worst-case (b & c), and probabilistic (d) robustness.

The three aforementioned methods all estimate the robustness of the model by
detecting the presence of AE or the AE that gives the maximum loss/safe-
radius. This type can be called deterministic robustness [62]. As argued by
[53], the three aforementioned methods suffer from two major drawbacks: i)
they fail to convey how robust the model is whenever an AE is found; ii) they
pose scalability challenges when the model is large. To overcome these
limitations, more recent works [53, 48, 62, 47, 20, 8, 7] develop a new
probabilistic view, by defining robustness as the proportion of AEs inside the
norm-ball $\eta$, cf. Fig. 2(d). This probabilistic metric is arguably more
practical for two reasons: i) binary/worst-case robustness focusing on extreme
cases is neither necessary nor realistic, especially when the model is large;
what is more relevant is knowing the proportion of AEs; ii) since all
practical applications have an acceptable level of risk, it suffices to
demonstrate that the probability of a violation is below a required threshold,
rather than confirming it to be exactly zero. Without loss of generality, we
illustrate probabilistic robustness of a DL classification task as:

######  Definition 1 (Probabilistic Robustness)

For a DL classifier $f$ that takes input $x\in\mathcal{X}$ and returns a
prediction label, the probabilistic robustness of an input $x$ in a norm ball
of radius $\gamma$, denoted as $B(x,\gamma)$, is:

$R(x,\gamma):=\int_{x^{\prime}\in
B(x,\gamma)}I_{\\{f(x^{\prime})=f(x))\\}}(x^{\prime})Pr(x^{\prime})\,dx^{\prime}$
(1)

where $I_{\mathcal{S}}(x)$ is an indicator function—it is equal to 1 when
$\mathcal{S}$ is true and equal to 0 otherwise; $Pr(\cdot)$ is the local
distribution of inputs representing how perturbations $x^{\prime}$ are
generated, which is precisely the “input model” used by [53, 54].

In this paper, we re-frame such a generic definition of probabilistic
robustness for T2I DMs and provide an efficient solution to its verification.

###  2.3 Robustness of Text-to-Image Diffusion Models

Despite variations of definitions, the key to evaluating and improving the
model’s robustness is detecting AEs. The approaches to this end are often
referred to as “adversarial attacks” (i.e., via input perturbations). For T2I
models, adversarial attack methods can also be classified into character-
level, word-level, sentence-level or multi-level, depending on the granularity
of input perturbations.

Recent studies [32, 10] have shown that T2I DMs are very sensitive to black-
box attacks like text perturbation. Although such perturbations are commonly
referred to as “attacks”, they are not necessarily malicious actions from
attackers [64]. They may also represent natural human errors that follow a
stochastic generation process. In [11], it proved that T2I DMs are vulnerable
to realistic errors (e.g., typo, glyph, phonetic) that humans can make and
reveal their non-trivial robustness issues. The work in [65] revealed the
vulnerability of T2I models and attributes it to the weak robustness of the
used text encoders. Aforementioned existing works only revealed the lack of
robustness of T2I DMs in the face of text perturbations and focused on
deterministic robustness (e.g., Maximised Prediction Loss). To the best of our
knowledge, there exists no dedicated exploration into the probabilistic
robustness of T2I models when they subject to stochastic perturbations, and
our ProTIP is the first verification framework for this problem, backed by
statistical guarantees. Moreover, ProTIP addresses unique challenges arising
from the generative characteristics of T2I DMs, by adopting sequential
analysis and adaptive concentration inequality to improve the verification
efficiency, details of which are provided in the following section.

##  3 Method: ProTIP

###  3.1 Problem Statement

A T2I DM that takes a text input $x\in\mathcal{X}$ and generates an image
$y\in\mathcal{Y}$ essentially characterises the conditional distribution
$Pr(Y\mid X=x)$111As usual, we use capital letters to denote random variables
and lower case letters for their specific realisations; $Pr(X)$ is used to
represent the distribution of variable $X$., i.e., the T2I DM is a function
$f:\mathcal{X}\rightarrow\mathcal{D}(\mathcal{Y})$ where $\mathcal{D}$
represents the space of all possible distributions over the image set
$\mathcal{Y}$. Accordingly, the general probabilistic robustness Def. 1 needs
to be adapted for T2I DMs as:

######  Definition 2 (Probabilistic Robustness of T2I DMs)

For a T2I DM $f$ that takes text inputs $X$ and generates a conditional
distribution of images $Pr(Y\mid X)$, the probabilistic robustness of the
given input $x$ is:

$R_{M}{(x,\gamma)}=\sum_{x^{\prime}:s(T(x),T(x^{\prime}))\geq\gamma}I_{\\{Pr(Y\mid
X=x)=Pr(Y\mid X=x^{\prime})\\}}(x^{\prime})Pr(x^{\prime})$ (2)

where $T$ denotes the CLIP [40] model’s text encoder, $s$ is a similarity
measurement function (e.g., cosine similarity), $\gamma$ denotes a given
threshold on similarity. While $I$ is an indication function as defined in
Def. 1, its value now depends on whether the output distributions before and
after the perturbation differ. $Pr(x^{\prime})$ indicates the probability that
$x^{\prime}$ is the next perturbed text generated.

Intuitively, Def. 2 suggests that $R_{M}(x,\gamma)$ is the expected
probability that the output image distribution remains unchanged for a random
perturbation $x^{\prime}$ that preserves a similar semantic meaning to $x$
(ensured by $s(T(x),T(x^{\prime}))\geq\gamma$). A “frequentist” interpretation
of $R_{M}(x)$222Notation-wise, we omit $\gamma$ from $R_{M}$ where $\gamma$
represents a hyperparameter thereafter., following the gist of Fig. 2(d), is:
it is the limiting relative frequency of perturbations for which the output
distribution is preserved, in an infinite sequence of independently generated
perturbations.

######  Definition 3 (Verification Target)

The probabilistic robustness of the T2I DM $f$ when processing an input $x$
cannot be less than a certain lower bound $b_{l}$, with sufficient confidence
$1-\sigma$, i.e.:

$Pr(R_{M}(x)\geq b_{l})\geq 1-\sigma$ (3)

where the pair $(b_{l},\sigma)$ is the given verification target.

To determine if a verification target is met for the input $x$, we propose the
ProTIP workflow in Fig. 3, addressing the following questions: i) How to
generate stochastic perturbations $x^{\prime}$ (i.e., the implementation of
$Pr(x^{\prime})$ in Eq. (2)); ii) Given a perturbed input $x^{\prime}$, how to
determine if its new output distribution is significantly different to the
original one (i.e., the implementation of $I(x^{\prime})$ in Eq. (2)); iii)
How to do decision makings based on statistical evaluations of $R_{M}(x)$ over
a sequence of perturbations (i.e., even if we implemented both
$Pr(x^{\prime})$ and $I(x^{\prime})$ in Eq. (2), the true $R_{M}(x)$ is still
unknown due to the fact that we cannot exhaustively take the sum of their
product over all possible perturbations $x^{\prime}$; thus, we can only
estimate it from a finite sample of perturbed inputs).

While the aforementioned question i) is established for which we adopt SOTA
methods, e.g., [34], to generate stochastic perturbations on text (cf. Sec.
3.2), questions ii) and iii) are relatively challenging. Because, both
$Pr(Y\mid X=x)$ and $Pr(Y\mid X=x^{\prime})$ are unknown non-parametric
distributions that require sampling images from them (which is costly for T2I
DMs) to determine if they are significantly different. To tackle, we propose
the sequential analysis with early stopping rules and adaptive concentration
inequality for the last two questions, corresponding to the “inner loop” and
“outer loop”of Fig. 3, respectively.

Figure 3: Workflow of ProTIP.

###  3.2 Stochastic Perturbation Generation

While ProTIP can cope with any text perturbations generated stochastically,
without loss of generality (WLOG), we study five character-level text
perturbation methods from [34]: insert, substitute, swap, delete, and keyboard
error. First, we set the perturbation rate like 10%, indicating that 10% of
the words in the sentence will be perturbed. Then, one of the six perturbation
methods as shown in Table 1 will be randomly selected. Taking insertion
perturbation as an example, a random letter is chosen from the 26 English
alphabets. Subsequently, a random word in the sentence is selected, and a
random position within that word is chosen to insert the selected letter.
Next, check if the 10% perturbation rate is met by the perturbed sentence. If
not, continue the perturbation process until we meet the perturbation rate
requirement; if yes and not a duplicate, stop perturbation. Table 1 lists the
details and examples of perturbations operations given the sample input “A red
ball on green grass under a blue sky”.

Table 1: Examples of stochastic text perturbations

Perturbation |  Description |  Example
---|---|---
Insert |  Insert a character randomly |  A reed ball on green grass under a blue sky.
Substitute |  Substitute a character randomly |  A red ball on green grass under a blae sky.
Swap |  Swap two characters randomly |  A red ball on green garss under a blue sky.
Delete |  Delete a character randomly |  A red ball on reen grass under a blue sky.
Keyboard |  Substitute a char. by keyboard distance |  A red ball on green grass under blue sjy.

As Def. 1, the intuition behind robustness suggests that the outputs of a DL
model should remain invariant to perturbations on the inputs, while the
inputs, before and after the perturbation, should be deemed very similar—thus,
Def. 1 restricts the perturbation in a norm-ball $B(x,\gamma)$ with a small
radius $\gamma$. Such perturbation distance control is easily achievable in
Computer Vision where the inputs (either raw pixel values or the latent space
feature values) can be normalised and projected into a continuous input space.
However, for DL models take text inputs (like T2I DMs), the discrete nature of
text make it more challenging to study robustness [27] as a single character
change may completely alter the semantics meaning. E.g., “a red ball on …” and
“a red bell on …” have totally different semantic meanings with only one
character perturbed. Thus, to study the robustness of T2I DMs, first we need
to ensure that the perturbed text $x^{\prime}$ has the same semantics to the
original text $x$. Following the idea from [11, 65, 29, 60] for the same
problem, we employ CLIP [40] as the model for text encoding, leveraging CLIP’s
ability to represent image and text information while preserving their
relationships. That is, as per Def. 2, we use CLIP’s text encoder $T$ to
extract the original embeddings $T(x)$ and perturbed sentence embeddings
$T(x^{\prime})$ to compute the similarity score $s(T(x),T(x^{\prime}))$ (e.g.,
by cosine similarity). Only when the similarity score is greater than a given
threshold $\gamma$, we deem $x^{\prime}$ is a valid perturbation, where
$\gamma$ can be determined by the method developed in [27] (which yields the
maximum perturbation distance in the continuous embedding space that preserves
the semantics meaning for a given text). In this way, we ensure that only
valid perturbations (sharing similar semantics as the original text) are used
for robustness evaluation in ProTIP.

###  3.3 Indication of Adversarial Examples

For a given perturbed input $x^{\prime}$, whether it is an AE depends on if
the T2I DM output distribution $Pr(Y\mid X=x^{\prime})$ is significantly
different from $Pr(Y\mid X=x)$. In this section, we describe the details of
how ProTIP solves the question as a sequential, two-sample statistical testing
problem, shown as the “inner loop” in Fig. 3. Although there are techniques
capable of measuring the distance between two given distributions, e.g., the
KL divergence [24] and the Maximum Mean Discrepancy [16], they are not a test
and thus they do not provide a statistically principled way to determine
whether or not the distributions are different.

####  3.3.1 Statistical hypothesis testing for indicating AE.

First, a set of images is generated for the perturbed input $x^{\prime}$,
denoted as $\\{y^{\prime}_{1},\dots,y^{\prime}_{n}\\}$, i.e., a set of
independent and identically distributed (i.i.d.) samples from the distribution
$Pr(Y\mid X=x^{\prime})$. Off-the-shelf two-sample test tools do not perform
well for high-dimensional data like images [16]. Thus, similar to [11], we
adapt CLIP Score to evaluate the correlation between a generated caption for
an image and the actual content of the image, which has been found to be
highly correlated with human judgement [17]. The metric is formally defined as
[17]:

$CLIPScore(x,y):=max(100\cdot cos(T(x),V(y)),0)$ (4)

which corresponds to the cosine similarity between visual CLIP embedding
$V(y)$ for an image $y$ and textual CLIP embedding $T(x)$ for a caption $x$.
The score is bound between 0 and 100, and the closer to 100, the better. Then,
for each image $y^{\prime}_{i}$, we can calculate the “CLIP score”
$c^{\prime}_{i}$ as the coexistence likelihood of the image $y^{\prime}_{i}$
and the original text $x$, by measuring
$c^{\prime}_{i}:=CLIPScore(x,y_{i}^{\prime})$, yielding a set of i.i.d. CLIP
scores $\\{c^{\prime}_{1},\dots,c^{\prime}_{n}\\}$. We can repeat the same
calculation for each generated image $y_{i}$ from the original text $x$, and
collectively form another set of i.i.d. CLIP scores $\\{c_{1},\dots,c_{n}\\}$
corresponding to $x$.

Intuitively, a higher $c^{\prime}_{i}$ indicates a “less adversarial”
$x^{\prime}$ [11, 9]. Thus, if $x^{\prime}$ is not an AE, then the statistics
on the group of $c^{\prime}_{i}$s should not be significantly smaller than the
group of $c_{i}$s, for which we can do an one-sided statistical hypothesis
testing with the following null and alternative hypotheses:

  * •

$\mathcal{H}_{0}$: There is no difference in the two groups of CLIP scores.

  * •

$\mathcal{H}_{1}$: The CLIP scores of the $c^{\prime}_{i}$ group is smaller
than the $c_{i}$ group

While there are established statistical testing methods, e.g., the t-test,
u-test, and their variants [25], that can be employed to answer the above
question, their applicability depends on whether the data distribution meets
certain assumptions. Our framework ProTIP is compatible with any statistical
tests in this step, cf. later Sec. 4 for our choice in the experiments. We can
now implement the AE indicator function in Def. 2 as:

$\begin{split}&I{(x^{\prime})}=\begin{cases}\begin{aligned}
&1,\quad\text{accept the null hypothesis }\mathcal{H}_{0}\\\
&0,\quad\text{reject the null hypothesis
}\mathcal{H}_{0}\end{aligned}\end{cases}\end{split}$ (5)

####  3.3.2 Sequential analysis with early stopping rules.

In contrast to conventional DL tasks like classification, where detecting an
AE involves an “one-off” test to see if the prediction label changes,
determining if a perturbation affects the T2I DM requires multiple generations
and comparing the distributional differences between two sets of images
generated before and after the perturbation. This shift in the evaluation
approach leads to a significantly increased computational workload. To improve
the efficiency of solving the hypothetical testing in Eq. (5), we employ
sequential analysis with early stopping rules [22, 38, 51]. Instead of
collecting all data with the maximum sample size (fixed at the design time)
and then analysing the data a single time by the end, sequential analysis
conducts interim analyses during data collection, so that we can prematurely
stop data collection at an interim analysis upon rejecting or accepting the
null hypothesis. Thanks to early stopping, sequential designs will, on
average, require fewer samples [25, Chap. 10].

In ProTIP, we adopt sequential analysis with both Efficacy and Futility
stopping rules, by controlling the $\alpha$ (Type I error, false positive when
a null hypothesis is incorrectly rejected) and the $\beta$ (Type II error,
false negative when the null hypothesis is accepted and it is actually false),
respectively. Intuitively, the Efficacy stopping rule says, if the analysis
reveals a statistically significant result at some interim stage, data
collection can be terminated because we reject $\mathcal{H}_{0}$ when
observing the p-value $p<\alpha_{k}$ (where $\alpha_{k}$ is the cumulative
Type I error rate at the $k$-th interim analysis, controlled by alpha-spending
functions [25]). On the other hand, we may also stop for futility, which means
it is either impossible or very unlikely for the final analysis to yield
$p<\alpha$. This can be implemented by controlling the Type II error rate
across interim analysis using a beta-spending function. We refer readers to
[25, Chap. 10] for more details on the topic.

Specifically, we employ the R package rpact [52] to perform the sequential
analysis by setting Pocock type alpha/beta spending functions [15] to
determine the sample size and the thresholds for efficacy/futility early
stopping rules. Details are presented in Sec. 4 and Appendix 0.C.

###  3.4 Decision-making for Verification

While we establish a method of indicating AEs for a given perturbation
$x^{\prime}$ in the last section, to make the decision if the verification
target (cf. Def. 3) is met we need to estimate $R_{M}$ over a population of
perturbations, for which we propose the “outer loop” in Fig. 3 and explain
details in this section.

Concentration inequalities [4], such as Chernoff inequality, Azuma’s bound,
and Hoeffding’s inequality, represent important statistical analytical methods
extensively employed in ensuring dependable decision-making with probabilistic
guarantees. Specifically, in probability theory, Hoeffding’s inequality [19]
provides a bound on the probability that the sum of bounded independent random
variables deviates from its expected value by more than a certain amount. In
ProTIP, for an input $x$, the T2I DM’s probabilistic robustness $R_{M}(x)$ is
the proportion of AEs over a population of all possible perturbations on $x$.
Normally this population is very large (if not infinite)333The population size
depends on the length of the original text $x$ and perturbation parameters
like perturbation rate. Note, ProTIP can also cope with extreme (and simpler)
cases in which the perturbation population is small and its members can be
enumerated, e.g.. when $x$ is a single word with one character to be changed.,
thus the ground truth of $R_{M}(x)$ can only be estimated as the sample mean
of samples drawn from the population. Irrespective of the sample size, there
will inevitably be discrepancies between the sample mean and the ground truth
population mean. Nevertheless, Hoeffding’s inequality allows us to establish
tight probabilistic bounds on this error (cf. Appendix 0.B for more on
Hoeffding’s inequality). A limitation of Hoeffding’s inequality is its
requirement for a predetermined, process-independent sample size. However, in
most circumstances, we generally have no idea how many samples are sufficient
to verify the model, a priori. Thus, allocating a large sample size that
accommodates the budget can result in unnecessary waste.

Inspired by [63, 62], our ProTIP employs an “adaptive” version of Hoeffding’s
inequality, in which the sample size itself is a variable. This enables the
sampling process to stop as soon as the “just right” number of perturbations
has been tested for making the verification decision. Moreover, the following
theorem proves that the theoretical guarantee on the estimation errors is
preserved:

######  Theorem 3.1 (Adaptive Hoeffding’s Inequality)

We know $I(x^{\prime}_{i})$ is a binary 0–1 random variable. Let
$\hat{\mu}_{I}^{(n)}=\frac{1}{n}\sum_{i=1}^{n}I(x^{\prime}_{i})$ (i.e., the
sample mean). Also let $J$ be a random variable on
$\mathbb{N}\cup\\{\infty\\}$, and
$\varepsilon(\sigma,n)=\sqrt{\frac{0.6\cdot\log(\log_{1.1}n+1)+1.8^{-1}\cdot\log\left(24/{\sigma}\right)}{n}}$,
then we have:

$Pr\left(|\hat{\mu}_{I}^{(J)}-R_{M}|\leq\varepsilon(\sigma,J)\right)\geq
1-\sigma$ (6)

where $R_{M}$ is the true population mean of $I(x^{\prime}_{i})$, and $\sigma$
is a given confidence level.

The proof for Theorem 3.1 is presented in [62], which adapts the more general
proof in [63] for the binary 0–1 variables $I_{i}$ and also rearranges terms
to align with the form of the original Hoeffding’s inequality. We further
prove two corollaries in Appendix 0.B to better explain our later experimental
results: One compares the tightness of the two bounds derived from the
original and adaptive Hoeffding’s inequality; the other concerns their
monotonicity with respect to $n$ and $\sigma$.

In ProTIP, we directly apply the result of Eq. (6) by sequentially increasing
the number of perturbations from $J=1$ to $J=j_{max}$ (the maximum sampling
budget for generating perturbations). Given the verification target
$(b_{l},\sigma)$ (cf. Def. 3), whenever the empirically estimated lower bound
on $R_{M}(x)$ yielded by Eq. (6) (after rearranging the inequality) is greater
or equal to the specified requirement $b_{l}$ (i.e.,
$\hat{\mu}_{I}^{(J)}-\varepsilon(\sigma,J)\geq b_{l}$), we stop generating new
perturbations and assert “pass” for the verification. Otherwise, when
$J=j_{max}$, ProTIP asserts “fail” with an evaluated robustness.

##  4 Experiments

To evaluate the effectiveness and efficiency of the proposed framework,
ProTIP, we conduct experiments with the following setup.

We choose two versions of the widely acclaimed and open-source444We conduct
experiments locally on our own servers for efficiency, as ProTIP generates a
large number of queries to the model under verification. Therefore, we exclude
experiments on non-open source models, such as DALLE-3 and Midjourney. Stable
Diffusion (SD) model [41], SD V1.5 and SD V1.4. While different versions of SD
models have the same structure, the higher version is further trained based on
the previous version. We use the MS-COCO dataset [28], a comprehensive
collection designed for common computer vision tasks including captioning. The
dataset comprises 328,000 images with captions, from which we randomly select
image captions as prompts for the T2I DM and then apply the stochastic
perturbation method discussed in Sec. 3.2 on these prompts to generate the
perturbed inputs.

All the experiments are conducted on NVIDIA GeForce RTX 3090 GPUs, Python
3.11, PyTorch 2.1.1. We use the off-the-shelf R package [39] to design group
sequential analysis parameters, cf. Appendix 0.C for details. All our models,
data, source code and experimental results are publicly accessible at
https://github.com/wellzline/ProTIP/.

###  4.1 Effectiveness of ProTIP

In order to evaluate the effectiveness of ProTIP, i.e., how accurate our
ProTIP is, we need to know the ground truth probabilistic robustness
$R_{M}(x)$ for the given prompt $x$. However, as articulated in Sec. 3.1, the
ground truth of $R_{M}(x)$ can never be known (as it would require
exhaustively testing all possible perturbations). Consequently, we can only
approximate the ground truth by using a significantly larger number of samples
than what would normally be used in ProTIP to demonstrate accuracy, which is
arguably a common practice in statistical inference, e.g., [53, 20, 8].
Specifically, we conduct the following two steps: i) estimate the ground truth
of $I(x^{\prime})$ for a given perturbation $x^{\prime}$ by generating a large
number of images; ii) then estimate the ground truth $R_{M}(x)$ using a large
number of $x^{\prime}$s with their ground truth $I(x^{\prime})$. For step 1,
we stop generating the images for the $x^{\prime}$ until the distribution of
its CLIP scores “converge” (i.e., the shape of the CLIP score distribution
shows no significant changes when new images are generated). We then check
whether the CLIP scores, as well as the CLIP score distribution of the
original prompt $x$, follow a normal distribution. If they do, we execute a
t-test, otherwise a u-test. In step 2, we generate a fixed number of $1,000$
perturbations, which is much larger than the adaptive number of perturbations
(around $50\\!\sim\\!350$) used in ProTIP. Then the original Hoffeding’s
inequality, which yields a tighter error bound (cf. Corollary 1) than the
adaptive version in ProTIP, is applied to approximate the ground truth
$R_{M}(x)$.

As shown in Fig. 4, for each given input $x$, we conduct 3 sets of comparative
experiments for different perturbation rates and different version of SD
models. In each experiment, we run ProTIP for 3 different confidence levels
$1-\sigma$ (coloured solid lines), and set the verification target to 0.8
(solid black line), while the dashed red line represents the (approximated)
ground truth. ProTIP would stop whenever the verification target is met (i.e.,
the intersection point of the coloured and black lines). But for illustration,
we also plot the assessment result after the intersection point, represented
by dotted lines.

For a given prompt, Fig. 4(a) illustrates its robustness estimation with 10%
perturbation rate, where the estimates based on different confidence levels
all converge to the ground truth. The noticeable gap between the approximated
ground truth and our results is expected since ProTIP focuses on the lower
bound estimation (cf. Eq. (4)), being conservative. Such gap can be reduced
when more perturbations are generated for a more accurate estimation (cf.
Corollary 2). For the case of Fig. 4(a), we observe that ProTIP only requires
$70\\!\sim\\!100$ perturbations to achieve the given verification target
$(0.8,\sigma)$.

Fig. 4(b) presents the results with an increased perturbation rate of 20%. As
expected, the ground truth robustness is lower, as well as the ProTIP
results—it asserts “unverified” reflects that the ground truth is lower than
the verification target. In Fig. 4(c), ProTIP is applied to a older version SD
V1.4 model. Compared to (a), it is evident that, under the same 10%
perturbation rate, the estimated robustness of SD V1.4 are lower than those of
SD V1.5, implying its weaker ability to withstand adversarial perturbations,
which conforms to the observation in [61, 26, 49]. While Fig. 4 only shows the
results of a single prompt, results for more prompts are presented in Appendix
0.D.

Figure 4: ProTIP results for a given prompt, with different confidence levels
$1-\sigma$.

###  4.2 Efficiency of ProTIP

The efficiency of ProTIP manifests in two aspects, firstly in identifying AEs,
as illustrated in the “inner loop” of Fig. 3. The aforementioned challenge of
identifying a perturbed input as an AE involves comparing distributional
differences which is computationally insensitive for T2I models. Thus, we
employ sequential analysis with efficacy and futility early stopping rules in
the statistical testing for indicating AEs. Following the parameter settings
of Appendix 0.C, WLOG, we divide the sequential analysis into 5 stages to
conduct interim analysis.

Fig. 5(a) illustrates the number of perturbations indicated as Non-AEs/AEs at
each interim analysis stage, for a given prompt processed by SD V1.5 with
perturbation rates of 10% and 20%, as well as by SD V1.4 with a 10%
perturbation rate. For V1.5 with 10% rate, we observe that, out of the total
400 perturbations, 347 are identified as Non-AEs. Thanks to the futility
stopping rule, 246 of these Non-AEs can be identified at stage 1, saving 4
times the cost of generating images compared to what would be required without
early stopping rules (i.e., test at the final stage 5). Considering other
stages as well, in total it reduces the computational cost by detecting 97% of
Non-AEs before reaching the final stage 5. Similarly, 68% of AEs are detected
early due to the efficacy stopping rule.

With a 20% perturbation rate and an older version, the total number of AEs
increased and Non-AEs decreased, yet overall, the effect of two early stopping
rules in the statistical testing for indicating Non-AEs/AEs remains evident.
While Fig. 5(a) demonstrates the results for only one prompt, statistics for
36 randomly selected prompts are presented in Table 2.

The other efficient aspect of ProTIP is demonstrated by introducing the
adaptive concentration inequalities to dynamically determine the “just-right”
number of perturbations whenever the verification target is met, cf. the
“outer loop” of Fig. 3. To compare with the original Hoeffding’s inequality
which requires a predetermined, process-independent sample size, we have to do
“what if” calculations by assuming the sample size used by it. Fig. 5(b)
presents the ProTIP results and the (original) Hoeffding’s inequality with
sample sizes of 50, 200, and 400 (vertical dashed lines in green). With sample
size 50, Hoeffding’s inequality asserts an incorrect verification result
“fail”, while ProTIP correctly verifies it, although with more samples of 225.
This is non-surprising as the error bound is bigger when with limited samples
(cf. Corollary 2). Thus no practitioners would apply Hoeffding’s inequality
with such small sample size, rather allocate a much larger sample size for a
small estimation error. The case with sample size 400 is replicating this
scenario, in which we observe that both methods make the correct verification
decision while ProTIP saves 175 perturbations. For the case of sample size
200, indeed both methods yield similar results (and Hoeffding’s inequality is
slightly better, cf. Corollary 1) using a similar number of samples. However,
in practice, we would never know the “just right” sample size a priori when
applying Hoeffding’s inequality, highlighting ProTIP’s superiority in terms of
efficiency.

(a) Number of perturbations (out of 400 perturbations) identified as Non-
AEs/AEs at each interim stage (S) in the sequential hypothesis testing.

(b) ProTIP with the (adaptive) sample size of 225 vs. Hoeffding’s inequality
with fixed sample size 50, 200, and 400.

Figure 5: Illustration of ProTIP efficiency for a given prompt. Table 2: The
mean percentage of perturbations indicated as Non-AE/AE by the two early
stopping rules before the final stage 5, over 36 randomly selected prompts.

Model | Pert. rate | Stage 1 | Stage 2 | Stage 3 | Stage 4
---|---|---|---|---|---
Efficacy | Futility | Efficacy | Futility | Efficacy | Futility | Efficacy | Futility
SD V1.5 | 10% | 12% | 31% | 6% | 21% | 5% | 11% | 4% | 5%
20% | 21% | 26% | 10% | 14% | 6% | 8.5% | 5% | 5%
SD V1.4 | 10% | 13% | 26% | 7% | 22% | 5% | 12% | 4% | 6%

###  4.3 Application of ProTIP for Ranking Defence Methods

As a robustness assessment tool, a natural use case of our ProTIP is to rank
common defence methods for T2I DMs. For character-level perturbations that are
perceivable and semantic, scrutinising the input is a direct and universally
applicable passive defence. Thus, we study three commonly used misspelling
checking tools: the Python Autocorrect 0.3.0 package [12], the Pyspellchecker
package [36], and Gramformer [37] for grammar error correction (GEC) on the
perturbed inputs before inputting them into the T2I DM.

Figure 6: Box plot of the probabilistic robustness with/without defence
methods, over 36 randomly selected prompts.

In Fig. 7, for an example prompt, all three defence tools may improve the
model’s resistance to stochastic perturbations in input text, compared to the
the case without using any defenders (the red curve). Notably, Autocorrect
demonstrates superior overall performance compared to the other two tools.
Such observations are further confirmed by the box plots in Fig. 6 over a set
of 36 randomly selected prompts. Our approach allows for the performance
ranking of these misspelling checking tools, as shown in the Fig. 6.

Figure 7: Probabilistic robustness estimated by ProTIP with/without defence
methods.

##  5 Conclusion

In this work, for the first time, we formalise the definition of probabilistic
robustness of T2I DMs and then establish an efficient framework, ProTIP, for
evaluating it. As a black-box verification method, ProTIP is based on
principled statistical inference approaches. It incorporates sequential
analysis with efficacy and futility early stopping rules in the statistical
hypothesis testing for identifying AEs, along with adaptive concentration
inequalities to dynamically adjust the number of stochastic perturbations
needed to make the verification decision. Empirical experiments have
substantiated the effectiveness and efficiency of ProTIP. Finally, we
demonstrate a use case of ProTIP to rank various commonly used defence
methods, highlighting its versatility and applicability.

## References

  * [1] Midjourney. https://www.midjourney.com/
  * [2] Aminifar, A.: Universal adversarial perturbations in epileptic seizure detection. In: 2020 International Joint Conference on Neural Networks (IJCNN). pp. 1–6. IEEE (2020)
  * [3] Betker, J., Goh, G., Jing, L., TimBrooks, Wang, J., Li, L., LongOuyang, JuntangZhuang, JoyceLee, YufeiGuo, WesamManassra, PrafullaDhariwal, CaseyChu, YunxinJiao, Ramesh, A.: Improving image generation with better captions
  * [4] Boucheron, S., Lugosi, G., Massart, P.: Concentration inequalities - a nonasymptotic theory of independence. In: Concentration Inequalities (2013)
  * [5] Carlini, N., Farid, H.: Evading deepfake-image detectors with white-and black-box attacks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 658–659 (2020)
  * [6] Chakraborty, A., Alam, M., Dey, V., Chattopadhyay, A., Mukhopadhyay, D.: A survey on adversarial attacks and defences. CAAI Transactions on Intelligence Technology 6(1), 25–45 (2021)
  * [7] Cohen, J., Rosenfeld, E., Kolter, Z.: Certified adversarial robustness via randomized smoothing. In: Proc. of the 36th Int. Conf. on Machine Learning. PMLR, vol. 97, pp. 1310–1320. PMLR (2019)
  * [8] Dong, Y., Huang, W., Bharti, V., Cox, V., Banks, A., Wang, S., Zhao, X., Schewe, S., Huang, X.: Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurance. ACM Trans. Embed. Comput. Syst. 22(3) (2023)
  * [9] Du, C., Li, Y., Qiu, Z., Xu, C.: Stable diffusion is unstable. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. abs/2306.02583 (2023)
  * [10] Fort, S.: Pixels still beat text: attacking the openai clip model with text patches and adversarial pixel perturbations. Stanislav Fort [Internet] 5 (2021)
  * [11] Gao, H., Zhang, H., Dong, Y., Deng, Z.: Evaluating the robustness of text-to-image diffusion models against real-world attacks. arXiv preprint arXiv:2306.13103 (2023)
  * [12] Gao, J., Lanchantin, J., Soffa, M.L., Qi, Y.: Black-box generation of adversarial text sequences to evade deep learning classifiers. In: 2018 IEEE Security and Privacy Workshops (SPW). pp. 50–56. IEEE (2018)
  * [13] Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., Vechev, M.: Ai2: Safety and robustness certification of neural networks with abstract interpretation. In: 2018 IEEE symposium on security and privacy (SP). pp. 3–18. IEEE (2018)
  * [14] Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: In Proc. of 3rd Int. Conf. on Learning Representations (ICLR) (2015)
  * [15] Gordon Lan, K., DeMets, D.L.: Discrete sequential boundaries for clinical trials. Biometrika 70(3), 659–663 (1983)
  * [16] Gretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B., Smola, A.: A kernel two-sample test. The Journal of Machine Learning Research 13(1), 723–773 (2012)
  * [17] Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., Choi, Y.: CLIPScore: A reference-free evaluation metric for image captioning. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 7514–7528. Association for Computational Linguistics (Nov 2021)
  * [18] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)
  * [19] Hoeffding, W.: Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding pp. 409–426 (1994)
  * [20] Huang, W., Zhao, X., Jin, G., Huang, X.: Safari: Versatile and efficient evaluations for robustness of interpretability. In: Proc. of the IEEE/CVF Int. Conf. on Computer Vision (ICCV). pp. 1988–1998 (2023)
  * [21] Huang, X., Kroening, D., Ruan, W., et al: A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review 37, 100270 (2020)
  * [22] Jennison, C., Turnbull, B.W.: Group sequential methods with applications to clinical trials. CRC Press (1999)
  * [23] Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: An efficient smt solver for verifying deep neural networks. In: Computer Aided Verification: 29th International Conference, CAV 2017, Proceedings, Part I 30\. pp. 97–117. Springer (2017)
  * [24] Kullback, S., Leibler, R.A.: On Information and Sufficiency. The Annals of Mathematical Statistics 22(1), 79 – 86 (1951)
  * [25] Lakens, D.: Improving Your Statistical Inferences. https://lakens.github.io/statistical_inferences/ (2022)
  * [26] Lee, T., Yasunaga, M., Meng, C., Mai, Y., Park, J.S., Gupta, A., Zhang, Y., Narayanan, D., Teufel, H.B., Bellagente, M., Kang, M., Park, T., Leskovec, J., Zhu, J.Y., Fei-Fei, L., Wu, J., Ermon, S., Liang, P.: Holistic evaluation of text-to-image models. In: Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023)
  * [27] Li, L., Ren, K., Shao, Y., Wang, P., Qiu, X.: Perturbscore: Connecting discrete and continuous perturbations in NLP. In: Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 6638–6648 (2023)
  * [28] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755. Springer (2014)
  * [29] Liu, H., Wu, Y., Zhai, S., Yuan, B., Zhang, N.: Riatig: Reliable and imperceptible adversarial text-to-image generation with natural prompts. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 20585–20594 (2023)
  * [30] Lyu, L.: A pathway towards responsible ai generated content. In: Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23. pp. 7033–7038 (2023)
  * [31] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. stat 1050, 9 (2017)
  * [32] Maus, N., Chao, P., Wong, E., Gardner, J.R.: Black box adversarial prompting for foundation models. In: The Second Workshop on New Frontiers in Adversarial Machine Learning (2023)
  * [33] Moosavi-Dezfooli, S.M., Fawzi, A., Fawzi, O., Frossard, P.: Universal adversarial perturbations. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1765–1773 (2017)
  * [34] Morris, J.X., Lifland, E., Yoo, J.Y., Grigsby, J., Jin, D., Qi, Y.: Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. In: Conference on Empirical Methods in Natural Language Processing (2020)
  * [35] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In: International Conference on Machine Learning (2021)
  * [36] Norvig, P.: pyspellchecker: A spell checker for python. GitHub repository (2024)
  * [37] Prithivida: Gramformer: A library for a family of algorithms to detect, highlight and correct grammar errors. GitHub repository (2021)
  * [38] Proschan, M.A., Lan, K.G., Wittes, J.T.: Statistical monitoring of clinical trials: a unified approach. Springer Science & Business Media (2006)
  * [39] R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2021)
  * [40] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [41] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [42] Ruan, W., Huang, X., Kwiatkowska, M.: Reachability analysis of deep neural networks with provable guarantees. In: Proc. of the 27th Int. Joint Conference on Artificial Intelligence (IJCAI’18). pp. 2651–2659 (2018)
  * [43] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35, 36479–36494 (2022)
  * [44] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. In: International Conference on Learning Representations (2021)
  * [45] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. In: In Proc. of 2nd Int. Conf. on Learning Representations (ICLR) (2014)
  * [46] Takagi, Y., Nishimoto, S.: High-resolution image reconstruction with latent diffusion models from human brain activity. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14453–14463 (2023)
  * [47] TIT, K., Furon, T., Rousset, M.: Gradient-informed neural network statistical robustness estimation. In: Ruiz, F., Dy, J., van de Meent, J.W. (eds.) Proc. of The 26th International Conference on Artificial Intelligence and Statistics. PMLR, vol. 206, pp. 323–334. PMLR (2023)
  * [48] Wang, B., Webb, S., Rainforth, T.: Statistically robust neural network classification. In: Uncertainty in Artificial Intelligence. pp. 1735–1745. PMLR (2021)
  * [49] Wang, H., Shen, Q., Tong, Y., Zhang, Y., Kawaguchi, K.: The stronger the diffusion model, the easier the backdoor: Data poisoning to induce copyright breaches without adjusting finetuning pipeline. In: NeurIPS 2023 Workshop on Backdoors in Deep Learning - The Good, the Bad, and the Ugly (2023)
  * [50] Wang, Y., Ma, X., Bailey, J., Yi, J., Zhou, B., Gu, Q.: On the convergence and robustness of adversarial training. In: International Conference on Machine Learning. pp. 6586–6595. PMLR (2019)
  * [51] Wassmer, G., Brannath, W.: Group sequential and confirmatory adaptive designs in clinical trials, vol. 301. Springer (2016)
  * [52] Wassmer, G., Pahlke, F.: rpact: Confirmatory adaptive clinical trial design and analysis. (2022)
  * [53] Webb, S., Rainforth, T., Teh, Y.W., Kumar, M.P.: A statistical approach to assessing neural network robustness. International Conferences on Learning Representations(ICLR). (2019)
  * [54] Weng, L., Chen, P.Y., Nguyen, L., Squillante, M., Boopathy, A., Oseledets, I., Daniel, L.: Proven: Verifying robustness of neural networks with a probabilistic approach. In: International Conference on Machine Learning. pp. 6727–6736. PMLR (2019)
  * [55] Weng, T.W., Zhang, H., Chen, P.Y., Yi, J., Su, D., Gao, Y., Hsieh, C.J., Daniel, L.: Evaluating the robustness of neural networks: An extreme value theory approach. In: Int. Conf. on Learning Representations. ICLR (2018)
  * [56] Wu, J.Z., Ge, Y., Wang, X., Lei, S.W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., Shou, M.Z.: Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7623–7633 (2023)
  * [57] Xiang, W., Tran, H.D., Johnson, T.T.: Output reachable set estimation and verification for multilayer neural networks. IEEE Transactions on Neural Networks and Learning Systems 29(11), 5777–5783 (2018)
  * [58] Yu, J., Xu, Y., Koh, J.Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B.K., Hutchinson, B.C., Han, W., Parekh, Z., Li, X., Zhang, H., Baldridge, J., Wu, Y.: Scaling autoregressive models for content-rich text-to-image generation. Trans. Mach. Learn. Res. 2022 (2022)
  * [59] Yuan, X., He, P., Zhu, Q., Li, X.: Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems 30(9), 2805–2824 (2019)
  * [60] Zhang, C., Wang, L., Liu, A.: Revealing vulnerabilities in stable diffusion via targeted attacks. arXiv preprint arXiv:2401.08725 (2024)
  * [61] Zhang, J., Xu, Z., Cui, S., Meng, C., Wu, W., Lyu, M.R.: On the robustness of latent diffusion models. arXiv preprint arXiv:2306.08257 (2023)
  * [62] Zhang, T., Ruan, W., Fieldsend, J.E.: Proa: A probabilistic robustness assessment against functional perturbations. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 154–170. Springer (2022)
  * [63] Zhao, S., Zhou, E., Sabharwal, A., Ermon, S.: Adaptive concentration inequalities for sequential decision problems. Advances in Neural Information Processing Systems 29 (2016)
  * [64] Zhao, X., Huang, W., Schewe, S., Dong, Y., Huang, X.: Detecting Operational Adversarial Examples for Reliable Deep Learning. In: 51st Annual IEEE/IFIP Int. Conf. on Dependable Systems and Networks. DSN’21, IEEE/IFIP, Taipei Taiwan (2021)
  * [65] Zhuang, H., Zhang, Y., Liu, S.: A pilot study of query-free adversarial attack against stable diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2384–2391 (2023)

##  Appendix 0.A Text-to-Image Diffusion Model

Diffusion models are probabilistic generative models that are trained to learn
a data distribution by the gradual denoising of a variable sampled from a
Gaussian distribution. One specific latent diffusion architecture is shown in
Fig. 8. Specifically, for a pre-trained T2I DM $\hat{x}_{\theta}$, given an
initial noise map $\gamma\sim\mathcal{N}(0,I)$ and a conditioning vector
$c=\Gamma(P)$ generated using a text encoder $\Gamma$ and a text input $P$,
generates an image $x_{\text{gen}}=\hat{x}_{\theta}(\gamma,c)$. For text-to-
image generation, the model is conditioned on the text prompt $P$ and guided
toward an image faithful to that prompt. The training objective of a diffusion
model $\hat{x}_{\theta}$ can be written as:

$E_{x,c,\gamma,t}[w_{t}||\hat{x}_{\theta}](\alpha_{t}x+\sigma_{t}\boldsymbol{\gamma},c)-x||^{2}_{2}]$
(7)

Where $x$ is the ground-truth image, $c$ is a conditioning vector (e.g.,
obtained from a text prompt), and $\alpha_{t}$, $\sigma_{t}$, $w_{t}$ are
terms that control the noise schedule and sample quality, and are functions of
the diffusion process time $t\sim\mathcal{U}([0,1])$. Consequently, the DM is
trained to denoise $z_{t}:=\alpha_{t}x+\sigma_{t}\gamma$ yielding x with the
squared error loss. At inference time, the diffusion model is sampled by
iteratively denoising $z_{t_{1}}\sim\mathcal{N}(0,I)$ using either the
deterministic DDIM [44] or the stochastic ancestral sampler [18]. Intermediate
points $z_{t_{1}},\ldots,z_{t_{T}}$, where $1=t_{1}>\ldots>t_{T}=0$, are
generated, with decreasing noise levels. These points,
$\hat{x}_{0}^{(t)}:=\hat{x}_{\theta}(z_{t},c)$, are functions of the
$x$-predictions.

Figure 8: Overview of the latent diffusion architecture. Image from the
paper[41].

##  Appendix 0.B Hoeffding’s Inequality

Reusing the notations in the main paper, let $I_{1},\dots,I_{n}$ be i.i.d.
samples drawn from a population, and $R$ denotes the population mean to be
estimated. Let $\hat{\mu}_{I}^{(n)}=\frac{1}{n}\sum_{i=1}^{n}I_{i}$, i.e., the
sample mean, we have the following Hoeffding’s Inequality [19].

######  Theorem 0.B.1 (Hoeffding’s Inequality)

$\displaystyle
Pr\left(\left|\hat{\mu}_{I}^{(n)}-R\right|\leq\varepsilon\right)$
$\displaystyle>1-\sigma$ (8)

where $\sigma=2e^{-2n\varepsilon^{2}}$, equivalently,
$\varepsilon=\sqrt{\frac{\log(2/\sigma)}{2n}}$.

From the two Theorems 0.B.1 and 3.1 regarding the original and adaptive
Hoeffding’s Inequality respectively, we may derive the following two
corollaries.

######  Corollary 1 (Tightness of the two bounds)

Give a confidence level $1-\sigma$ and the same number of samples $n$, the
estimation error (between the sample mean and population mean) $\varepsilon$
derived from the original Hoeffding’s Inequality is always smaller than the
adaptive Hoeffding’s Inequality.

###### Proof

As Theorems 0.B.1 and 3.1, the estimation error $\varepsilon$ from the
original and adaptive Hoeffding’s Inequality are
$\sqrt{\frac{\log(2/\sigma)}{2n}}$ and
$\sqrt{\frac{0.6\cdot\log(\log_{1.1}n+1)+1.8^{-1}\cdot\log\left(24/{\sigma}\right)}{n}}$,
respectively. We may prove the former is smaller than the latter either
analytically or empirically. The inequalities of Corollary 1 can be written:

$\displaystyle\sqrt{\frac{\log(2/\sigma)}{2n}}<\sqrt{\frac{0.6\cdot\log(\log_{1.1}n+1)+1.8^{-1}\cdot\log\left(24/{\sigma}\right)}{n}}$
(9)

Where $n\geq 1\text{ and }\sigma\in(0,1)$.

$\displaystyle\frac{\log\left(2/\sigma\right)}{2n}$
$\displaystyle<\frac{0.6\cdot\log\left(\log_{1.1}n+1\right)+1.8^{-1}\cdot\log\left(24/{\sigma}\right)}{n}$
(10a) $\displaystyle\log(2/\sigma)$
$\displaystyle<1.2\cdot\log\left(\log_{1.1}n+1\right)+\frac{10}{9}\cdot\log\left(24/\sigma\right)$
(10b) $\displaystyle\log\left(2/\sigma\right)$
$\displaystyle<\log\left[\left(\log_{1.1}n+1\right)^{1.2}\cdot\left(24/\sigma\right)^{\frac{10}{9}}\right]$
(10c)

Because $n\geq 1$, hence, $\log_{1.1}n>0$, $\log_{1.1}n+1>1$, and
$\left(\log_{1.1}n+1\right)^{1.2}>1$, the right side of (10c) is.

$\displaystyle\log\left[\left(\log_{1.1}n+1\right)^{1.2}\cdot\left(24/\sigma\right)^{\frac{10}{9}}\right]>\log\left(24/\sigma\right)^{\frac{10}{9}}>\log\left(24/\sigma\right)$
(11)

Hence, (10c) will be:

$\displaystyle\log\left(2/\sigma\right)<\log\left(24/\sigma\right),\sigma\in(0,1)$
(12)

Therefore the equation holds, and the proof is complete. ∎

######  Corollary 2 (Monotonicity to $n$ and $\sigma$)

For both the original and adaptive Hoeffding’s Inequalities, the estimation
errors (between the sample mean and population mean) $\varepsilon$ are
monotonically decreasing to sample size $n$ and $\sigma$.

###### Proof

Proof sketch: By taking the (partial) derivatives of the two analytical
expressions of $\varepsilon$ r.w.t. $n$ and $\sigma$ respectively, we may
establish negative results. ∎

##  Appendix 0.C Sequential Analysis Parameter Design

###  0.C.1 Design parameters and output of group sequential design

  * •

Type of design: Pocock type alpha spending

  * •

Information rates: 0.200, 0.400, 0.600, 0.800, 1.000

  * •

Significance level: 0.0500

  * •

Type II error rate: 0.3000

  * •

Type of beta spending: Pocock type beta spending

Derived from User Defined Parameters

  * •

Maximum number of stages: 5

  * •

Stages: 1, 2, 3, 4, 5

Default Parameters

  * •

Two-sided power: FALSE

  * •

Binding futility: FALSE

  * •

Test: one-sided

  * •

Tolerance: 1e-08

Output

  * •

Power: 0.1655, 0.3637, 0.5316, 0.6452, 0.7000

  * •

Futility bounds (non-binding): -0.145, 0.511, 1.027, 1.497

  * •

Cumulative alpha spending: 0.01477, 0.02616, 0.03543, 0.04324, 0.05000

  * •

Cumulative beta spending: 0.08862, 0.15694, 0.21255, 0.25945, 0.30000

  * •

Critical values: 2.176, 2.144, 2.113, 2.090, 2.071

  * •

Stage levels (one-sided): 0.01477, 0.01603, 0.01729, 0.01833, 0.01918

Group Sequential Design Characteristics

  * •

Number of subjects fixed: 4.7057

  * •

Shift: 7.2491

  * •

Inflation factor: 1.5405

  * •

Informations: 1.450, 2.900, 4.349, 5.799, 7.249

  * •

Power: 0.1655, 0.3637, 0.5316, 0.6452, 0.7000

  * •

Rejection probabilities under H1 : 0.16549, 0.19825, 0.16786, 0.11361, 0.05478

  * •

Futility probabilities under H1 : 0.08862, 0.06832, 0.05561, 0.04690

  * •

Ratio expected vs fixed sample size under H1 : 0.7938

  * •

Ratio expected vs fixed sample size under a value between H0 and H1 : 0.7776

  * •

Ratio expected vs fixed sample size under H0 : 0.5869

###  0.C.2 Sample Size Calculation for a Continuous Endpoint

Sequential analysis with a maximum of 5 looks (group sequential design),
overall significance level 5% (one-sided). The sample size was calculated for
a two-sample t-test, $H_{0}:\mu(1)-\mu(2)=0$, $H_{1}:\text{effect}=0.5$,
standard deviation = 1, power 70%.

| Stage 1 | Stage 2 | Stage 3 | Stage 4 | Stage 5
---|---|---|---|---|---
Information rate | 20% | 40% | 60% | 80% | 100%
Efficacy boundary (z-value scale) | 2.176 | 2.144 | 2.113 | 2.090 | 2.071
Futility boundary (z-value scale) | -0.145 | 0.511 | 1.027 | 1.497 | -
Overall power | 0.1655 | 0.3637 | 0.5316 | 0.6452 | 0.7000
Expected number of subjects | 60.9
Number of subjects | 23.6 | 47.2 | 70.9 | 94.5 | 118.1
Cumulative alpha spent | 0.0148 | 0.0262 | 0.0354 | 0.0432 | 0.0500
Cumulative beta spent | 0.0886 | 0.1569 | 0.2126 | 0.2595 | 0.3000
One-sided local significance level | 0.0148 | 0.0160 | 0.0173 | 0.0183 | 0.0192
Efficacy boundary (t) | 0.959 | 0.644 | 0.512 | 0.436 | 0.385
Futility boundary (t) | -0.060 | 0.150 | 0.246 | 0.311 | -
Overall exit probability (under H0) | 0.4570 | 0.2977 | 0.1526 | 0.0688 | -
Overall exit probability (under H1) | 0.2541 | 0.2666 | 0.2235 | 0.1605 | -
Exit probability for efficacy (under H0) | 0.0148 | 0.0113 | 0.0087 | 0.0062 | -
Exit probability for efficacy (under H1) | 0.1655 | 0.1982 | 0.1679 | 0.1136 | -
Exit probability for futility (under H0) | 0.4423 | 0.2864 | 0.1439 | 0.0626 | -
Exit probability for futility (under H1) | 0.0886 | 0.0683 | 0.0556 | 0.0469 | -
Table 3: Group Sequential Design Characteristics

### Legend

  * •

(t): treatment effect scale

###  0.C.3 Design Plan Parameters and Output for Means

### Design Parameters

  * •

Information rates: 0.200, 0.400, 0.600, 0.800, 1.000

  * •

Critical values: 2.176, 2.144, 2.113, 2.090, 2.071

  * •

Futility bounds (non-binding): -0.145, 0.511, 1.027, 1.497

  * •

Cumulative alpha spending: 0.01477, 0.02616, 0.03543, 0.04324, 0.05000

  * •

Local one-sided significance levels: 0.01477, 0.01603, 0.01729, 0.01833,
0.01918

  * •

Significance level: 0.0500

  * •

Type II error rate: 0.3000

  * •

Test: one-sided

### User Defined Parameters

  * •

Alternatives: 0.5

### Default Parameters

  * •

Mean ratio: FALSE

  * •

Theta H0: 0

  * •

Normal approximation: FALSE

  * •

Standard deviation: 1

  * •

Treatment groups: 2

  * •

Planned allocation ratio: 1

### Sample Size and Output

  * •

Reject per stage [1]: 0.16549

  * •

Reject per stage [2]: 0.19825

  * •

Reject per stage [3]: 0.16786

  * •

Reject per stage [4]: 0.11361

  * •

Reject per stage [5]: 0.05478

  * •

Overall futility stop: 0.2595

  * •

Futility stop per stage [1]: 0.08862

  * •

Futility stop per stage [2]: 0.06832

  * •

Futility stop per stage [3]: 0.05561

  * •

Futility stop per stage [4]: 0.04690

  * •

Early stop: 0.9047

  * •

Maximum number of subjects: 118.1

  * •

Maximum number of subjects (1): 59.1

  * •

Maximum number of subjects (2): 59.1

  * •

Number of subjects [1]: 23.6

  * •

Number of subjects [2]: 47.2

  * •

Number of subjects [3]: 70.9

  * •

Number of subjects [4]: 94.5

  * •

Number of subjects [5]: 118.1

  * •

Expected number of subjects under H0: 45

  * •

Expected number of subjects under H0/H1: 59.6

  * •

Expected number of subjects under H1: 60.9

  * •

Critical values (treatment effect scale) [1]: 0.959

  * •

Critical values (treatment effect scale) [2]: 0.644

  * •

Critical values (treatment effect scale) [3]: 0.512

  * •

Critical values (treatment effect scale) [4]: 0.436

  * •

Critical values (treatment effect scale) [5]: 0.385

  * •

Futility bounds (treatment effect scale) [1]: -0.0605

  * •

Futility bounds (treatment effect scale) [2]: 0.1496

  * •

Futility bounds (treatment effect scale) [3]: 0.2457

  * •

Futility bounds (treatment effect scale) [4]: 0.3108

  * •

Futility bounds (one-sided p-value scale) [1]: 0.55773

  * •

Futility bounds (one-sided p-value scale) [2]: 0.30485

  * •

Futility bounds (one-sided p-value scale) [3]: 0.15231

  * •

Futility bounds (one-sided p-value scale) [4]: 0.06717

### Legend

  * •

(i): values of treatment arm i

  * •

(k): values at stage k

##  Appendix 0.D Experimental results

Figure 9: More results on ProTIP effectiveness (Part 1). Figure 10: More
results on ProTIP effectiveness (Part 2). Figure 11: Probabilistic robustness
with/without defence methods (Part 1). Figure 12: Probabilistic robustness
with/without defence methods (Part 2).