# DiffusionPID: Interpreting Diffusion via Partial Information Decomposition

Shaurya Dewan*
Rushikesh Zawar*
Prakanshul Saxena*
Yingshan Chang
Andrew Luo
Yonatan Bisk
Carnegie Mellon Univeristy
(srdewan, rzawar, prakanss, yingshac, afluo<EMAIL_ADDRESS>

###### Abstract

Text-to-image diffusion models have made significant progress in generating
naturalistic images from textual inputs, and demonstrate the capacity to learn
and represent complex visual-semantic relationships. While these diffusion
models have achieved remarkable success, the underlying mechanisms driving
their performance are not yet fully accounted for, with many unanswered
questions surrounding what they learn, how they represent visual-semantic
relationships, and why they sometimes fail to generalize. Our work presents
Diffusion Partial Information Decomposition (DiffusionPID), a novel technique
that applies information-theoretic principles to decompose the input text
prompt into its elementary components, enabling a detailed examination of how
individual tokens and their interactions shape the generated image. We
introduce a formal approach to analyze the uniqueness, redundancy, and synergy
terms by applying PID to the denoising model at both the image and pixel
level. This approach enables us to characterize how individual tokens and
their interactions affect the model output. We first present a fine-grained
analysis of characteristics utilized by the model to uniquely localize
specific concepts, we then apply our approach in bias analysis and show it can
recover gender and ethnicity biases. Finally, we use our method to visually
characterize word ambiguity and similarity from the model’s perspective and
illustrate the efficacy of our method for prompt intervention. Our results
show that PID is a potent tool for evaluating and diagnosing text-to-image
diffusion models. (We share the code in the supplemental and will release it
publicly.)

##  1 Introduction

Figure 1: Concept Figure. Left: Our "baseball" uniqueness map specifically
highlights the seam region of the tennis ball as it is visually very similar
to that of a baseball, Center: We see a high synergy for "bat" with "baseball"
and "overhead" respectively which shows that it uses these contextual cues to
generate the images in the right settings, Right: Our redundancy map between
"queen" and "crown" correctly focuses on the crown and facial region.

The role of individual inputs and the pretraining data in diffusion-based
image generation remains opaque, with little insight into how they translate
often ambiguous text-based prompts into images. If asked to imagine a lush
forest at the edge of a sandy beach, you can fill in missing details,
picturing warm golden sand, splashing waves, and sunlight filtering through
dense green foliage. Our prior experience allows us to easily visualize
complex semantic associations and imagine novel scenes.

In the previous example, you are able to both generate and justify your
choices for how to complete the scene. Recent advances in diffusion-based
generative models have achieved remarkable success in high-quality text-
conditioned image generation, mimicking this human ability to a certain
extent. However, the lack of transparency in these models raises important
questions about their interpretability and controllability, making it
challenging to identify biases, errors, or inconsistencies in the generated
images. Text-to-image diffusion models can be brittle and prone to failure
when faced with out-of-distribution inputs, ambiguous text descriptions, or
nuanced contextual dependencies. This results in unrealistic, inconsistent, or
even misleading generations, leading to unnatural (and linguistically non-
sensical) prompt engineering [1, 2] to achieve desired results by identifying
and exploiting spurious correlations captured by the model [3]. The lack of
interpretability also hinders the ability to correct or refine the models when
they produce undesirable outputs, limiting the potential applications of these
generative models in real-world scenarios. Frequently, in multi-concept
sentence-based conditioning of generative image models, concepts that are
redundant to humans may be critical to the model. For instance, in the phrase
"he swung a baseball bat", the word "baseball" may be redundant to humans, but
it could be a crucial factor in the model’s decision to generate a baseball
bat instead of the animal as can be seen in Fig 1. This work posits that a
crucial step towards more interpretable and robust models is to develop tools
that can help explain higher-order contributions and interactions of
individual words to the final image output. This would enable users to
understand how the model is using the input text to generate the image,
identify potential biases or errors, and refine the model’s performance.

To that end, we introduce DiffusionPID, a novel approach that leverages
partial information decomposition (PID), from information theory [4], to
interpret the diffusion process and uncover the contributions of individual
words and their interactions to the generated image. Compared to prior work
which typically examined the contribution of tokens individually, our approach
builds upon mutual information (MI) to examine the redundancy (R), synergy
(S), and uniqueness (U) of the conditioning tokens. Specifically, our approach
enables us to capture the interactions between concepts, revealing how
individual words and phrases combine to influence the generation of specific
image features. By quantifying these interactions, DiffusionPID provides a
more comprehensive understanding of the diffusion process, enabling the
development of more interpretable and controllable text-to-image models.

This work contributes: (1) a method to compute the image and pixel-level PID
of diffusion-generated images with respect to concepts from the input prompt;
(2) an evaluation of the DiffusionPID framework for elucidating model dynamics
(via R, S, U) on complex linguistic concepts such as synonyms and homonyms and
exposing ethnic/gender bias in the model; and finally, (3) a method of prompt
intervention via PID to remove redundant words and retain unique and
synergistic words.

##  2 Related Work

##### Text-to-Image (T2I) Diffusion Models.

Diffusion-based text-to-image models have shown impressive results in image
synthesis [5, 6, 7, 8]. These models are capable of generating images that are
both realistic and semantically consistent with the input text. Recent
datasets of challenging prompts and benchmarks based on visual question
answering and scene graphs have enabled researchers to assess and refine their
models [9, 10, 11, 12]. Inspired by the observation that existing models
struggle to accurately bind attributes to objects [13], some approaches
propose to modify the cross-attention layers by explicitly amplifying the
token-wise attention used for text conditioning [14, 15, 16, 17, 18], or by
composing multiple outputs to generate semantically compositional images [19].
These methods aim to improve the model’s ability to capture fine-grained
semantic relationships between text and image. These diverse approaches
demonstrate the ongoing effort to develop more effective and flexible text-to-
image synthesis models and highlight the need for continued research in this
area.

Interpretability in Deep Learning: Interpretability is key to understanding
and trusting the decisions made by complex models. LIME [20] presents a
technique for interpreting individual predictions of any machine learning (ML)
model via local linear models, while SHAP [21] assigns each feature an
importance value based on Shapley values from cooperative game theory. CAM
[22] and Grad-CAM [23] localize image regions in a CNN using a weighted sum of
intermediate layers’ activations. Network Dissection [24] operates at the
neuron level, tracking image regions corresponding to an excited neuron across
all the dataset images, which GAN Dissection [25] extends for GANs. MILAN [26]
and CLIP-Dissect [27] identify semantic concepts for an individual neuron.
Benchmarks such as ARO [28], Winoground [29], EQBEN [30], and WHOOPS [31]
introduce datasets to specifically probe the compositional and semantic
understanding of and show that most SOTA models struggle with relational
concepts. Some works have also explored the application of PID in different
settings. [32] uses PID to measure the contribution of each neuron towards a
target concept and [33] applies PID on generic multimodal datasets and models.
Finally, MULTIVIZ [34] analyzes the behavior of multimodal models,
specifically focusing on the unimodal, cross-modal, and multi-modal
contributions. Substantial progress has been made in understanding and
interpreting ML models. Given the explosion of work on diffusion models, we
primarily focus on interpreting these models.

Interpretability of Diffusion Models: Interpretability of diffusion models
remains an open challenge. Data attribution [35, 36, 37, 38] is a commonly
used approach to identify training samples most responsible for the appearance
of a generated image. Similarly, [39, 40] decompose the images associated with
a particular concept into the implicit semantic concepts the generative model
considers to be most similar. Others [41, 42, 43, 44, 45] have discovered a
semantic latent space for diffusion models such that modifications in this
space edit attributes of the original generated images. [46, 47] are similar
but operate in the text embedding space. Recently, there have also been works
that try to gain a neuron-level and network-level understanding of these
models. For instance, [48] divides the latent vector into multiple groups of
elements and designs different noise schedules for each group so that each
group controls only certain elements of data, explicitly giving interpretable
meaning. [49] identifies clusters of neurons corresponding to a given concept
with gradient statistics. [47, 50] show that diffusion models struggle on
prompts containing homonyms (words with multiple meanings) and [51, 52]
suggested the presence of ethnic, gender, and semantic biases in diffusion
models – we investigate both of these conditions in sections 4.3 and [4.1,
4.2], respectively. Recently, DAAM [53] proposed to analyze the cross-
attention layers of diffusion models to produce attribution maps for tokens
from the input prompt. However, these maps are generally very noisy and offer
limited information if the model fails to generate a given concept. [54]
introduced a method to compute the mutual information (MI) and conditional
mutual information (CMI) between the input text prompt and each generated
pixel. While their work provides stronger and more meaningful insights, their
approach only provides coarse information on the model. Our work enables fine-
grained model understanding by decomposing the mutual information between the
text prompt and image output by using partial information decomposition (PID).

##  3 Method

Image diffusion models learn to model an image distribution, $X$, through a
progressive noise addition and removal process. The forward diffusion process,
defined by $x_{\alpha}=\sqrt{\sigma(\alpha)}x+\sqrt{\sigma(-\alpha)}\epsilon$,
gradually adds Gaussian noise, $\epsilon\sim N(0,I)$, to the image, $x\sim
p(X)$, over a sequence of timesteps, where $\alpha$ denotes the log SNR/noise
schedule and $\sigma$ is the sigmoid function. A model learns to reverse this
process, denoising the data to gradually transform the noise back into samples
from the original data distribution. This denoising process has been applied
to text-to-image generation with incredible success by conditioning the
process on text prompts $y$. We employ the denoising process to measure the
influence of individual tokens (from the input prompt) and their interactions
on the resulting generation. We consider the diffusion model to be an optimal
denoiser that can predict the added ground-truth noise, $\epsilon$, at noise
level $\alpha$ as:

$\hat{\epsilon}_{\alpha}(x)=arg\min_{\hat{\epsilon}(.)}E_{p(x),p(\epsilon)}[\|\epsilon-\hat{\epsilon}_{\alpha}(x_{\alpha})\|^{2}]$
(1)

We follow [54]’s definition of mutual information (MI) and conditional mutual
information (CMI) computed using a diffusion model. Extending Eq 1 to the
conditional case where we assume $\hat{\epsilon}_{\alpha}(x_{\alpha}|y)$ is
the optimal denoiser for the conditional distribution $p(x|y)$, the Log
Likelihood Ratio (LLR), which is also the mutual information (MI) $i(y;x)$,
between a pixel $x$ and an input text prompt $y$ is defined as111The second
definition is derived using the orthogonality principle in [54].:

$\displaystyle log\ p(x\mid y)-log\ p(x)$ $\displaystyle=\frac{1}{2}\int
E_{p(\epsilon)}[||\epsilon-\hat{\epsilon}_{\alpha}(x_{\alpha})||^{2}-||\epsilon-\hat{\epsilon}_{\alpha}(x_{\alpha}\mid
y)||^{2}]d\alpha$ $\displaystyle log\ p(x\mid y)-log\ p(x)$
$\displaystyle=\frac{1}{2}\int
E_{p(\epsilon)}[||\hat{\epsilon}_{\alpha}(x_{\alpha})-\hat{\epsilon}_{\alpha}(x_{\alpha}\mid
y)||^{2}]d\alpha$ (2)

Essentially, MI quantifies the information an input variable $y$ individually
provides about an output variable $x$. This definition of pixel-wise MI
derived from [55] can be intuitive and easily understood as the quantity by
which the text prompt $y$ increases the probability of observing $x$ relative
to the prior probability without text conditioning. The same work provides an
alternate definition for MI as:

$\displaystyle i(y;x)$ $\displaystyle=-log\ p(y)+log\ p(y|x)=log\ p(x|y)-log\
p(x)$ (3)

Similarly for the CMI of phrase $y_{1}$ given the context of the phrase
$y_{2}$ in the input text prompt $y$:

$\displaystyle i(y_{1};x|y_{2})$ $\displaystyle=-log\ p(y_{1}|y_{2})+log\
p(y_{1}|x,y_{2})=log\ p(x|y_{1},y_{2})-log\ p(x|y_{2})$ (4)

From Eq 4, CMI can be understood as the information that $y_{1}$ contains
about $x$ beyond what is already contained in $y_{2}$. Following [55], we
provide the mathematical definitions for the various terms in the partial
information decomposition (PID) at the pixel-level. We start with the MI of
two input events (phrases) with an output variable (pixel) defined as:

$i(y_{1},y_{2};x)=r(y_{1},y_{2};x)+u(y_{1}\backslash
y_{2};x)+u(y_{2}\backslash y_{1};x)+s(y_{1},y_{2};x)$ (5)

Here, $y_{1}\backslash y_{2}$ means $y_{1}$ excluding $y_{2}$.
$r(y_{1},y_{2};x)$ is the redundant or overlapping information between $y_{1}$
and $y_{2}$ (redundancy), $u(y_{1}\backslash y_{2};x)$ is the unique
information contributed by $y_{1}$, $u(y_{2}\backslash y_{1};x)$ is the unique
information contributed by $y_{2}$, and $s(y_{1},y_{2};x)$ is the new
information contributed by the combination of $y_{1}$ and $y_{2}$ (synergy)
that neither of them could contribute on their own. It is important to note
that the uniqueness defined here is with respect to the other input variable,
i.e., it is the unique information the given phrase contributes that the other
phrase does not.

A natural measure of redundancy is the expected value of the minimum
information that any source provides about each outcome of $x$. This captures
the idea that redundancy is the information common to all sources (the minimum
information that any source provides) while taking into account that sources
may provide information about different outcomes of $x$. However, the
redundant information must capture when two predictor variables are carrying
the same information about the target, not merely the same amount of
information. This means that the sign/direction of information matters. To
account for this and to ensure the definition complies with all the required
axioms on redundancy, the redundancy is broken down into a positive
$r^{+}(y_{1},y_{2};x)$ and negative $r^{-}(y_{1},y_{2};x)$ component
associated with the informative $i^{+}(y_{i};x)$ and misinformative
$i^{-}(y_{i};x)$ MI terms respectively. Thus, the equation for computing the
redundancy can be derived as:

$\displaystyle r^{+}(y_{1},y_{2};x)$
$\displaystyle=\min_{y_{i}}i^{+}(y_{i};x)=\min_{y_{i}}[-log\ p(y_{i})],\quad
y_{i}\in\\{y_{1},y_{2}\\}$ (6) $\displaystyle r^{-}(y_{1},y_{2};x)$
$\displaystyle=\min_{y_{i}}i^{-}(y_{i};x)=\min_{y_{i}}[-log\ p(y_{i}|x)]$ (7)
$\displaystyle r(y_{1},y_{2};x)$ $\displaystyle=r^{+}-r^{-}=\min_{y_{i}}[-log\
p(y_{i})]-\min_{y_{i}}[-log\ p(y_{i}|x)]$ (8) $\displaystyle r(y_{1},y_{2};x)$
$\displaystyle=\min_{y_{i}}[-log\ p(y_{i})]-\min_{y_{i}}[-log\ p(x|y_{i})+log\
p(x)-log\ p(y_{i})]$ (9)

To compute the probability of an input event (phrase) $p(y)$ in the above
equations, we make use of BERT [56]. One of the tasks BERT was trained on was
to predict a masked token by predicting the probabilities of all possible
tokens from a fixed dictionary and taking the maximum. We use this to obtain
the probability of a given phrase. In the unconditional case, we only require
the objective probability of the phrase with no other tokens except the
special [MASK] token present in the string fed to BERT ("[MASK]").

Given the definitions for MI and redundancy, the uniqueness of an input
event/phrase and the synergy between phrases can be derived as:

$\displaystyle u(y_{1}\backslash y_{2};x)$
$\displaystyle=i(y_{1};x)-r(y_{1},y_{2};x)$ (10) $\displaystyle
s(y_{1},y_{2};x)$
$\displaystyle=i(y_{1},y_{2};x)-r(y_{1},y_{2};x)-u(y_{1}\backslash
y_{2};x)-u(y_{2}\backslash y_{1};x)$ (11)

It is important to note that all our definitions for all the information terms
require the evaluation of an integral over an infinite range of SNRs. In
practice, we make use of truncated logistic-based importance sampling to
evaluate this integral, similar to [54].

##  4 Experiments

We use our method to conduct a detailed analysis of diffusion models in
various situations. We introduce several tasks along with corresponding
datasets to study these models’ behavior in depth. For our experiments, we
primarily focus on the pre-trained Stable Diffusion 2.1 model from Hugging
Face. Latent diffusion models encode images into a lower dimensional latent
space before the denoising process. Thus, all our PID computations occur in
this latent space, i.e., we consider the image in this lower dimensional space
as $x$. During visualization, the heatmaps are bilinearly interpolated from
this latent space to the original image resolution. Finally, we make use of 50
samples for evaluating the integral over SNRs using importance sampling.

###  4.1 Gender Bias

Setup: Gender is a social construct and a complex study in its own right. In
this work, we search for the perpetuation of traditional gender roles and
associations. While there is a rich literature on the implications of such
biases and how models can exacerbate them [57, 58, 59, 60, 61, 62, 63, 64,
65], we simply present PID as another tool in that investigative process. In
this study, we specifically evaluate whether the diffusion model exhibits
gender bias in generating images of people in various occupations. Our
objective is to determine if the model consistently associates specific
genders with certain occupations. For this task, we take a set of 188 common
occupations such as professor, valet, receptionist, janitor, etc., and create
376 prompts (188 x 2) by joining the occupation with each gender, male or
female, one at a time. More details on the dataset creation process can be
found in the supplemental. To check for bias, we compare the image-level
redundancy values of each gender with the occupations. The image-level value
is simply computed by using the expectation of the MSE term over all pixels
instead of the pixel-level MSE values in our equations. The final redundancy
values are normalized across the entire dataset to the range $[0,1]$. We
hypothesize that whenever the redundancy of one gender is much higher than
that of the other with the occupation, the model is biased towards the higher
redundancy gender for that occupation.

Results: We can see in Table 2 that vocations that are typically stereotyped
to be performed by males such as plumber, carpenter, and police officer, have
high redundancy values for the male gender as compared to the female gender.
Similarly, we see higher redundancy for the female gender for female
stereotyped jobs such as babysitter and teacher. We also observe that the
average redundancy for females across all occupations is very low, indicating
significant model bias against generating females for any occupation. Thus, it
is clear that the model has learnt gender biases.

Occupation | Male | Female
---|---|---
Surgeon | 0.539 | 0.055
Soldier | 0.250 | 0.136
Judge | 0.304 | 0.286
Doctor | 0.871 | 0.090
Plumber | 0.605 | 0.038
Carpenter | 0.365 | 0.093
Police Officer | 0.390 | 0.091
Babysitter | 0.240 | 0.531
Teacher | 0.098 | 0.419
Average | 0.286 | 0.194
Table 1: Gender Bias: Redundancy between gender and occupation

| Occupation
---
Black | Asian | Caucasian | Hispanic
Athlete | 0.321 | 0.132 | 0.167 | 0.156
Artist | 0.106 | 0.069 | 0.062 | 0.045
| Engineer
---
0.126 | 0.156 | 0.080 | 0.097
Physicist | 0.109 | 0.209 | 0.162 | 0.064
Butcher | 0.110 | 0.179 | 0.474 | 0.396
Coach | 0.118 | 0.107 | 0.433 | 0.128
Nurse | 0.106 | 0.106 | 0.127 | 0.219
Agriculturist | 0.046 | 0.117 | 0.337 | 0.450
Average | 0.133 | 0.233 | 0.255 | 0.236
Table 2: Ethnicity Bias: Redundancy between ethnicity and occupation

###  4.2 Ethnic Bias

Setup: Similar to gender, race and ethnicity are not hard and fast classes,
but indicate how individuals self-identify. Previous work [66, 64, 67, 68, 69,
70] has shown that deep learning models are prone to learning racial and
ethnic biases, especially when the underlying training data is imbalanced.
With the recent shift towards diffusion-generated synthetic datasets [71, 72],
it has become crucial to test for the presence of such biases in diffusion.
Similar to the experiment above, we examine the diffusion model for ethnic
bias to see if it associates specific ethnicities with specific occupations.
We use the same 188 occupations used in the gender bias experiment but now we
combine them with each ethnicity (Caucasian, Black, Asian, and Hispanic)
instead of gender. We end up creating a dataset of 752 (188 x 4) image-prompt
pairs. Refer to the supplemental for more details on the dataset creation
process. The same approach is adopted as the previous experiment where we
compare the normalized image-level redundancy values of the various
ethnicities with the occupations.

Results: We can see from Table 2 that jobs that are commonly stereotyped for
certain ethnicities such as athlete with Black, engineer with Asian, and so
on, have higher redundancies with those respective ethnicities. We also
compare the redundancies of each ethnicity averaged over all occupations and
observe a very low value for the Black ethnic group which means that the model
is heavily biased against this group and is less likely to generate people
from this group for any occupation on average. Thus, we clearly see that the
model has learnt biases on the basis of ethnicity.

###  4.3 Homonyms

A diffusion model can behave in different manners when faced with prompts
containing homonyms in different contexts. A word is a homonym when it can
take on different meanings when used with different contextual words which we
term as modifiers. For example, the phrases "football match" and "he lit a
match" should generate two completely different visuals. We examine how
diffusion models handle such prompts in this experiment.

Setup: For this task, we use our approach to analyze the novel information
contributed towards the image generation process by different modifiers. We
create a dataset of 242 prompts containing homonyms in different contexts. We
study the synergy map between the homonyms and the modifiers to see if the
model is able to extract new information from their combination to generate an
image of the homonym in the right context.

Results: In the left figure of Fig 3, we can see our synergy maps correctly
highlight the area pertaining to the new information contributed by the
combination of the homonym ("bowl") and modifier ("ceramic" and "game"
respectively). However, in the right figure of Fig 3, the model fails to
generate the homonym ("mole") in the right context even when given the
appropriate modifiers ("coworker" and "searching"). This failure can be
understood by the low synergy maps which reveal that the model fails to learn
the synergistic relationship between the homonym and the modifiers. Thus, the
synergy maps help us decode these kinds of failures of the model.

Figure 2: Homonyms. We see that the word "baseball" provides the required
synergistic context with the homonym "bat" to pick the sports setting over the
animal. This effect can be confirmed in the synergy maps and image-level
synergy values (S) as well where we observe a high synergy for "bat" with
"baseball" compared to other words such as "He" and "swung". Figure 3:
Homonyms. Left: Successful generation of homonym "bowl" in different contexts
due to high synergy with modifiers "bowl" and "game". Right: Failure case
where the model generates the homonym "mole" with the same semantic meaning,
the animal, due to its failure to use contextual information from words like
"coworker" as can be seen in the synergy map.

###  4.4 Synonyms

Setup: Dealing with synonyms, i.e., different words with the same meaning, can
be challenging for diffusion models. Here, we probe the diffusion model with
prompts containing known synonym pairs to see how it handles them and if it
correctly identifies the given words as synonyms. To that end, we generated a
dataset of images from 132 text prompts containing synonym pairs. More details
on the dataset creation process can be found in the supplementary. Our
hypothesis is that the redundancy will be high between words that the model
considers as synonyms.

Results: In Fig 4, we can see how our approach very clearly outputs high
redundancy in the region of the object that both synonyms, "cube" and
"cuboid", refer to. Similarly, the redundancy map for the words "bed" and
"mattress" correctly highlights the bed region. To have a fair comparison with
MI, CMI, and DAAM, we take the intersection between the maps of the individual
synonyms produced by each of these methods (details in supplementary). It is
visible from Fig 4 that these methods fail to find the region of interest and
produce very sparse maps. Hence, our method is better suited for interpreting
what words/phrases the diffusion model considers semantically similar.

Figure 4: Synonyms. Our redundancy map is able to highlight that the model
considers the pairs "bed" and "mattress" (left) and "cube" and "cuboid"
(right) as semantically similar.

###  4.5 Co-Hyponyms

Previous work has shown that diffusion models sometimes fail to generate
certain objects mentioned in the prompt. One such set of cases the model
struggles with is prompts containing co-hyponyms, i.e., concepts that are
semantically very similar but not identical. The model tends to generate only
one of the objects rather than both of them or a fused version of the objects.
We hypothesize that this phenomenon occurs because the model confuses the two
concepts to carry the same semantic meaning even if they don’t for us as human
beings. This means that the redundancy between them is expected to be very
high which is also what we can see in our experiments. We introduce 2 prompt
datasets containing co-hyponyms and are discussed below.

####  4.5.1 Co-Hyponym COCO

Setup: To construct this dataset, we used the super-category hierarchy of the
COCO dataset [73] to extract co-hyponym pairs for the prompts. More details on
the dataset creation process can be found in the supplementary.

Results: In Fig 5, the redundancy is visibly high in the region where the
diffusion model has fused features from the words "cat" and "elephant".
Similarly, the redundancy is highly activated in the region of the only
generated object due to the high semantic similarity of the words "pizza" and
"sandwich". Thus, it is apparent that for cases where the diffusion model
fails to learn the difference in the semantic meaning of two co-hyponyms, it
confuses the two to be referring to the same concept, signified by the high
redundancy, and fails to generate one of the objects. Similar to the synonym
experiment above, we compare our redundancy map with the intersection maps
from MI, CMI, and DAAM and observe that they provide little information for
interpreting the reason for the model’s failure. Thus, the redundancy map
proves to be a useful tool to understand why the model fails here.

Figure 5: COCO Co-Hyponyms. The redundancy map proves to be very useful in
finding out the reason behind the model’s failures in these figures. It
confuses the co-hyponym pairs ("sandwich", "pizza")(left) and ("elephant",
"cat")(right) to have the same meaning for the co-hyponyms as seen from the
redundancy maps, which results in erroneous generations.

####  4.5.2 Co-Hyponym WORDNET

Setup: We make use of Wordnet’s [74] hyponym and hypernym relations between
the synsets of the words to obtain 798 co-hyponym pairs for the prompts. Refer
to the supplementary for more details on the dataset creation process.

Results: In the left figure of Fig 6, we can see the fusion of features from
both objects mentioned in the prompt, while in the right figure, we see only
one of the objects has been generated. The redundancy maps correctly highlight
the image regions pertaining to both the co-hyponyms. Again it is observed
that MI, CMI, and DAAM only very sparsely highlight the correct region of
overlap. This reinforces our findings from the previous COCO-based experiment
that the model mixes up the co-hyponyms and our redundancy map from PID helps
pin down the reason behind this phenomenon.

Figure 6: Wordnet Co-Hyponyms. Similar to the COCO Co-Hyponyms dataset, the
redundancy maps tell us that the model confuses the co-hyponym pairs
("barrier", "railing")(left) and ("chair", "sofa")(right) to have the same
meaning for the co-hyponyms, resulting in erroneous generations.

###  4.6 Prompt Intervention

We use our method to identify redundant words in the prompts, remove them, and
verify that this intervention results in little change to the image. We test
this task on all the datasets involving redundant terms: co-hyponym wordnet,
co-hyponym coco, synonyms, and occupation bias (ethnic and gender). For more
details on how we edit the image, refer to the supplementary.

Figure 7: Prompt Intervention. The redundancy is highly activated in the face
region of the giraffe. On a closer look, we see that the face is that of a cow
meaning that the word "cow" is redundant. This is confirmed as the image
changes very little on omitting it from the prompt. Figure 8: Prompt
Intervention. The redundancy is highly activated in the mouse region meaning
that the word "mouse" is redundant. The uniqueness of "mouse" is also very low
and spread out. This redundancy is confirmed as the image changes very little
on omitting it from the prompt.

Both figures 7 and 8 clearly show that whenever the redundancy is high in a
region corresponding to a particular object and the uniqueness for that
object’s term is low/spread out, then removing that object has minimal effect
on the image. On the other hand, we see that the MI, CMI, and DAAM maps are
highly activated for these redundant terms, indicating that these metrics are
not ideal for identifying redundant terms in the input prompt. For instance,
in the first figure, the redundancy map is highly activated in the mouse
region and the uniqueness of the term "mouse" is spread out. Removing the word
"mouse" results in an image very similar to the original whereas removing the
word "laptop" drastically changes the image. The MI, CMI, and DAAM maps are
notably high for the "mouse" term, incorrectly suggesting that removing this
term should have significantly altered the image. A similar effect can be
observed for the term "cow" in the second figure.

###  4.7 Most Representative Features

The most representative features of a concept are those characteristics that
uniquely define it from the model’s perspective. Given a set of classes, the
uniqueness map obtained from PID can be used to localize these features. As we
can see in Fig 9, when the input prompt is "hair dryer and toothbrush", even
though the image is a mix of a hair dryer and a toothbrush, the uniqueness map
highlights the region corresponding to the toothbrush bristles. Similarly, we
can see the bear’s facial features being highlighted correctly in the other
example. Thus, our uniqueness maps prove useful for this task of identifying
the unique characteristics of an object.

Figure 9: Most Representative Features. Left: The "toothbrush" uniqueness map
correctly captures the toothbrush bristles, their most distinct feature.
Right: The "bear" uniqueness map correctly captures the bear region,
specifically the face.

##  5 Discussion

Limitations and Future Work: Although we have exhibited the efficacy and
benefits of DiffusionPID, there remains some scope for improvement that future
work can explore. In all our experiments, we compute the PID terms only for
two phrases from the prompt but PID can be extended to more than two input
variables. Another interesting research direction could be to compute PID on
other, non-diffusion-based models. Also, our approach requires access to the
diffusion model, making it difficult to apply it to closed-source models,
which means that alternative methods of computing PID need to be explored.
There are also some other applications that could be tried with PID such as
using the uniqueness information to localize distinctive features to
differentiate closely related classes in image classification. Recently there
have been works such as [75] that integrate other forms of conditioning such
as masks to enable better control of diffusion models. PID-based analysis of
these multimodal forms of conditioning is another worthwhile research
direction for the future.

Conclusion: The fine-grained breakdown of MI afforded by our approach,
DiffusionPID, allows us to understand the decision-making process of diffusion
models, identify their shortcomings, and pin down the reasons for their
failures. This understanding is critical given the misalignment between the
model’s world understanding and ours. We can use the insights furnished from
our approach to make diffusion models better aligned with humans’ conceptual
understanding and counter its limitations and biases. Our work can serve as a
strong fundamental basis for further research on using information theoretic
concepts such as PID to dissect, study, and enhance deep learning models.

## References

  * Liu and Chilton [2022] Vivian Liu and Lydia B Chilton.  Design guidelines for prompt engineering text-to-image generative models.  CHI ’22, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450391573.  doi: 10.1145/3491102.3501825.  URL https://doi.org/10.1145/3491102.3501825.
  * Wen et al. [2023] Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=VOstHxDdsN.
  * Nam et al. [2022] Junhyun Nam, Sangwoo Mo, Jaeho Lee, and Jinwoo Shin.  Breaking the spurious causality of conditional generation via fairness intervention with corrective sampling.  _arXiv preprint arXiv:2212.02090_ , 2022.
  * Williams and Beer [2010] Paul L. Williams and Randall D. Beer.  Nonnegative decomposition of multivariate information, 2010.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ , 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  _ArXiv_ , abs/2205.11487, 2022.
  * Hu et al. [2023] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith.  Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering.  _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 20349–20360, 2023.
  * Lin et al. [2024] Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan.  Evaluating text-to-visual generation with image-to-text generation.  _ArXiv_ , abs/2404.01291, 2024.
  * Cho et al. [2024] Jaemin Cho, Yushi Hu, Jason Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont-Tuset, and Su Wang.  Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation.  In _ICLR_ , 2024.
  * Conwell and Ullman [2022] Colin Conwell and Tomer David Ullman.  Testing relational understanding in text-guided image generation.  _ArXiv_ , abs/2208.00005, 2022.
  * Feng et al. [2023] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Training-free structured diffusion guidance for compositional text-to-image synthesis.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=PUIqjT4rzq7.
  * Chefer et al. [2023a] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  _ACM Transactions on Graphics (TOG)_ , 42:1 – 10, 2023a.
  * Rassin et al. [2023] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik.  Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=AOKU4nRw1W.
  * Wang et al. [2023a] Ruichen Wang, Zekang Chen, Chen Chen, Jiancang Ma, H. Lu, and Xiaodong Lin.  Compositional text-to-image synthesis with attention map control of diffusion models.  _ArXiv_ , abs/2305.13921, 2023a.
  * Phung et al. [2023] Quynh Phung, Songwei Ge, and Jia-Bin Huang.  Grounded text-to-image synthesis with attention refocusing.  _ArXiv_ , abs/2306.05427, 2023.
  * Liu et al. [2022] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum.  Compositional visual generation with composable diffusion models.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVII_ , pages 423–439. Springer, 2022.
  * Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  “why should i trust you?”: Explaining the predictions of any classifier.  _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2016.
  * Lundberg and Lee [2017] Scott M. Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In _Neural Information Processing Systems_ , 2017.
  * Zhou et al. [2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2921–2929, 2016.
  * Selvaraju et al. [2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 618–626, 2017.
  * Bau et al. [2017] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Network dissection: Quantifying interpretability of deep visual representations.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6541–6549, 2017.
  * Bau et al. [2018] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William T Freeman, and Antonio Torralba.  Gan dissection: Visualizing and understanding generative adversarial networks.  _arXiv preprint arXiv:1811.10597_ , 2018.
  * Hernandez et al. [2021] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas.  Natural language descriptions of deep visual features.  In _International Conference on Learning Representations_ , 2021.
  * Oikarinen and Weng [2022] Tuomas P. Oikarinen and Tsui-Wei Weng.  Clip-dissect: Automatic description of neuron representations in deep vision networks.  _ArXiv_ , abs/2204.10965, 2022.
  * Yuksekgonul et al. [2023] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou.  When and why vision-language models behave like bags-of-words, and what to do about it?  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=KRLUvxh8uaX.
  * Diwan et al. [2022] Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, and Kyle Mahowald.  Why is winoground hard? investigating failures in visuolinguistic compositionality.  _arXiv preprint arXiv:2211.00768_ , 2022.
  * Wang et al. [2023b] Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang.  Equivariant similarity for vision-language foundation models.  _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 11964–11974, 2023b.
  * Bitton-Guetta et al. [2023] Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz.  Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images.  _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2616–2627, 2023.
  * Ehrlich et al. [2023] David Alexander Ehrlich, Andreas Christian Schneider, Viola Priesemann, Michael Wibral, and Abdullah Makkeh.  A measure of the complexity of neural representations based on partial information decomposition.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=R8TU3pfzFr.
  * Liang et al. [2023] Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard J. Chen, Zihao Deng, Nicholas Allen, Randy P. Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency.  Quantifying & modeling multimodal interactions: An information decomposition framework.  In _Neural Information Processing Systems_ , 2023.
  * Liang et al. [2022] Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis-Philippe Morency, and Ruslan Salakhutdinov.  Multiviz: Towards visualizing and understanding multimodal models.  In _International Conference on Learning Representations_ , 2022.
  * Zheng et al. [2024] Xiaosen Zheng, Tianyu Pang, Chao Du, Jing Jiang, and Min Lin.  Intriguing properties of data attribution on diffusion models.  In _International Conference on Learning Representations (ICLR)_ , 2024.
  * Georgiev et al. [2023] Kristian Georgiev, Joshua Vendrow, Hadi Salman, Sung Min Park, and Aleksander Madry.  The journey, not the destination: How data guides diffusion models.  In _Arxiv preprint arXiv:2312.06205_ , 2023.
  * Dai and Gifford [2023] Zheng Dai and David Kenneth Gifford.  Training data attribution for diffusion models.  _ArXiv_ , abs/2306.02174, 2023.
  * Wang et al. [2023c] Sheng-Yu Wang, Alexei A. Efros, Jun-Yan Zhu, and Richard Zhang.  Evaluating data attribution for text-to-image models.  In _ICCV_ , 2023c.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _ArXiv_ , abs/2208.01618, 2022.
  * Chefer et al. [2023b] Hila Chefer, Oran Lang, Mor Geva, Volodymyr Polosukhin, Assaf Shocher, Michal Irani, Inbar Mosseri, and Lior Wolf.  The hidden language of diffusion models.  _arXiv preprint arXiv:2306.00966_ , 2023b.
  * Kwon et al. [2022] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh.  Diffusion models already have a semantic latent space.  abs/2210.10960, 2022.
  * Li et al. [2023] Hang Li, Chengzhi Shen, Philip Torr, Volker Tresp, and Jindong Gu.  Self-discovering interpretable diffusion latent directions for responsible text-to-image generation.  _arXiv preprint arXiv:2311.17216_ , 2023.
  * Haas et al. [2023] René Haas, Inbar Huberman-Spiegelglas, Rotem Mulayoff, and Tomer Michaeli.  Discovering interpretable directions in the semantic latent space of diffusion models.  _ArXiv_ , abs/2303.11073, 2023.
  * Park et al. [2023] Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, and Youngjung Uh.  Understanding the latent space of diffusion models through the lens of riemannian geometry.  In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 24129–24142. Curran Associates, Inc., 2023.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/4bfcebedf7a2967c410b64670f27f904-Paper-Conference.pdf.
  * Dalva and Yanardag [2023] Yusuf Dalva and Pinar Yanardag.  Noiseclr: A contrastive learning approach for unsupervised discovery of interpretable directions in diffusion models, 2023.
  * Wu et al. [2022] Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung M. Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang.  Uncovering the disentanglement capability in text-to-image diffusion models.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1900–1910, 2022.
  * White and Cotterell [2022] Jennifer C. White and Ryan Cotterell.  Schrodinger’s bat: Diffusion models sometimes generate polysemous words in superposition, 2022.  URL https://arxiv.org/pdf/2211.13095.pdf.
  * Lee et al. [2023] Sangyun Lee, Gayoung Lee, Hyunsu Kim, Junho Kim, and Youngjung Uh.  Diffusion models with grouped latents for interpretable latent space.  In _ICML 2023 Workshop on Structured Probabilistic Inference $\\{$$\backslash$&$\\}$ Generative Modeling_, 2023.
  * Liu et al. [2023] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao.  Cones: Concept neurons in diffusion models for customized generation.  In _International Conference on Machine Learning_ , 2023.
  * Rassin et al. [2022] Royi Rassin, Shauli Ravfogel, and Yoav Goldberg.  DALLE-2 is seeing double: Flaws in word-to-concept mapping in text2image models.  _CoRR_ , abs/2210.10606, 2022.  doi: 10.48550/ARXIV.2210.10606.  URL https://doi.org/10.48550/arXiv.2210.10606.
  * Luccioni et al. [2023] Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite.  Stable bias: Evaluating societal representations in diffusion models.  In _Neural Information Processing Systems_ , 2023.
  * Orgad et al. [2023] Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov.  Editing implicit assumptions in text-to-image diffusion models.  _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7030–7038, 2023.
  * Tang et al. [2022] Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture.  What the daam: Interpreting stable diffusion using cross attention.  _arXiv preprint arXiv:2210.04885_ , 2022.
  * Kong et al. [2023] Xianghao Kong, Ollie Liu, Han Li, Dani Yogatama, and Greg Ver Steeg.  Interpretable diffusion via information decomposition.  _arXiv preprint arXiv:2310.07972_ , 2023.
  * Finn and Lizier [2017] Conor Finn and Joseph T. Lizier.  Pointwise information decomposition using the specificity and ambiguity lattices.  _ArXiv_ , abs/1801.09010, 2017.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _North American Chapter of the Association for Computational Linguistics_ , 2019.
  * Wang et al. [2019] Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez.  Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 5310–5319, 2019.
  * Zhao et al. [2017] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang.  Men also like shopping: Reducing gender bias amplification using corpus-level constraints.  In _Conference on Empirical Methods in Natural Language Processing_ , 2017.
  * Srinivasan and Bisk [2021] Tejas Srinivasan and Yonatan Bisk.  Worst of both worlds: Biases compound in pre-trained vision-and-language models.  _ArXiv_ , abs/2104.08666, 2021.
  * Burns et al. [2018] Kaylee Burns, Lisa Anne Hendricks, Kate Saenko, Trevor Darrell, and Anna Rohrbach.  Women also snowboard: Overcoming bias in captioning models.  _arXiv preprint arXiv:1803.09797_ , 2018.
  * Atay et al. [2023] Mustafa Atay, Megh Poudyel, and Saul Evora.  Evaluation of gender bias in masked face recognition with deep learning models.  _2023 IEEE Symposium Series on Computational Intelligence (SSCI)_ , pages 829–835, 2023.
  * Islam and Islam [2023] Ahsan Ul Islam and Abm Rezbaul Islam.  Gender bias in pre-trained deep learning models.  _2023 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE)_ , pages 1–6, 2023.
  * Hall et al. [2023] Siobhan Mackenzie Hall, Fernanda Gonçalves Abrantes, Hanwen Zhu, Grace Sodunke, Aleksandar Shtedritski, and Hannah Rose Kirk.  Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://openreview.net/forum?id=BNwsJ4bFsc.
  * Hirota et al. [2022] Yusuke Hirota, Yuta Nakashima, and Noa García.  Quantifying societal bias amplification in image captioning.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 13440–13449, 2022.
  * Bhargava and Forsyth [2019] Shruti Bhargava and David A. Forsyth.  Exposing and correcting the gender bias in image captioning datasets and models.  _ArXiv_ , abs/1912.00578, 2019.
  * Stock and Cisse [2018] Pierre Stock and Moustapha Cisse.  Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 498–512, 2018.
  * Khan and Fu [2021] Zaid Khan and Yun Raymond Fu.  One label, one billion faces: Usage and consistency of racial categories in computer vision.  _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , 2021.
  * Kärkkäinen and Joo [2021] Kimmo Kärkkäinen and Jungseock Joo.  Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation.  _2021 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 1547–1557, 2021.
  * Sengupta and Srivastava [2022] Kinshuk Sengupta and Praveen Ranjan Srivastava.  Causal effect of racial bias in data and machine learning algorithms on user persuasiveness & discriminatory decision making: An empirical study.  _ArXiv_ , abs/2202.00471, 2022.
  * Sap et al. [2019] Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith.  The risk of racial bias in hate speech detection.  In _Annual Meeting of the Association for Computational Linguistics_ , 2019.
  * Sun et al. [2023] Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li.  JourneyDB: A benchmark for generative image understanding.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://openreview.net/forum?id=vfzXDRTcF4.
  * Wang et al. [2022] Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau.  DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models.  _arXiv:2210.14896 [cs]_ , 2022.  URL https://arxiv.org/abs/2210.14896.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Miller [1995] George A. Miller.  Wordnet: A lexical database for english.  _Commun. ACM_ , 38:39–41, 1995.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023.
  * Finn and Lizier [2018] Conor Finn and Joseph T Lizier.  Pointwise partial information decomposition using the specificity and ambiguity lattices.  _Entropy_ , 20(4):297, 2018.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _Conference on Computer Vision and Pattern Recognition 2023_ , 2023.
  * Tsaban and Passos [2023] Linoy Tsaban and Apolinário Passos.  Ledits: Real image editing with ddpm inversion and semantic guidance.  2023\.
  * Fellbaum [1998] Christiane Fellbaum.  _WordNet: An Electronic Lexical Database_.  The MIT Press, 05 1998.
  * Lin et al. [2015] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár.  Microsoft coco: Common objects in context, 2015.
  * OpenAI [2023] OpenAI.  Chatgpt: A large language model trained by openai.  https://chat.openai.com/, 2023.  Accessed: 2024-05-20.
  * [82] Meta AI — meta.ai.  https://www.meta.ai/.  [Accessed 22-05-2024].
  * Hassabis and Team [2024] Demis Hassabis and Google DeepMind Team.  Introducing gemini: Google’s most capable ai model yet, 2024.  URL https://blog.google/technology/ai/introducing-gemini/.  Accessed: 2024-05-20.

##  6 Appendix

NOTE: We also share our code and data as a zip file in the supplementary
material.

###  6.1 MI and PID

Here, we explain the concepts of mutual information and partial information
decomposition in greater detail.

Mutual Information (MI) is a measure that estimates the amount of information
obtained for an output variable from a predictor variable passed through a
function. It measures to what extent an input variable reduces the uncertainty
in the model’s prediction. In deep learning, where understanding the
dependencies and mapping between input features and target variables is vital,
MI serves as a powerful tool to quantify these dependencies accurately.

Partial Information Decomposition (PID), as defined by [76], shows that two
predictor variables $X_{1}$ and $X_{2}$ can contribute information about a
target variable $Y$ in four possible ways. $X_{1}$ can provide some
information about $Y$ independently/uniquely, $X_{2}$ too can provide some
information about $Y$ independently/uniquely, both $X_{1}$ & $X_{2}$ can
contribute some overlapping/redundant information, i.e., some of the
information contributed by both is the same, and $X_{1}$ & $X_{2}$ can
contribute synergistic information which is information neither of them can
contribute on their own and only in conjunction with the other. [76] also
state that mutual information can be derived from first principles as
fundamentally pointwise quantities where they measure the information content
of individual events. The overall (average) mutual information can then be
calculated by taking the expectation over all events for the relevant
variables.

###  6.2 Baselines for Redundancy

When comparing CMI, MI, and DAAM with our redundancy maps, we take the
intersection of the maps these methods produce for the two words/phrases
individually. This is done in the following steps:

  * •

The individual maps are thresholded to obtain binary masks. The threshold we
employ is $\mu+1.5\sigma$, where $\mu$ refers to the mean of the map being
thresholded and $\sigma$ refers to the standard deviation. We use 1.5 times
the standard deviation to discard roughly 86.64 % of the map’s values as we
observe that they are irrelevant and correspond to the low-activated
background region.

  * •

We multiply the binary masks of the two words’ maps to obtain the final
regions of intersection/overlap between the two maps.

  * •

Finally, the above intersection map $im$ is multiplied with each of the two
maps, $m_{1}$ and $m_{2}$, followed by the computation of the pixel-level mean
of the two masked maps to obtain the final map $f$ as:

$f=(m_{1}*im+m_{2}*im)/2$ (12)

###  6.3 Image Editing via Prompt Intervention

We follow a similar process to other image editing methods [77, 78] where we
progressively add Gaussian noise to the image and pass it through the
diffusion model for denoising but now this reverse diffusion process is
conditioned on the edited prompt instead of the original prompt. We edit the
original prompt by removing a word from it to see if the edited image is
similar to the original to help determine if the omitted word is redundant.
For instance, we remove the word "desk" from the prompt "The desk was a sturdy
table perfect for working" to produce the edited prompt "The was a sturdy
table perfect for working" which is then used to condition the denoising
process. The intuition behind this experiment is that for words that
contribute little information on top of the rest of the prompt, the model will
keep the edited image very similar to the original as it will follow a similar
path as that would be followed if it were conditioned on the original prompt.

###  6.4 Datasets

We create datasets for each of our numerous experiments. We primarily rely on
existing hierarchical datasets, Wordnet [79] and COCO [80], and free-access
Large Language Models, namely Chat-GPT [81], Meta.ai [82], and Gemini [83]. We
probe these models with prompts like "make sentences with pairs of synonyms of
common tangible nouns", "make sentences with category and subcategory of
common tangible nouns", "generate sentences with synonyms of common tangible
nouns and both of those synonyms must be present in the sentence, "generate
pairs of sentences containing homonyms in different contexts" and so on. A
combination of the outputs produced by these models was used to obtain the
final set of prompts for each task. Once we had the prompts, we fed them into
the open-source diffusion model, Stable Diffusion 2.1 [7], to generate images
corresponding to those prompts. During the image generation, we kept the seed
constant (42). Following are the prompt-image pair datasets we introduce:

  * •

Gender Bias Dataset: Here we generated a list of 188 most common occupations
from the LLMs ([82],[81],[83]). These occupations are then combined with each
gender one at a time to produce a total of 376 prompts. For instance, the
occupation "Doctor" is used to generate two prompts, "Female Doctor" and "Male
Doctor". A sample of this dataset is visible in Fig 10.

Figure 10: Gender Bias Dataset Sample

  * •

Ethinic Bias Dataset: Instead of combining the previously mentioned 188
occupations with a gender, we now combine them with one of four ethnicities
(Caucasian, Asian, Black, Hispanic) in each prompt to produce a total of 752
prompts. For example, the occupation "Doctor" is used to generate four
prompts, "Caucasian Doctor", "Asian Doctor", "Black Doctor", and "Hispanic
Doctor". A sample of this dataset is visible in Fig 11.

Figure 11: Ethnic Bias Dataset Sample

  * •

Homonym Dataset: For this dataset, we generated 121 pairs of prompts, where
each pair corresponds to one homonym but in different contexts. Thus, in
total, we produced 242 prompts, all of which were generated from the LLMs
[82],[81],[83]. A sample of this dataset is visible in figures 12(a).

(a) Homonym Dataset Sample

(b) Co-Hyponym Dataset (COCO) Sample

  * •

Synonym Dataset: The entire dataset of 132 prompts containing synonym pairs
was generated using the previously mentioned LLMs [82],[81],[83]. A sample of
this dataset is visible in Fig 13.

Figure 13: Synonym Dataset Sample

  * •

Co-Hyponym Dataset (Wordnet): To create this dataset, we first sampled co-
hyponyms from Wordnet for the most frequently occurring tangible nouns in the
COCO dataset. Next, we generated prompts with the format: "a co-hyponym and a
co-hyponym". For example, one of the most commonly used nouns across the COCO
dataset is "bench", so we find its co-hyponyms from Wordnet: "box", "seat",
"chair", "ottoman", "stool", "sofa" and "couch". Then, these are used to
produce prompts such as "a box and a bench", "a chair and a bench", "a sofa
and a bench" and so on. A sample of this dataset is visible in Fig 14.

Figure 14: Wordnet Dataset Sample

  * •

Co-Hyponym Dataset (COCO): We took the super-categories of the COCO dataset
and paired the objects within each individual super-category to produce a list
of 197 co-hyponym pairs which are then converted to prompts using a similar
format as that used for the Wordnet dataset above. For example, "vehicle" was
a super-category, so "a car and a airplane", "a car and a boat" and "a
motorcycle and a truck" are a few prompts produced from that super-category. A
sample of this dataset is visible in Fig 12(b).

###  6.5 More Results

In this section, we present more results from our experiments.

####  6.5.1 Homonyms

Below are some more examples of results for Homonym Experiments.

.

####  6.5.2 Synonyms

Below are some more examples of results for Synonyms Experiments.

####  6.5.3 COCO Co-Hyponyms

Below are the results of Co-Hyponyms.

####  6.5.4 Wordnet Co-Hyponyms

Below are some more results for the Co-Hyponym experiments on the images
generated from prompts made through Wordnet.

####  6.5.5 Prompt Intervention

Below are some more results for the Prompt Intervention experiments.