# Enhancing Prompt Following with Visual Control
Through Training-Free Mask-Guided Diffusion

Hongyu Chen1 Yiqi Gao2* Min Zhou1 Peng Wang2 Xubin Li1
Tiezheng Ge1 Bo Zheng 1
{yinchen.chy, yunqi.zm, lxb204722, tiezheng.gtz<EMAIL_ADDRESS>
{gyqjz<EMAIL_ADDRESS>
1Alibaba Group
2Northwestern Polytechnical University Both authors contributed equally to
this work.

###### Abstract

Recently, integrating visual controls into text-to-image (T2I) models, such as
ControlNet method, has received significant attention for finer control
capabilities. While various training-free methods make efforts to enhance
prompt following in T2I models, the issue under visual control is still rarely
studied, especially in the scenario that visual controls are misaligned with
text prompts. In this paper, we address the challenge of “Prompt Following
With Visual Control” and propose a training-free approach named Mask-guided
Prompt Following (MGPF). Object masks are introduced to distinct aligned and
misaligned parts of visual controls and prompts. Meanwhile, a network, dubbed
as Masked ControlNet, is designed to utilize these object masks for object
generation in the misaligned visual control region. Further, to improve
attribute matching, a simple yet efficient loss is designed to align the
attention maps of attributes with object regions constrained by ControlNet and
object masks. The efficacy and superiority of MGPF are validated through
comprehensive quantitative and qualitative experiments.

##  1 Introduction

Adding image conditions to text-to-image (T2I) models [26, 19, 51] for
accurate layout and shape control has attracted significant attention.
Notably, ControlNet [51] stands out as a widely adopted solution for its
effectiveness in producing high-quality images with diverse visual controls
such as Canny Edge [6], Depth Map [36], and other form images. Though many
training-free methods [8, 14, 4, 2, 27, 47, 1, 7, 22, 20, 29] have been
proposed to make generated images consistent with text prompts in T2I models,
addressing the “Prompt Following” issue [5], research on this issue within
ControlNet is rare. This paper delves into this unexplored area, labeled as
“Prompt Following Under Visual Control”.

Figure 1: The input prompt and visual control are only partially aligned as
the left part displays. The ControlNet method result omits yellow flowers and
a grassy park, inaccurately painting the cup as blue not white. While our
approach MGPF can utilize object masks and produce images with these desired
features.

The key challenge lies in ensuring the generated images match not only the
text prompts but also the layouts and shapes specified by the visual controls.
This task becomes more crucial when users can only provide partially aligned
prompt-image pairs since it is hard to find visual controls exactly matching
their prompts. For example, as Fig. 1 shows, the canny edge condition derived
from an indoor setting only partially aligns with the prompt. This
misalignment influences the effectiveness of the prompt and results in missing
elements like flowers and grass in the generated image (termed as “Object
Missing”). Moreover, the attributes of objects are incorrect, such as the
color of the cup being blue instead of white (termed as “Attribute Mismatch”).
This issue may arise from the inconsistency between the attention maps of the
attribute words and that of the corresponding object during the denoising
process [43].

Directly transferring prompt following methods from T2I to the ControlNet
scenario does not yield satisfactory results. Specifically, in optimizing
Object Missing, some works [8, 14, 4, 2, 27, 47] focus on augmenting the
attention values of object words in the prompt during cross-attention, proving
effective in T2I scenarios. However, when applied to ControlNet [51], these
methods still suffer from the misalignment of prompts and visual controls,
leading to a weakened effect. In addressing Attribute Mismatch, some
studies[37, 8, 20] suggest bringing the attention maps of attribute words and
object words within U-net closer to each other. This process may affect the
original layout distribution of objects. Such approaches are effective in T2I
scenarios without layout and shape constraints, but may conflict with the
visual controls in ControlNet.

To address these issues, we introduce additional masks for objects aligned
with prompts, and propose a novel training-free approach named Mask-guided
Prompt Following (MGPF). Regarding the “Object Missing” challenge, we
introduce Masked ControlNet to replace the original ControlNet branch and
mitigate conflicts between prompts and the misaligned part of visual controls.
Specifically, we employ object masks to divide ControlNet features into two
parts, only integrating features aligned with the prompt into the U-net. We
note that partial ControlNet features can effectively control particular
spatial compositions in the generated image without disrupting non-adjacent
regions or introducing artifacts. This is likely because the U-net remains
fixed during training, preserving its ability to generate images. Also,
combining ControlNet features with U-net features maintains spatial
information in the ControlNet features. Additionally, ControlNet features are
combined with U-net features through element-wise addition, preserving the
spatial information from visual controls.

Furthermore, we propose Attribute Matching loss for binding attributes to
objects and avoid disturbing visual controls. As depicted in Fig. 2, we give
different prompts to U-net and ControlNet and find that the cross-attention
between U-net and text features determines the attribute of objects while
ControlNet predominantly determines the layout and shape of objects. Inspired
by this, we utilize classifier guidance [9] and minimize the divergence
distance between attention maps of attribute words in Stable Diffusion and
those of object words in ControlNet. Additionally, we impose a constraint to
ensure that the regions of highest attention for attribute words and
corresponding object words within the object mask regions. For the misaligned
prompt and visual control, we just make the attention maps for attributes and
objects within the U-net consistent.

To evaluate our method, we extract various visual control and object masks
from the COCO dataset [21] and design paired prompts that include a variety of
attributes and objects. The experiments demonstrate that our model outperforms
state-of-the-art methods.

In summary, our works can be summarized as: (i) We explore the challenge of
prompt following under visual control and introduce object masks for better
semantic expression. (ii) We propose a training-free method called Masked-
guided Prompt Following, including Masked ControlNet and Attribute-matching
Loss for precise object and attribute control. (iii) Extensive comparisons
show that our method is simple yet effective for controllable text-to-image
generation within multiple conditions.

Figure 2: Under visual control, the prompt is respectively fed into or unfed
into the U-net and ControlNet. If U-net does not received the prompt,
regardless of whether ControlNet has received it or not, the image lacks
yellow and purple as mentioned. But if U-net gets the prompt, it is the
opposite. This reveals that attribute words mostly work through the cross-
attention between U-net and the prompt features.

##  2 Related Works

Text-to-Image Generation. Early research in text-to-image generation primarily
utilized Generative Adversarial Networks (GANs), as demonstrated in studies by
Tao Tang et al. [44], Xu et al. [48], and Zhang et al. [50]. However, recent
advancements have shifted focus towards diffusion models [34, 35, 10, 3, 15,
42, 28], gaining prominence for their exceptional capabilities in generating
high-quality and high-resolution images. A notable example in this domain is
the Denoising Diffusion Probabilistic Model (DDPM) [15], which introduces
standard noise in its forward process and reconstructs the original image from
the noisy data in the reverse process. In this reverse process, a U-Net-based
model [39] is employed to estimate the noise at each step. This era is further
marked by groundbreaking large-scale text-to-image architectures such as
Stable Diffusion [38], Imagen by Saharia et al. [40], and DALL·E 3 by OpenAI
[5]. Besides, these models have been applied to a diverse range of tasks
beyond those initially mentioned, including 3D generation [30, 12, 17], video
editing [13, 23, 25, 31, 49] and motion generation [32, 45]. Building upon
these state-of-the-art models, our work aims to enhance the prompt-following
capabilities of these systems under controlled conditions.

Prompt Following. Generating images that precisely align with complex text
prompts remains a formidable challenge, as proved in recent studies [8, 11,
14, 46]. This issue is particularly pronounced in scenarios involving complex
scenes or multi-object compositions [24, 11, 8]. StructureDiffusion [11]
attempts to address this by leveraging linguistic structures to guide cross-
attention maps in the generation process but falls short in resolving semantic
discrepancies at the sample level. Composable Diffusion [22] decomposes
complex texts into simpler segments and then composes the image from these
segments, though its effectiveness is limited to simpler conjunctions.
Additionally, A&E [8] aims to enhance object presence in generated images by
optimizing cross-attention maps during inference. However, it struggles with
intricate prompts, particularly those with complex backgrounds, and does not
address attribute-related issues effectively. In response, Linguistic
Diffusion [37] introduces a specialized KL-divergence loss to fine-tune the
cross-attention maps of modifiers and entities, making significant progress.
Despite these advancements, existing methods primarily focus on general text-
to-image model improvements. In contrast, our paper targets enhancing prompt
following under visual control, an area where previous approaches have shown
limitations.

Controlling Text-to-Image Diffusion Models. The state-of-the-art diffusion
models primarily discussed earlier are geared towards text-to-image
generation, relying heavily on text prompts for control. However, achieving
precise control solely through text prompts presents inherent challenges.
Recent developments have introduced methods to exert additional visual control
over existing text-to-image models. Innovations like ControlNet [51] and T2I
Adapter [26] facilitate this enhanced control, enabling models such as Stable
Diffusion [38] (SD) to adapt to various image contexts with minimal additional
training. ControlNet [51] integrates supplementary features from diverse image
settings into the core SD U-Net model, while T2I Adapters [26] employ a
different approach in incorporating these features and managing inputs. Our
research focuses on scenarios using ControlNet [51], where we observe a
notable decline in the quality of prompt following compared to general
scenarios. To the best of our knowledge, ours is the first work addressing
this specific challenge. We propose a novel training-free method to resolve
this issue without compromising flexibility.

##  3 Method

This section elaborates on the proposed Mask-guided Prompt Following (MGPF).
As illustrated in Fig. 3, this method adopts SD for achieving high-quality
text-to-image generation and ControlNet to integrate diverse visual controls.
MGPF consists of two key components: 1. Masked ControlNet, which leverages
object masks to control the layout and shape of specific regions, thereby
eliminating conflicts with misaligned portions of the prompt and enhancing
object generation; 2. Attribute-matching Loss, which employs related loss
functions to bind attributes within the prompt to their corresponding objects.
The text prompt$\mathcal{P}=\\{p\\}_{i=1}^{L}$and image condition $I$ serve as
fundamental inputs for controlled image synthesis, where $L$ denotes the
length of prompt. In addition, object masks $\mathcal{M}=\\{m\\}_{i=1}^{N}$
are provided, where $m_{i}$ denotes pixel indices associated with the $i_{th}$
object described in $\mathcal{P}$ and $I$.

Figure 3: Overview of the proposed Mask-guided Prompt Following (MGPF) method.
Given a prompt and a canny edge condition with misaligned elements such as
door edges, along with two object masks indicating “dog” and “skateboard”, our
approach involves two modules to enhance prompt following. In Masked
ControlNet, We union all object masks into a single composite, being reshaped
and element-wise multiplied to corresponding ControlNet features, effectively
eliminating the influence of undesired visual clues. Incorporated with
Attribute-matching Loss, we parse the prompt into attribute-object pairs like
“yellow dog” and “green skateboard”, obtaining their cross-attention maps from
U-net and ControlNet. Subsequently, specific loss functions shift these
attention maps in U-net and ControlNet for better attribute binding.

###  3.1 Masked ControlNet

To address the challenge of generating objects in areas where the prompt and
visual controls are misaligned, We propose the Masked ControlNet, which is
then integrated with Stable Diffusion(SD) [38]. SD, built on a U-net
architecture, includes an encoder, a middle block and a decoder, all
interconnected via skip connections. It utilizes the prompt as a basis
condition for text-to-image generation. To incorporate more visual controls,
ControlNet [51] is developed using a trainable copy of the 12 encoding blocks
and 1 middle block of SD. The outputs from ControlNet are subsequently merged
into the 12 skip connections and the middle block of the U-net, thereby
serving as supplementary conditions for the generation process. Consequently,
when presented with a textual prompt $\mathcal{P}$ and an image condition $I$,
the computation of the score for classifier-free guidance [16] in multi-
conditions proceeds as follows:

$\begin{split}&\epsilon_{CFG}(z_{t},\mathcal{P},I)=\epsilon_{\theta}(z_{t},\emptyset,\emptyset)+\\\
&w\cdot(\epsilon_{\theta}(z_{t},\mathcal{P},I)-\epsilon_{\theta}(z_{t},\emptyset,\emptyset)),\end{split}$
(1)

where $z_{t}$ is the latent and $w$ is the interpolation coefficient which
improves the alignment of generated samples with their conditioning. The
output $y_{c}$ of the ControlNet $\phi$ can be simplified as follows:

$\begin{split}y_{c}=\sum_{j=1}^{J}\phi(z_{t},\mathcal{P},I;\theta_{b_{j}}),\end{split}$
(2)

where $J$ is the number of blocks in ControlNet and $\theta_{b_{j}}$ is the
parameter of the $j-th$ block.

During the training of the ControlNet, visual and textual controls are
perfectly alignment. However, When the prompt contains objects that are not
spatial compositions of the visual control, the generated result may fail to
depict these objects. To address this issue, we initially generate object mask
$\mathcal{OM}$, indicating the specific region in visual control that follows
the textual prompt. Then we apply this mask to guide the ControlNet [51] to
focus to this region, ensuring the extra prompt guidance such as a new object
beyond the region is effective.

Concretely, we calculated the union of all object masks and obtained
$\mathcal{OM}=(m_{1}\cup m_{2}...\cup m_{N})$. Then We recalculated $y_{c}$
with the below equation:

$\begin{split}y_{c}=\sum_{j=1}^{J}dm_{j}\cdot\phi_{2}(z_{t},\mathcal{P},I;\theta_{b_{j}}),\end{split}$
(3)

where $dm_{j}$ is the reshape form of $\mathcal{OM}$ and is adapted to the
resolution of the $j-th$ block. Through this approach, ControlNet [51]
selectively disregards features associated with misaligned visual elements
outside of $\mathcal{OM}$. This exclusion facilitates precise spatial control
by ControlNet [51] over specific image components and enables SD to generate
objects as per the misaligned text descriptions. Our experimental results show
that the application of masks to ControlNet features substantially mitigates
conflicts between mismatched textual and visual controls, effectively
addressing the problem of object missing in generated images

###  3.2 Attribute-Matching Loss

To improve attribute matching under visual controls, we employ various loss
functions that iteratively denoise and update the noise map $z_{t}$ at each
time step $t=(T,T-1,...,1)$ in the denoising process. We use spaCy’s
transformer-based dependency parser to analyze the text prompt $\mathcal{P}$,
extracting a set of attribute-object pairs
$S=\sum_{i=1}^{N_{o}}(a_{i},o_{i})$, where $N_{o}$ is the total number of
objects, and $o_{i}$ and $a_{i}$ denote the word indices of the $i-th$ object
and its corresponding attribute, respectively. The set $S$ is then divided
into two subsets: $s_{1}$ which aligns with the visual control containing $N$
objects, and $s_{2}$, corresponding to the remaining $N_{o}-N$ objects in the
misaligned portion of the prompt.

Our aim is to achieve semantic alignment between attribute words $a_{i}$ and
their corresponding objects $o_{i}$, while maintaining the integrity of the
image condition’s control.To this end, as shown in Fig. 3, we design two
specific attribute-matching-loss functions: Language-guided loss and Mask-
guided loss. These functions facilitate the alignment of $o_{i}$ and $a_{i}$
through cross-attention maps. Specifically, during the forward process, we can
obtain the cross-attention maps $A$ in SD and ControlNet named $A_{s}$ and
$A_{c}$ respectively. For the $i-th$ prompt token $p_{i}$, the cross-attention
map $A_{i}$ is calculated as follows:

$\begin{split}A_{i}=softmax(\frac{QK_{i}^{\top}}{\sqrt{d}}),\end{split}$ (4)

where the query $Q$ is derived from $z_{t}$, the key $K_{i}$ is derived from
the word embedding of $p_{i}$, and $d$ is the latent dimension of of $Q$ and
$K_{i}$. $A_{s}$ and $A_{c}$ indicate the relevance between the given text
prompt words and visual pixels. Moreover, $A_{c}$ extra indicates the
relevance between the visual control and visual pixels.

Our main idea is to maximize the overlap of the attentive areas of the
attribute-object pairs $(a_{i},o_{i})$ in $s$, on the condition of $I$ and
object masks $\mathcal{OM}$. To this end, we have formulated a loss function,
designated as $L_{I}$. This function refines the attribute cross-attention
maps in For pairs in $A_{s}$ and object cross-attention maps in $A_{c}$ for
pairs in subset $s_{1}$. The goal is to progressively align these maps more
closely, while distancing them from unrelated cross-attention maps,
represented as $\bar{A_{c}}$. For the pairs in subset $s_{2}$, our approach
primarily aligns their cross-attention maps within $A_{s}$ ensuring effective
semantic alignment.

$\begin{split}&L_{I}=\sum_{(a_{i},o_{i})\in
s_{1}}dist(A_{c}(o_{i}),A_{s}(a_{i}))\\\ &-\sum_{(a_{i},o_{i})\in
s_{1}}\sum_{w\in(a_{i},o_{i})}\sum_{A_{u}\in\bar{A_{c}}}\frac{1}{|\bar{A_{c}}|}dist(A_{u},A_{s}(w))\\\
&+\sum_{(a_{i},o_{i})\in s_{2}}dist(A_{s}(o_{i}),A_{s}(a_{i})),\end{split}$
(5)
$\begin{split}dist(A_{i},A_{j})=\frac{1}{2}D_{KL}(A_{i}||A_{j})+\frac{1}{2}D_{KL}(A_{j}||A_{i}),\end{split}$
(6)

where $dist$ is a measure function indicating the distance between the
attention maps and
$D_{KL}(A_{i}||A_{j})={\textstyle\sum_{pixels}}A_{i}log(A_{i}/A_{j})$.

Furthermore, to meet the condition $\mathcal{M}$, another loss termed
$L_{\mathcal{M}}$ is designed to constrain the target objects and attributes
attentive area to the predefined object masks.

$\begin{split}L_{\mathcal{M}}=-\sum_{i=1}^{N}\sum_{w\in
s_{1,i}}\sum_{m\in\mathcal{M}_{i}}(A_{s}(w,m)-A_{s}(w,\bar{m})),\end{split}$
(7)

where $A_{s}(w,m)$ is a value of a cross-attention map in $A_{s}$ for word $w$
in $s_{1,i}$ at pixel $m$ in the $i-th$ object mask.

Then, all the losses are aggregated and back-propagation is computed to update
the $z_{t}$ as follows:

$\begin{split}z_{t}{{}^{\prime}}\overset{}{\leftarrow}z_{t}-\alpha\nabla_{z_{t}}(l_{I},l_{\mathcal{M}}).\end{split}$
(8)

Algorithm 1 MGPF: Training-Free Prompt Following with Visual Controls

Input: A prompt $\mathcal{P}$, a visual control $I$, corresponding semantic
masks $\mathcal{M}$, the source and target initial latent noise maps
$z^{s}_{T}$ and $z_{T}$.
Output: Latent map $z^{s}_{0}$, updated latent map $z_{0}$ corresponding to
all the inputs.

1: $S=\\{(a_{i},o_{i})\\}_{i=1}^{N}\leftarrow\text{Prompt
Parser}(\mathcal{P})$

2: for $t=T,T-1,...,1$ do

3:
$\epsilon_{s},A_{s}\leftarrow\epsilon_{\theta}(z^{s}_{t},\mathcal{P},t),A_{c}\leftarrow\epsilon_{\phi}(z^{s}_{t},\mathcal{P},I,t)$;

4: $z^{s}_{t-1}\leftarrow\text{Sample}(z^{s}_{t},\epsilon_{s})$;

5: $z_{t}{{}^{\prime}}\leftarrow\text{MGPF}(A_{s},A_{c},\mathcal{M},S)$;

6: $\epsilon=\epsilon_{\theta,\phi}(z_{t}{{}^{\prime}},\mathcal{P},I,t)$;

7: $z_{t-1}\leftarrow\text{Sample}(z_{t},\epsilon)$;

8: end for

Return $z^{s}_{0},z_{0}$

Our algorithm can be summarized as Algorithm 1, which requires no training.

##  4 Experiments

Table 1: Quantitative results of all baseline models. We give results under
four conditions (soft edge, canny edge, depth, segmentation). VQA-F, VQA-B,
text-text, image-text represent LLM foreground VQA score, LLM background VQA
score, clip text similarity between input text prompt and blip captions, clip
image-text similarity respectively. Our method achieves the best results in
all metrics.

Method | Soft edge | Canny edge
---|---|---
| VQA-AM | VQA-OG | Text-text | Image-text | Aesthestic | VQA-AM | VQA-OG | Text-text | Image-text | Aesthetic
SD [38] | 0.6321 (-22.8%) | 0.4901 (-27.0%) | 0.5816 (-10.11%) | 0.2737 (-9.4%) | 5.24 (+0.9%) | 0.6561 (-23.2%) | 0.4600 (-30.9%) | 0.5778 (-11.5%) | 0.2738 (-10.3%) | 5.20 (+0.3%)
SD(mask) [38] | 0.5596 (-31.6%) | 0.4513 (-32.7%) | 0.5872 (-9.2%) | 0.2793 (-7.5%) | 5.24 (+0.9%) | 0.6423 (-24.8%) | 0.4917 (-26.2%) | 0.6151 (-5.8%) | 0.2878 (-5.7%) | 5.31 (+2.5%)
A&E [8] | 0.7136 (-12.8%) | 0.5316 (-20.8%) | 0.5624 (-13.1%) | 0.2717 (-10.0%) | 5.08 (-2.1%) | 0.7033 (-17.6%) | 0.4903 (-26.4%) | 0.5675 (-13.1%) | 0.2716 (-11.0%) | 5.02 (-3.1%)
BoxDiff [47] | 0.6426 (-21.5%) | 0.5104 (-23.9%) | 0.5655 (-12.6%) | 0.2717 (-10.0%) | 5.21 (+0.4%) | 0.6647 (-22.1%) | 0.4841 (-27.3%) | 0.5710 (-12.6%) | 0.2741 (-10.2%) | 5.15 (-0.6%)
Structure [11] | 0.6501 (-20.6%) | 0.4899 (-27.0%) | 0.5685 (-12.1%) | 0.2733 (-9.5%) | 5.24 (+0.9%) | 0.6535 (-23.5%) | 0.4594 (-31.0%) | 0.5767 (-11.7%) | 0.2730 (-10.5%) | 5.20 (+0.4%)
Linguistic [37] | 0.7556 (-7.7%) | 0.5171 (-23.0%) | 0.5854 (-9.5%) | 0.2786 (-7.8%) | 5.16 (-0.6%) | 0.7482 (-12.4%) | 0.4946 (-25.7%) | 0.5869 (-10.2%) | 0.2777 (-9.0%) | 5.15 (-0.6%)
Ours | 0.8186 | 0.6710 | 0.6470 | 0.3020 | 5.19 | 0.8537 | 0.6658 | 0.6532 | 0.3051 | 5.18

Method | Depth | Segmentation
---|---|---
| VQA-AM | VQA-OG | Text-text | Image-text | Aesthetic | VQA-AM | VQA-OG | Text-text | Image-text | Aesthetic
SD [38] | 0.6638 (-20.8%) | 0.5790 (-11.7%) | 0.5963 (-9.1%) | 0.2849 (-6.7%) | 5.30 (+1.1%) | 0.6265 (-20.3%) | 0.5613 (-16.6%) | 0.6051 (-7.8%) | 0.2870 (-5.3%) | 5.36 (+0.9%)
SD(mask) [38] | 0.5953 (-28.9%) | 0.4454 (-32.1%) | 0.5992 (-8.7%) | 0.2828 (-7.4%) | 5.27 (+0.6%) | 0.5910 (-24.8%) | 0.5159 (-23.3%) | 0.6135 (-6.6%) | 0.2896 (-4.4%) | 5.36 (+0.9%)
A&E [8] | 0.7150 (-14.6%) | 0.5960 (-9.2%) | 0.5775 (-12.0%) | 0.2817 (-7.8%) | 5.07 (-3.2%) | 0.7195 (-8.5%) | 0.5794 (-13.9%) | 0.6001 (-8.6%) | 0.2889 (-4.7%) | 5.22 (-1.7%)
BoxDiff [47] | 0.6556 (-21.7%) | 0.5986 (-8.8%) | 0.5863 (-10.7%) | 0.2831 (-7.3%) | 5.24 (0.0%) | 0.6574 (-16.4%) | 0.6007 (-10.7%) | 0.6043 (-8.0%) | 0.2879 (-5.0%) | 5.31 (0.0%)
Structure [11] | 0.6630 (-20.9%) | 0.5784 (-11.8%) | 0.5950 (-9.3%) | 0.2850 (-6.7%) | 5.30 (1.1%) | 0.6254 (-20.4%) | 0.5602 (-16.7%) | 0.6053 (-7.8%) | 0.2864 (-5.5%) | 5.35 (0.8%)
Linguistic [37] | 0.7477 (-10.7%) | 0.6031 (-8.1%) | 0.6042 (-7.9%) | 0.2869 (-6.1%) | 5.26 (0.4%) | 0.7734 (-1.6%) | 0.6703 (-0.4%) | 0.6493 (-1.1%) | 0.3004 (-0.9%) | 5.31 (0.0%)
Ours | 0.8376 | 0.6560 | 0.6562 | 0.3055 | 5.24 | 0.7861 | 0.6728 | 0.6565 | 0.3030 | 5.31

In this section, we evaluate the effectiveness of our method in enhancing
prompt following, focusing on object generation and attribute matching under
visual controls. In Sec. 4.1, we delve into the specifics of our
implementation details, the datasets, and evaluation metrics utilized in our
experiments. Additionally, we demonstrate the efficiency of our method with
both quantitative and qualitative results. Extensive experiments prove that
our method can significantly boost the semantic alignment in text-to-image
generation.

###  4.1 Evaluation Setup

Benchmarks. Considering the absence of established benchmarks for the
evaluation of object missing and attribute mismatch under diverse visual
controls, we construct a new benchmark. Our experiments involve four widely-
used image conditions: depth, canny edge, soft edge, and segmentation.
Following A&E’s approach [8, 37], we select images from the COCO dataset
including 80 object types categorized into general, fruit, and animal groups.
To compose our prompts, these objects are combined with 11 color-related
attributes. The prompts are constructed with 1 or 2 objects in the image and
randomly selected attributes. Additionally, to evaluate object generation
capabilities beyond visual controls, we incorporated textual prompts from the
COCO dataset into our prompts. Moreover, we consider two types of prompt: (1)
a [color A] [object A], [text prompt beyond visual control], and (2) a [color
A] [object A] and a [color B] [object B], [text prompt beyond visual control]
and yield a total of 1384 samples. Details of different objects, attributes,
and construction methods can be found in the supplementary material.

Evaluation Metrics. To quantitatively evaluate the performance of our method,
we follow existing works [8] by utilizing text-text similarity and CLIP [33]
image-text distances. Considering these two metrics are insufficient for
evaluating object generation beyond visual control and attribute matching, we
adopt the recently introduced VQA-based metric [18]. This metric use task-
related information as questions and evaluate the probability of a “yes”
response from our VQA model. For attribute matching, the question is designed
as “A [attribute] [object]?” for each pair in the text prompt, with the
cumulative “yes” probability serving as the metric. With respect to object
generation beyond visual control, the question is “[text prompt beyond visual
condition]?”, and its “yes” probability is also computed. We refer to these
metrics as VQA-AM (Attribute Matching) and VQA-OG (Object Generation),
respectively. Additionally, based on concerns that some attention based
methods might degrade image quality, we employ an aesthetic metric [41] to
evaluate the aesthetics of the generated images. We also conduct human
evaluations for further validation.

Baselines. We compare our method with several training-free methods that
intend to improve object generation and attributes matching. We re-implement
these methods integrated with ControlNet [51], including: Stable diffusion
[38], A&E [8], Structure [11], Lingustic [37], BoxDiff [47] and Stable
Diffusion&mask [38]. For BoxDiff, we calculate object bounding boxes from
their masks, as an additional input. In Stable Diffusion&mask, we utilize
object masks to directly modify visual controls, integrating these with
prompts for image generation.

###  4.2 Results

Figure 4: Qualitative comparison using prompts from our dataset. We show
images generated by all our baseline methods. We use the same seed across all
approaches.

Quantitative Results. We evaluated the effectiveness of every competing method
using the aforementioned text-text, text-image, VQA-AM, VQA-OG, and aesthetic
metrics. We demonstrate superior performance of our method across depth, soft
edge, canny edge, and segmentation conditions, as detailed in Tab. 1. For the
widely used text-text and image-text similarity metrics, our approach
significantly outperforms all baseline models in all four conditions,
demonstrating unparalleled prompt consistency. Considering that these two
metrics do not entirely capture the ability of attribute matching and object
generation beyond visual control, we further calculate the VQA-AM and VQA-OG
metrics. For VQA-AM, our method remains superior to all baselines.
Specifically, it outperforms the SD [38]+ControlNet and the most robust
Linguistic [37] baseline by 22.78% and 7.7% under the soft edge condition,
respectively. (23.15% and 12.36% for canny edge; 20.75% and 10.73% for depth;
20.30% and 1.62% for segmentation). This demonstrates the superiority of our
method to align attributes in the text prompt with corresponding objects in
visual controlled scenarios. Regarding VQA-OG, which reflects object
generation beyond visual control, our approach also achieved the best
performance, surpassing the strongest baseline by 22.94% under soft edge
conditions (25.71% for canny edge; 8.06% for depth; 0.37% for segmentation).
Additionally, to address concerns that cross-attention based methods might
affect image quality, we evaluate the aesthetic scores and find that all
baseline methods exhibit comparable aesthetic scores and our method do not
degrade the image quality.

Figure 5: Qualitative ablation results. MC, ML, and LL denote Masked
Controlnet, Mask Loss, and Language-Guided Loss respectively.

Qualitative Results. Fig. 4 illustrates a qualitative comparison of all
baseline methods. Our analysis focuses on two aspects: the quality of
attribute matching and the effectiveness of object generation beyond visual
control. Regarding attribute matching, we can see that most baseline methods
fail to achieve precise binding of attributes. While the Linguistic+ControlNet
[37] baseline occasionally facilitates attribute matching, it also produces
artifacts, owing to the attention map shifting for both attributes and objects
in the U-Net. In contrast, our method effectively binds attributes to their
corresponding objects while preserving image quality, demonstrating the
effectiveness of our attribute match losses. In terms of object generation
beyond visual control, our method accurately produces objects in the
misaligned portion of the prompt, outperforming other baselines. Notably,
these models fail to maintain consistency with the text prompt, even when
directly masking the misaligned part of the input visual control.

Table 2: Ablation studies of each module in our method, we conduct experiments
under canny edge condition and report text-text, image-text, VQA-AM, VQA-OG
metrics. ML: Mask-guided Loss, MC: Masked ControlNet, LL: Language-Guided
Loss.

Method | Canny edge
---|---
| VQA-AM | VQA-OG | Text-text | Image-text
Baseline | 0.6561 | 0.4600 | 0.5778 | 0.2738
+ML | 0.7715(+17.6%) | 0.4792 (+4.2%) | 0.5936 (+2.7%) | 0.2783 (+1.6%)
+MC | 0.5845(-10.9%) | 0.6139 (+33.5%) | 0.5817(+0.7%) | 0.2832(+3.4%)
+LL | 0.8453(+28.8%) | 0.5218(+13.4%) | 0.6067(+5.0%) | 0.2822(+3.1%)
+ML+LL |  0.8554(+30.4%) | 0.5175(+12.5%) | 0.6080(+5.2%) | 0.2828(+3.3%)
+ML+MC | 0.7791(+18.7%) | 0.6529(+42.0%) | 0.6367(+10.2%) | 0.3017(+10.2%)
+MC+LL | 0.8492(+29.4%) |  0.6738(+46.5%) | 0.6524(+12.9%) | 0.3040(+11.0%)
+MC+LL+ML | 0.8537(+30.1%) | 0.6658(+44.7%) |  0.6532(+13.1%) |  0.3051(+11.4%)

Ablation Study. Initially, we quantify the impact of each component within our
method as shown in Tab. 2. We perform experiments under the canny edge
condition and use SD [38]+ControlNet as the baseline. The incorporation of
Language-guided Loss (LL) and Mask-guided Loss (ML) improves the baseline
model across all metrics, particularly in VQA-AM, proving the effectiveness of
these losses in enhancing attribute matching. Notably, Masked ControlNet (MC)
improves the VQA-OG by a large margin(33.5%), but affected the VQA-AM
(decreasing by 10.9%). We observed that combining LL and ML achieves the best
results in VQA-AM, indicating superior attribute matching. Integrating LL with
MC, we obtain the highest VQA-OG score. Finally, with all three components,
our method achieves the balance results in both VQA-AM and VQA-OG metrics.
Additionally, Fig. 5 shows the result of progressively integrating our modules
into the base SD [38]+ControlNet model. The addition of MC improves object
generation in the latter part of the text prompt. For instance, in the second
row, the image distinctly features a tree in the park. Integrating LL enhances
attribute matching (e.g., a pink cat and black clock in the first row, and a
red bird in the second), though it introduced some artifacts in the generated
images, such as an additional block in the first case and red color artifacts
in the second. When further combining ML, we solve the artifacts, achieving
the best performance in both attribute matching and object generation.

Figure 6: Qualitative Comparison using ChilloutMix, we compare our method with
baseline model.

###  4.3 Robustness to Other Models: ChilloutMix

Additionally, we have cooperated our method to other popular models,
specifically ChilloutMix111https://civitai.com/models/6424. Fig. 6 shows the
visualization results of our approach and the SD [38]+ControlNet models with a
fixed random seed. Our method effectively generates objects that extend beyond
visual control and matches attributes to their corresponding objects. For
example, in the first column, our approach accurately binds the attribute
“blue” to the “remote”, while the baseline method fails. Moreover, our method
consistently produces better scenarios that more closely to prompts such as
“on the lawn in front of trees,” thereby demonstrating the generalizability of
our proposed approach. For more cases, please refer to the supplementary
material.

###  4.4 Human Evaluations

Table 3: Human Evaluation results compared with all baseline methods. AM, OG, VC and aesthetic represents attributes matching, object generation beyond visual control, visual control following and aesthetic respectively. Model | AM | OG | VC | aesthetic
---|---|---|---|---
Ours | 129 | 67 | 170 | 118
Linguistic [37] | 97 | 51 | 168 | 98
A&E [8] | 76 | 35 | 156 | 60
Structure [11] | 71 | 45 | 171 | 116
boxdiff [47] | 82 | 38 | 161 | 88
SD [38] | 72 | 42 | 172 | 111
SD(mask) [38] | 74 | 54 | 172 | 109
No winner | 38 | 102 | 2 | 17

To further validate our approach, we conduct a user study with images
generated by our method and compare them to those from various baseline
methods. survey respondents are asked to select the image set that best
aligned with the input prompt under visual controls, focusing on Attribute
Matching (AM), Object Generation Beyond Control (OG), Visual Controls
Following (VC), and Aesthetics. We randomly sample 200 prompts from our
constructed dataset, covering all four kinds of visual controls, and generate
images using all methods. The results, presented in Tab. 3, indicate the
number of selections for each method. For AM and OG, the majority of survey
respondents prefer the results produced by our method, thereby providing
additional evidence for its effectiveness. Regarding VC and Aesthetics, we can
see that our method neither disrupts the control of image conditions nor
degrades the aesthetic quality of the generated images. Further details are
available in the supplementary material.

##  5 Conclusions

In this work, we propose a training-free method for the prompt following under
visual control. To enhance object generation, we introduce object masks and
propose Masked ControlNet to handle the misalignment between the prompt and
visual control. To tackle the issue of attribute mismatch, we propose two
straightforward losses to align the cross-attentions between attribute and
object pairs, achieving effective attribute binding. Experimental results
demonstrate the superiority of our approach. Additionally, the integration of
our method with any Diffusion-based model expands its range of applications
and customization possibilities.

One limitation of our method is that in the more complex scenarios, such as
attribute match involving multiple or smaller objects under visual control, it
is challenging to accurately bind attributes to these objects. This may arise
from the low resolution of the current attention maps, limiting our capacity
to perform precise localized attribute binding. To alleviate this, we suggest
incorporating cross-attention also in higher-resolution layers. We will leave
this for future works.

## References

  * [1] Aishwarya Agarwal, Srikrishna Karanam, KJ Joseph, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan.  A-star: Test-time attention segregation and retention for text-to-image synthesis.  arXiv preprint arXiv:2306.14544, 2023.
  * [2] Omri Avrahami, Ohad Fried, and Dani Lischinski.  Blended latent diffusion.  ACM Transactions on Graphics (TOG), 42(4):1–11, 2023.
  * [3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  Nov 2022.
  * [4] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In European conference on computer vision, pages 707–723. Springer, 2022.
  * [5] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, and Yunxin Jiao.  Dall-e 3.  https://cdn.openai.com/papers/dall-e-3.pdf, 2023.
  * [6] John Canny.  A computational approach to edge detection.  IEEE Transactions on pattern analysis and machine intelligence, (6):679–698, 1986.
  * [7] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng.  Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing.  Apr 2023.
  * [8] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  ACM Transactions on Graphics (TOG), 42(4):1–10, 2023.
  * [9] Prafulla Dhariwal and Alexander Quinn Nichol.  Diffusion models beat gans on image synthesis.  In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 8780–8794, 2021.
  * [10] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2021.
  * [11] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Training-free structured diffusion guidance for compositional text-to-image synthesis.  arXiv preprint arXiv:2212.05032, 2022.
  * [12] Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel.  Scenescape: Text-driven consistent scene generation.  Feb 2023.
  * [13] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Tokenflow: Consistent diffusion features for consistent video editing.  Jul 2023.
  * [14] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [15] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [16] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [17] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner.  Text2room: Extracting textured 3d meshes from 2d text-to-image models.  Mar 2023.
  * [18] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu.  T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation.  arXiv preprint arXiv:2307.06350, 2023.
  * [19] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou.  Composer: Creative and controllable image synthesis with composable conditions.  arXiv preprint arXiv:2302.09778, 2023.
  * [20] Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva.  Divide & bind your attention for improved generative semantic nursing.  Jul 2023.
  * [21] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [22] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and JoshuaB. Tenenbaum.  Compositional visual generation with composable diffusion models.  Jun 2022.
  * [23] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia.  Video-p2p: Video editing with cross-attention control.  Mar 2023.
  * [24] Wan-DuoKurt Ma, J.P. Lewis, W.Bastiaan Kleijn, and Thomas Leung.  Directed diffusion: Direct control of object placement through attention guidance.  Feb 2023.
  * [25] Eyal Molad, Eliahu Horwitz, Dani Valevski, AlexRav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen.  Dreamix: Video diffusion models are general video editors.  Feb 2023.
  * [26] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  arXiv preprint arXiv:2302.08453, 2023.
  * [27] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [28] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  * [29] GeonYeong Park, Jeongsol Kim, Beomsu Kim, SangWan Lee, and JongChul Ye.  Energy-based cross attention for bayesian context update in text-to-image diffusion models.  Jun 2023.
  * [30] Ben Poole, Ajay Jain, JonathanT Barron, Ben Mildenhall, Google Research, and UC Berkeley.  Dreamfusion: Text-to-3d using 2d diffusion.
  * [31] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  Mar 2023.
  * [32] Sigal Raab, Inbal Leibovitch, Guy Tevet, Moab Arar, AmitH. Bermano, and Daniel Cohen-Or.  Single motion diffusion.  Feb 2023.
  * [33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [34] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.
  * [35] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  Cornell University - arXiv,Cornell University - arXiv, Feb 2021\.
  * [36] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE transactions on pattern analysis and machine intelligence, 44(3):1623–1637, 2020.
  * [37] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik.  Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment.  arXiv preprint arXiv:2306.08877, 2023.
  * [38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [39] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation, page 234–241.  Jan 2015.
  * [40] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [41] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * [42] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [43] Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture.  What the daam: Interpreting stable diffusion using cross attention.  arXiv preprint arXiv:2210.04885, 2022.
  * [44] Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao-Yuan Jing, Fei Wu, and Bingkun Bao.  Df-gan: A simple and effective baseline for text-to-image synthesis.
  * [45] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, AmitH. Bermano, and Daniel Cohen-Or.  Human motion diffusion model.  Sep 2022.
  * [46] Ruichen Wang, Zekang Chen, Chen Chen, Jian Ma, Haonan Lu, and Xiaodong Lin.  Compositional text-to-image synthesis with attention map control of diffusion models.
  * [47] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou.  Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7452–7461, 2023.
  * [48] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018.
  * [49] Shuai Yang, Yifan Zhou, Ziwei Liu, and ChenChange Loy.  Rerender a video: Zero-shot text-guided video-to-video translation.
  * [50] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang.  Cross-modal contrastive learning for text-to-image generation.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2021.
  * [51] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.