# Attention Calibration for Disentangled Text-to-Image Personalization

Yanbing Zhang1,2, Mengping Yang1,2, Qin Zhou1,2∗, Zhe Wang1,2∗
1 Department of Computer Science and Engineering, ECUST, China
2 Key Laboratory of Smart Manufacturing in Energy Chemical Process, ECUST,
China
{zhangyanbing<EMAIL_ADDRESS>{sunniezq,
<EMAIL_ADDRESS>
###### Abstract

Recent thrilling progress in large-scale text-to-image (T2I) models has
unlocked unprecedented synthesis quality of AI-generated content (AIGC)
including image generation, 3D and video composition. Further, personalized
techniques enable appealing customized production of a novel concept given
only several images as reference. However, an intriguing problem persists: Is
it possible to capture multiple, novel concepts from one single reference
image? In this paper, we identify that existing approaches fail to preserve
visual consistency with the reference image and eliminate cross-influence from
concepts. To alleviate this, we propose an attention calibration mechanism to
improve the concept-level understanding of the T2I model. Specifically, we
first introduce new learnable modifiers bound with classes to capture
attributes of multiple concepts. Then, the classes are separated and
strengthened following the activation of the cross-attention operation,
ensuring comprehensive and self-contained concepts. Additionally, we suppress
the attention activation of different classes to mitigate mutual influence
among concepts. Together, our proposed method, dubbed DisenDiff, can learn
disentangled multiple concepts from one single image and produce novel
customized images with learned concepts. We demonstrate that our method
outperforms the current state of the art in both qualitative and quantitative
evaluations. More importantly, our proposed techniques are compatible with
LoRA and inpainting pipelines, enabling more interactive experiences.

Figure 1:  Given one _individual_ image from specific users, our proposed
method is capable of producing _customized_ images for each concept contained
in the input image, _e.g.,_ given a single input image with a man and a woman,
our method excels in achieving innovative renditions of both combined (_left_)
and independent (_right_) concepts, without compromising the fidelity and
identity preservation, and more importantly, manifesting satisfactory
interactive generation conditioned by various text prompts. Note that we
employ notation $V_{i}^{*}$ to denote the modifier of the $i$-th concept. Our
code and data will be publicly available at:
https://github.com/Monalissaa/DisenDiff.

11footnotetext: Corresponding author

##  1 Introduction

Recently developed large-scale text-to-image models [40, 42, 2, 38] have shown
unprecedented capabilities in synthesizing high-quality and diverse images
based on a target text prompt. Built on these models, personalized techniques
[12, 41] are further introduced to customize the models for synthesizing
personal concepts with sufficient fidelity.

Given as input just a few images of the personal concepts (e.g., family,
friends, pets, or individual objects), personalized text-to-image models aim
to learn a new word embedding to represent a specific concept [52, 46].
However, existing methods still lack the flexibility to render all existing
concepts in a given image, or only focus on a specific concept [13, 26]. Given
a unique photo from a user (which could be people rarely seen together or
uncommon furniture pieces), with multiple concepts occurring in the complex
scene, the user naturally desires the ability to freely synthesize the
concepts by composing multiple objects or focusing on only one of them. For
example, two specific individuals at a beach, or alternatively, one of them in
Times Square, as shown in Fig. 1.

Figure 2: Failure case of Custom Diffusion [24]. In the third column, we show
the example encompassing two failure settings: appearance inconsistency with
the input image and ambiguous object not included in the target text. In the
second column, we show the result from our method.

To achieve flexible renditions of the concepts, instead of using a single new
word to represent one concept [24, 20], we employ multiple new words to
represent multiple concepts. For example, considering an image containing a
distinct chair and lamp (as shown in Fig. 2), we utilize the prompt
“$V_{1}^{*}$ chair and $V_{2}^{*}$ lamp” to distinguish between them, with
“$V_{1}^{*}$” serving as the modifier for “chair” and “$V_{2}^{*}$” as the
modifier for “lamp”. This intuitive formulation poses two key challenges.
Firstly, the new word embeddings are likely to map confusing information,
failing to maintain visual-fidelity to the target concepts. Secondly, with a
relatively small training set (e.g., only one image), the model is prone to
synthesizing multiple subjects, even when the target prompt pertains to a
single concept. For example, as depicted in Fig. 2, the ideal output should
exclusively feature the specified lamp when the target text is “A $V_{2}^{*}$
lamp”. Nonetheless, the image generated by the current state-of-the-art model
not only includes a lamp that doesn’t match the color and texture of the input
image but also involves a chair that shouldn’t be present.

In this paper, we propose a novel personalized T2I model, referred to as
DisenDiff (i.e., Disentangled Diffusion), to address the above-mentioned
issues. To preserve the good generalization ability in pre-trained large-scale
models, we follow [24, 46] to only update the light-weight modules ($W_{K}$
and $W_{V}$ matrices) within the cross-attention units along with new token
embeddings to extend concepts. Our key insight is that current methods lack
the necessary guidance for the optimization process, resulting in cluttered
attention maps (as shown in Fig. 4, the first row). Consequently, existing
methods struggle to synthesize each concept effectively.

Based on the above observations, we strive to generate precise attention maps
from the following two aspects. Building on the discovery that the attention
map of the class token can roughly align with the location of the concept,
then we propose a modifier-class alignment term to bind the attention map of
each new modifier with its corresponding class token, correcting attention to
focus on the region of the related concept. However, the attention maps of
different class tokens often exhibit overlaps, leading to the incorrect
attribute binding [5] and mutual entanglement. To achieve effective
decoupling, we introduce the separate and strengthen (s&s) strategy to allow
flexibly synthesizing each concept independently. By minimizing the
overlapping regions between the attention maps of different class tokens, we
can effectively mitigate the co-occurring issue when targeting at a specific
concept. To further enhance the independence of concepts, we introduce a
suppression technique to sharpen the boundaries of class tokens’ attention
maps. Our contributions are summarized below:

  * •

We propose DisenDiff to comprehend multiple personal concepts from only a
single image. By using diverse target texts, it can render
combined/independent concepts in imaginary contexts while preserving high
fidelity to the input image.

  * •

We employ two key constraints to attain precise attention maps for crucial
tokens. The binding constraint locates new modifiers to different concepts,
while the s&s constraint decouples these concepts.

  * •

We conduct experiments on various datasets and demonstrate that our method
outperforms the current state of the art in quantitative and qualitative
aspects. Additionally, we show the flexibility of our approach by applying it
to extended tasks.

##  2 Related Work

Text-to-image generative models. The objective of text-to-image (T2I) tasks
[58, 30] is to generate an image corresponding to a given textual description.
Thanks to large-scale datasets [43, 4] and advancements in language models
[22, 35, 36], T2I models have witnessed remarkable progress. While Generative
adversarial networks (GANs) [39, 55, 21, 28] and autoregressive (AR)
transformers [37, 54, 9, 11] have delivered impressive results, diffusion
models [8, 17] have taken the lead in T2I generation. These models employ
denoising processes in image space [32, 42, 2, 18, 51] or latent space [38,
40, 14], resulting in unprecedented image generation quality. However, they
encounter challenges when generating specific objects, such as custom
furniture, even with detailed prompts. We aim to augment these models to
accurately capture the appearances of novel concepts from real-world images.

Text-guided image editing. With the surge of powerful T2I models, numerous
studies have delved into enhancing the controllability of diffusion models to
cater to diverse user demands. Approaches such as [10, 5, 49] refine the
cross-attention units to encompass all subject tokens, motivating the model to
fully convey the semantics in the input prompt. Techniques like [33, 27, 6]
implement region control in T2I generation by using bounding boxes and paired
object labels as inputs. Additionally, [56] and [50] harness pre-trained
diffusion models for image-to-image translation. A substantial body of work
also focuses on local or global modifications of single images using existing
T2I models. Notable examples include SINE [57] and UniTune [47], which achieve
image editing by fine-tuning the diffusion model. Other methods like prompt-
to-prompt [15], null-text inversion [31], and [34] impose constraints on
latent noise during inference time without model training. While our
objectives share some common ground with these methods, our primary focus is
optimizing the model to seamlessly extend personalized concepts into new
prompts.

T2I personalization Personalization techniques adapt diffusion models to learn
new concepts from user-provided images, often relying on a small dataset of
3-5 images or even a single image. Textual Inversion [12] uses pseudo-words to
represent new concepts through a visual reconstruction objective. To leverage
semantic priors from pre-trained models, DreamBooth [41] utilizes a unique
identifier and class name within the input text to represent new concepts.
Custom-Diffusion [24] and Perfusion [46] compose multiple new concepts by
updating only the cross-attention Keys and Values along with new token
embeddings. When working with a dataset containing just a single image,
current methods [52, 20, 13, 26] typically begin with additional domain-
specific pre-training on a large dataset before adapting to the new concept.
In contrast to these methods, we aim to address the more challenging problem
of acquiring multiple concepts from a single image without domain-specific
pre-training. Recently, break-a-scene [1] tackles the similar task using a
two-phase customization method. However, its approach necessitates an extra
input of an object mask, while our approach solely processes the reference
image.

Figure 3: Method overview. Our method applies constraints to the cross-
attention maps of crucial tokens, ensuring the accurate representation of
multiple concepts. We introduce new modifiers, denoted as $V_{i}^{*}$, along
with the $i$-th class name, to represent the $i$-th personalized concept. Our
attention calibration mechanism mainly includes three parts: the suppression
technique performs self-sharpening and filters noisy small patches, the
$\mathcal{L}_{\text{bind}}$ loss steers new modifiers towards the
corresponding classes, and the $\mathcal{L}_{\text{s\&s}}$ loss guarantees the
independence and completeness of the learned concepts. Figure 4: Comparison of
generated attention maps and images. The first row displays the results of
Custom Diffusion [24], while the second row shows our results. During the
training stage, when we obtain accurate attention maps for important tokens
(left), it leads to the ideal output during the inference stage (right),
maintaining high-concept similarity with the input image.

##  3 Method

Our objective is to understand multiple concepts within a single image. To
this end, we propose a novel attention calibration mechanism to help generate
accurate cross-attention maps in our T2I model. Firstly, the cross-attention
maps are calculated as the activation responses between each word of the input
text and the intermediate visual features. Then, we impose constraints on the
cross-attention maps between both the modifier-class token pairs and class-
class token pairs to bind the cross-attention maps of each modifier with its
corresponding class (modifier-class constraint), as well as to ensure full
comprehension of each class and separation between different classes (class-
class constraint). To further mitigate the cross-interference issue in our T2I
model, we introduce a suppression technique to obtain a sharper attention map
for each class token. A schematic workflow of our method is presented in Fig.
3.

###  3.1 Preliminary

Stable Diffusion. In our experiments, we use Stable Diffusion [44] as our
backbone model, inheriting the structure of the Latent Diffusion Model (LDM)
[40]. It primarily consists of three components: a pre-trained text encoder
$\tau_{\theta}$ from CLIP [35], a VAE [23] model $\mathcal{E}$, and a U-Net
diffusion model $\epsilon_{\theta}$ trained on the latent space $z$ of the
pre-trained VAE. Given the noisy latent code $z_{t}$ at $t$ timestep, the
diffusion model predicts the random added noise $\epsilon$. The training
objective of the diffusion model is formulated as follows:

$\mathbb{E}_{\mathcal{E}(x),y,\epsilon\sim\mathcal{N}(0,1),t}\left[\left\|\epsilon-\epsilon_{\theta}\left(z_{t},t,\tau_{\theta}(y)\right)\right\|_{2}^{2}\right],$
(1)

where $x$ denotes the input image, $y$ is the input text. Following [40],
prior knowledge in CLIP is integrated via the cross-attention mechanism.

Integrating textual features via cross-attention. Formally, the intermediate
spatial representation $\phi(z_{t})$ of the denoiser U-Net is mapped to a
query matrix $Q=W_{Q}\cdot\phi(z_{t})$, while text embeddings
$\tau_{\theta}(y)$ are mapped to a key matrix $K=W_{K}\cdot\tau_{\theta}(y)$
and a value matrix $V=W_{V}\cdot\tau_{\theta}(y)$, using learnable projection
matrices $W_{Q}$, $W_{K}$, and $W_{V}$. Then, the cross-attention maps are
obtained as:

$A_{t}=\operatorname{Softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right),$ (2)

where $d$ is the projection dimension of keys $K$ and queries $Q$. Here,
$A_{t}\in\mathbb{R}^{r\times r\times N}$, $r$ is the spatial dimension of the
$\phi(z_{t})$, and $N$ is the number of input tokens. The updated spatial
representations integrating text priors are then obtained as
$\phi(z_{t})=A_{t}V$, as illustrated in Fig. 3.

Text encoding. Generally, during training of a T2I system, a suitable text
prompt is required in addition to the selected single image. In this paper, we
adopt a manner similar to [41], incorporating new modifiers and the classes to
be modified into the input text. For example, if the target image contains a
cat and a dog, the text prompt would be “$V_{1}^{*}$ cat and $V_{2}^{*}$ dog”.
The modifier tokens “$V_{i}^{*}$” are initialized with rare vocabulary. Given
only a single training image, the T2I model will likely lack the diversity of
generation, known as the language drift [25, 29] problem. Using our text
prompt, we can easily select regularized images with the same caption to
mitigate the issue of language drift, enabling our model to generate a variety
of cats and dogs (not limited to the ones present in the target images, as
shown in Fig. 5, left of the second row).

Current methods are prone to overfitting when the training data only consists
of a single image, resulting in ambiguous attention maps for each token (as
shown in the first row of Fig. 4). As demonstrated in P2P [15], the spatial
layout and geometry of the generated images depend on the cross-attention
maps. Therefore, our primary focus is to optimize the model to produce
accurate cross-attention maps, elaborated in the following part.

###  3.2 Coherent binding of modifiers with classes

Based on the cross-attention maps ($A_{t}$) obtained by a previous method
(shown in Fig. 4, the first row), we can observe that while $A_{t}$ of new
modifiers are chaotic ($A_{t}^{m_{1}}$ and $A_{t}^{m_{2}}$), cross-attention
of class tokens can roughly capture the semantic boundaries ($A_{t}^{c_{1}}$
and $A_{t}^{c_{2}}$). We attribute it to the fact that the majority of
parameters in the T2I models are frozen, preserving the category information
of class tokens. To aid the new modifiers in understanding their
responsibilities, we define the constraint to bind the cross-attention maps of
modifiers with their corresponding class tokens as

$\mathcal{L}_{\text{bind}}\left(A_{t}^{m_{i}},A_{t}^{c_{i}}\right)=1-\frac{A_{t}^{m_{i}}\cap
A_{t}^{c_{i}}}{A_{t}^{m_{i}}\cup A_{t}^{c_{i}}},$ (3)

where $A_{t}^{m_{i}}$ and $A_{t}^{c_{i}}$ represent the attention map of the
$i$-th modifier and the $i$-th class at $t$ timestep, respectively. The
$\mathcal{L}_{\text{bind}}$ loss is formulated to reduce the intersection over
union (IoU) [53] between these two attention maps, encouraging a close
alignment between the activations of the modifiers and the class tokens. To
prevent substantial influence on $A_{t}^{c_{i}}$, we detach its gradient
during the loss computation.

Nonetheless, there are two potential issues when we directly apply this
constraint. Given that $A_{t}$ is the result of the Softmax operation (i.e.,
${\textstyle\sum_{i=1}^{N}A_{t}^{i}(h,w)}=1$, where $A_{t}^{i}(h,w)$ denotes
the activation of the $i$-th token at pixel $(h,w)$), input tokens would
contend for attention at the same position. Consequently, a precise pixel-to-
pixel correspondence between $A_{t}^{m_{i}}$ and $A_{t}^{c_{i}}$ can not be
established. Furthermore, our intention is for the activations of
$A_{t}^{m_{i}}$ to fully encompass the corresponding object, thereby capturing
all its attributes comprehensively. However, as depicted in Fig. 4, it is
evident that within the object region, certain activations of $A_{t}^{c_{2}}$
exhibit high values, while others appear considerably lower. This poses a
challenge for the attention $A_{t}^{m_{i}}$ to sustain a comprehensive focus
on the object. To address these challenges, we employ a Gaussian filter on
$A_{t}$, which leads to the generation of smooth attention maps referred to as
$G(A_{t})$. This smoothing process helps to alleviate the pixel-wise
competition among tokens and facilitates more comprehensive attention to the
object. Consequently, by using the loss function
$L_{\text{bind}}\left(G(A_{t}^{m_{i}}),G(A_{t}^{c_{i}})\right)$, we encourage
$A_{t}^{m_{i}}$ to have coherent attention areas with $A_{t}^{c_{i}}$, while
achieving a broader coverage of the object, without the need for precise
point-to-point binding. For simplicity, in the subsequent sections of this
paper, unless explicitly specified otherwise, we apply a Gaussian filter to
$A_{t}$.

###  3.3 Separating and strengthening attention maps for multiple classes

Figure 5: Qualitative results of independent (left) and combined (right)
concepts. The target prompt in each row represents a distinct context
including learned concepts. Our method shows the highest visual similarity to
the input image compared to Custom Diffusion and DreamBooth (especially in the
first row, the results containing the specific toy) while preserving robust
editability. Additionally, we show the ability to address the language drift
issue and the disentanglement capability on the left of the second and last
row, respectively.

Given a single image as the training set, it’s inevitable for one class token
to attend to multiple concepts simultaneously. For instance, in the first row
of Fig. 4, specifically in $A_{t}^{c_{1}}$, the attention dedicated to the
“cat” token is not solely limited to the “cat” concept. It also exhibits some
degree of attention towards the “dog” concept. Thus, $A_{t}^{m_{1}}$
incorporates attributes associated with the “dog” concept due to its binding
with $A_{t}^{c_{1}}$. To ensure independent editing of concepts without
interference, it is necessary to separate the attention regions of different
objects (i.e., $A_{t}^{c_{i}}$ and $A_{t}^{c_{j}}$). A straightforward
approach is to minimize the overlap between attention maps of different object
tokens as

$\mathcal{L}_{\text{separate}}\left(A_{t}^{c_{i}},A_{t}^{c_{j}}\right)=A_{t}^{c_{i}}\cap
A_{t}^{c_{j}}.$ (4)

The utilization of $\mathcal{L}_{\text{separate}}$ effectively prevents the
activations of class tokens from overlapping. However, it may come with a side
effect of reducing the area of $A_{t}^{c_{i}}$, potentially leading to a loss
of identity for the corresponding class, which can be found in the supplement.
To simultaneously minimize the overlap among attention maps and preserve the
class identity, we design the following constraint,

$\mathcal{L}_{\text{s\&s}}\left(A_{t}^{c_{i}},A_{t}^{c_{j}}\right)=\frac{A_{t}^{c_{i}}\cap
A_{t}^{c_{j}}}{A_{t}^{c_{i}}\cup A_{t}^{c_{j}}},$ (5)

where “s&s” stands for “separate and strengthen.” The
$\mathcal{L}_{\text{s\&s}}$ loss strikes a balance between avoiding overlap
with other objects and ensuring comprehensive coverage of the target object,
thus improving the accuracy and fidelity of the attention mechanism.

Suppression. The utilization of the $\mathcal{L}_{\text{s\&s}}$ loss can
potentially lead to another issue where the attention map $A_{t}^{c_{1}}$
captures a significant portion of the activations, while $A_{t}^{c_{2}}$
exhibits very few activations. This imbalance in activation distribution
between different class tokens can result in an uneven emphasis on certain
classes. To address it, we introduce a suppression mechanism. Specifically,
before computing the $\mathcal{L}_{\text{s\&s}}$, we apply an element-wise
multiplication operation to $A_{t}^{c_{i}}$ (i.e.,
$f_{m}(A_{t}^{c_{i}})=A_{t}^{c_{i}}\odot A_{t}^{c_{i}}$). Given that
activations fall within the range of $[0,1]$, $f_{m}(A_{t}^{c_{i}})$ filters
out activations that are less important for the class. As a result, the loss
$L_{\text{s\&s}}(f_{m}(A_{t}^{c_{i}}),f_{m}(A_{t}^{c_{j}}))$ is designed to
separate and strengthen their attentions, preventing encroachment upon other
classes from within its own boundaries. Additionally, $A_{t}^{m_{i}}$ can be
bound with a more distinct $A_{t}^{c_{i}}$.

In summary, the total training loss is formulated as:

$\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L}_{\text{base}}+\sum_{i=1}^{\mathcal{S}}\mathcal{L}_{\text{bind}}\left(G(A_{t}^{m_{i}}),f_{m}(G(A_{t}^{c_{i}}))\right)$
(6)
$\displaystyle+\sum_{i=1}^{\mathcal{S}}\sum_{j=i+1}^{\mathcal{S}}\mathcal{L}_{\text{s\&s}}\left(f_{m}(G(A_{t}^{c_{i}})),f_{m}(G(A_{t}^{c_{j}}))\right),$

where $\mathcal{S}$ is the number of classes in the input image, and
$\mathcal{L}_{\text{base}}$ is the base loss of the T2I model in Eq. 1.
$\mathcal{L}_{\text{s\&s}}$ is responsible for refining the attention maps
related to class tokens, while $\mathcal{L}_{\text{bind}}$ is responsible for
constraining new modifier tokens to acquire correct attributes. The auxiliary
functions $G(\cdot)$ and $f_{m}(\cdot)$ facilitate the optimization process.
The synergy among these constraints results in the generation of precise and
interpretable attention maps for input tokens, shown in the second row of Fig.
4.

##  4 Experiments

###  4.1 Experimental Settings

Datasets. We conducted experiments on ten datasets spanning a large range of
categories including people, animals, furniture, and people with pets/toys.
Please note, instead of concentrating only on one concept, our datasets
contain two distinct concepts within each image. During the inference phase,
we test 30 different prompts for each image: 10 for combined concepts, 10
specifically targeting the first concept, and 10 focusing on the second
concept.

Figure 6: Quantitative evaluation results. (a) Compared to state-of-the-art
methods, our approach (green) achieves the highest image-alignment score,
particularly noticeable in Concept2, while maintaining a text-alignment score
similar to that of other methods. (b) Ablation study results. Our full method
(green) strikes the best balance between reconstruction and editability.

Compared methods. We compare with three personalized T2I methods, which all
utilize new word embeddings to represent novel concepts. (1) Textual Inversion
(TI): In TI, only the new token embedding representing the novel concept is
updated, while the other parameters remain frozen. (2) DreamBooth (DB): DB
updates all layers of the T2I model to maintain visual fidelity and employs a
prior preservation loss to mitigate language drift. (3) Custom-Diffusion (CD):
CD updates the most relevant weights related to the input textual features,
including $W_{K}$ and $W_{V}$ within the cross-attention units, as well as the
new token embedding. The implementation details are provided in the
supplement.

Evaluation metrics. The synthetic images should faithfully capture the visual
characteristics of the input image while accurately conveying all elements of
the target text. We employ two key metrics: (1) The image-alignment metric
evaluates the reconstruction of concepts, which measures the pairwise CLIP-
space cosine similarity [12] between the generated images and the
corresponding real images. (2) The text-alignment metric assesses the editing
effectiveness of the fine-tuned model by calculating the text-image similarity
between the generated images and the provided prompts using CLIP [16].
Notably, these two indicators often conflict with each other [46]. For each
concept, we synthesize 16 samples per prompt, using 50 DDIM steps and a
guidance scale of 6. For comparison, we provide scores for combined concepts,
the first concept, the second concept, and their average (referred to as
Combined, Concept1, Concept2, and Mean in Fig. 6). For instance, if the
training image caption is “$V_{1}^{*}$ cat and $V_{2}^{*}$ dog”, the test
prompts of the Combined, Concept1 and Concept2 settings are “$V_{1}^{*}$ cat
and $V_{2}^{*}$ dog in a garden”, “$V_{1}^{*}$ cat wearing a hat”, “A pink
$V_{2}^{*}$ dog”, respectively. When testing on independent concepts, we
calculate the image-alignment metric between the synthesized images and the
segmented image containing only the corresponding subject.

Implementation details. We fine-tune the Stable Diffusion [44] model for 250
steps, with a batch size of 8 and a learning rate of $8\times 10^{-5}$.
Similar to [24], we employ clip-retrieval [3] to select $200$ samples from
LAION-5B [43] dataset as regularization images. Captions of these selected
images exhibit a similarity of over 0.85 in the CLIP textual embedding space
with the input text. Meanwhile, we use the data augmentation in [24]. In our
experiments, we apply the proposed cross-attention calibration to the
$16\times 16$ attention units, which have been shown to contain the most
semantic information [15].

###  4.2 Comparison Results

Quantitative comparisons. Fig. 6a illustrates the results averaged across ten
datasets. As shown, we outperform all the compared methods, especially on the
image-alignment scores. Specifically, despite Textual Inversion (TI) achieving
the highest text-alignment score, it has the lowest image-alignment score,
indicating its struggle to maintain the appearance of concepts. DreamBooth
(DB) outperforms TI in image-alignment score but falls significantly short
compared to our approach in both metrics. Custom Diffusion (CD) maintains a
better balance between the two metrics and competes with ours in combined
concepts scores and Concept1 scores. However, there is a noticeable
performance gap in the scores for Concept2. In summary, we achieve the highest
image fidelity while maintaining strong text editing effectiveness. Detailed
results for each dataset can be found in the supplement.

Qualitative comparisons. We visually demonstrate the favorable outcomes in
Fig. 5. Concretely, we design diverse target prompts to assess the learned
independent concepts and combined concepts in different editing scenarios,
including scene changes, object addition, style transfer, property change,
accessory addition, interactions between multiple concepts, concept
decoupling, and the ability to address the language drift (e.g., generating a
specific cat consistent with the input and a dog with a breed distinct from
the one present in the input). As shown in Fig. 5, images synthesized by DB
either lack key attributes of the concepts or suffer from severe overfitting
to the input image. With most of its parameters frozen, CD improves
editability and reconstruction compared to DB. However, it still struggles to
preserve concepts’ appearances or decouple from the input image, especially as
shown in the first and last rows of Fig. 5. By incorporating cross-attention
calibration, our method achieves high visual fidelity and maintains effective
cross-concept disentanglement during T2I generation. For the sake of space
efficiency, additional results including Textual Inversion are provided in the
supplement.

###  4.3 Ablation Studies

We conduct ablation studies to show the effectiveness of each component and
analyze the influence of different design choices, adopting the same setup
described in Sec. 4.1.

To assess the necessity of each component, we set up the following experiment
settings: (1) Removing the $\mathcal{L}_{\text{s\&s}}$ loss, (2) removing the
$\mathcal{L}_{\text{bind}}$ loss, (3) removing the suppression strategy, (4)
removing the Gaussian filter, (5) applying twice suppression (in contrast to
one-time). Detailed results are presented in Fig. 6b. As shown, our full model
achieves a balanced performance between visual fidelity and editing
effectiveness for both combined and independent concepts. Removing either the
$\mathcal{L}_{\text{bind}}$ or $\mathcal{L}_{\text{s\&s}}$ loss results in a
significant decrease in image-alignment for both Concept1 and Concept2.
Similarly, the removal of the Gaussian filter leads to a notable reduction in
image-alignment for combined concepts. No suppression significantly harms
image-alignment for Concept2, confirming the benefits of sharper boundaries in
$A_{t}^{s_{i}}$ for understanding multiple concepts (as explained in Sec.
3.3). Meanwhile, this also leads to lower text-alignment for both Concept1 and
Concept2. Furthermore, applying twice suppression has detrimental effects on
image-alignment as it filters out important information.

Figure 7: Applications in image inpainting. Given an input image and its
corresponding mask, our method can seamlessly inpaint the learned concepts
into the masked region. Figure 8: Integrating with LoRA [19]. Our method can
incorporate the LoRA parameters to fully convey the semantics (e.g., enhancing
texture details).

On the other hand, there are two design choices worth considering. As
indicated in [45], averaging all scales of attention layers, instead of just
using the $16\times 16$ scale, could potentially yield improved attribution
maps for each input word. Therefore, we explore (1) impose constraints on the
average of all scales attention layers. Additionally, we investigate releasing
more parameters, specifically (2) updating the $W_{Q}$, $W_{K}$, and $W_{V}$
matrices within the cross-attention units (in contrast to our approach, which
only updates the $W_{K}$ and $W_{V}$). As depicted in Fig. 6b, operating on
all scales of attention layers resulted in the model’s inability to
reconstruct Concept2. Updating $W_{Q}$, $W_{K}$, and $W_{V}$ does help the
model remember the appearances of concepts but leads to a significant decrease
in text-alignment. This suggests that updating more parameters does not
preserve the good features of the pre-trained model.

###  4.4 Applications

Personalized concept inpainting. With any image and its corresponding mask,
our method can seamlessly integrate learned concepts into the masked region
while preserving the rest of the image, as shown in Fig. 7. Users can
effortlessly perform inpainting by simply modifying the text prompt, thanks to
our method’s conversion of concepts into new word embeddings.

Compatible with LoRA [19]. LoRA techniques, actively discussed in the
community, such as CivitAI [7], have gained popularity for enhancing specific
capabilities of T2I models, such as improving the ability to refine images.
LoRA adds small, trainable parameters to the frozen T2I models for fine-
tuning, and our method is orthogonal with it. Therefore, we combine the LoRA
with our trained model to unlock a wider range of applications, as shown in
Fig. 8. This combination is akin to domain-specific pre-training on a large
dataset before personalization [26, 13], with the added benefit of having
access to a wealth of readily available LoRA parameters in the community.

Figure 9: Applications in extending three concepts. Enabling edits on three
concepts within a single image.

Extending to three concepts. We explore the application of our method to the
more challenging task of capturing three concepts from a single image, as
shown in Fig. 9. In this scenario, we employ the $\mathcal{L}_{\text{s\&s}}$
loss for each pair of the three class tokens to disentangle these concepts.

##  5 Conclusions and Limitations

We propose the DisenDiff to mimic multiple concepts from a single image. We
introduce constraints on the cross-attention units to attain precise attention
maps for crucial tokens, mitigating the overfitting to the single image and
accurately capturing concept appearances. Consequently, our method enables
diverse edits involving combined or independent concepts while enhancing the
visual similarity between the synthesized images and the input image.
Furthermore, we show the flexibility of our method by evaluating several
applications.

Limitations. Disentangling fine-grained categories becomes notably challenging
when two subjects from the same category co-exist in a single image, such as
Golden Retriever and Border Collie dogs. Additionally, while our method can
handle images with three concepts, its performance degrades considerably. This
can be attributed to the limitations of existing T2I models in such scenarios,
as well as the need for algorithm adjustments to address these specific
challenges. We believe that there is considerable room to enhance the
performance in these complex tasks.

Acknowledgment. This work is supported by Shanghai Science and Technology
Program ”Federated based cross-domain and cross-task incremental learning”
under Grant No. 21511100800, Natural Science Foundation of China under Grant
No. 62076094 and No. 62201341.

## References

  * Avrahami et al. [2023] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski.  Break-a-scene: Extracting multiple concepts from a single image.  In _SIGGRAPH Asia 2023 Conference Papers_ , New York, NY, USA, 2023\. Association for Computing Machinery.
  * Balaji et al. [2022] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  _arXiv preprint arXiv:2211.01324_ , 2022.
  * Beaumont [2022] Romain Beaumont.  Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them.  https://github.com/rom1504/clip-retrieval, 2022.
  * Byeon et al. [2022] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim.  Coyo-700m: Image-text pair dataset.  https://github.com/kakaobrain/coyo-dataset, 2022.
  * Chefer et al. [2023] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–10, 2023.
  * Chen et al. [2023] Minghao Chen, Iro Laina, and Andrea Vedaldi.  Training-free layout control with cross-attention guidance.  _arXiv preprint arXiv:2304.03373_ , 2023.
  * Civitai [2022] Civitai.  Civitai.  https://civitai.com/, 2022.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Ding et al. [2022] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang.  Cogview2: Faster and better text-to-image generation via hierarchical transformers.  _Advances in Neural Information Processing Systems_ , 35:16890–16902, 2022.
  * Feng et al. [2022] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Training-free structured diffusion guidance for compositional text-to-image synthesis.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Gafni et al. [2022] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman.  Make-a-scene: Scene-based text-to-image generation with human priors.  In _European Conference on Computer Vision_ , pages 89–106. Springer, 2022.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Gal et al. [2023] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–13, 2023.
  * Gu et al. [2022] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10696–10706, 2022.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or.  Prompt-to-prompt image editing with cross-attention control.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7514–7528, 2021.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Ho et al. [2022] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  _Journal of Machine Learning Research_ , 23(47):1–33, 2022.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Jia et al. [2023] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su.  Taming encoder for zero fine-tuning image customization with text-to-image diffusion models.  _arXiv preprint arXiv:2304.02642_ , 2023.
  * Kang et al. [2023] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.  Scaling up gans for text-to-image synthesis.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Kenton and Toutanova [2019] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186, 2019.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1931–1941, 2023.
  * Lee et al. [2019] Jason Lee, Kyunghyun Cho, and Douwe Kiela.  Countering language drift via visual grounding.  _arXiv preprint arXiv:1909.04499_ , 2019.
  * Li et al. [2023a] Dongxu Li, Junnan Li, and Steven CH Hoi.  Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing.  _arXiv preprint arXiv:2305.14720_ , 2023a.
  * Li et al. [2023b] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22511–22521, 2023b.
  * Li et al. [2022] Zhiheng Li, Martin Renqiang Min, Kai Li, and Chenliang Xu.  Stylet2i: Toward compositional and high-fidelity text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18197–18207, 2022.
  * Lu et al. [2020] Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin.  Countering language drift with seeded iterated learning.  In _International Conference on Machine Learning_ , pages 6437–6447. PMLR, 2020.
  * Mansimov et al. [2016] Elman Mansimov, Emilio Parisotto, Lei Jimmy Ba, and Ruslan Salakhutdinov.  Generating images from captions with attention.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , 2016.
  * Mokady et al. [2023] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6038–6047, 2023.
  * Nichol et al. [2022] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  GLIDE: towards photorealistic image generation and editing with text-guided diffusion models.  In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , pages 16784–16804. PMLR, 2022\.
  * Park et al. [2022] Dong Huk Park, Grace Luo, Clayton Toste, Samaneh Azadi, Xihui Liu, Maka Karalashvili, Anna Rohrbach, and Trevor Darrell.  Shape-guided diffusion with inside-outside attention.  _arXiv preprint arXiv:2212.00210_ , 2022.
  * Patashnik et al. [2023] Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or.  Localizing object-level shape variations with text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022\.
  * Reed et al. [2016] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.  Generative adversarial text to image synthesis.  In _International conference on machine learning_ , pages 1060–1069. PMLR, 2016.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Stable diffusion [2022] Stable diffusion.  https://huggingface.co/runwayml/stable-diffusion-v1-5, 2022.
  * Tang et al. [2022] Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture.  What the daam: Interpreting stable diffusion using cross attention.  _arXiv preprint arXiv:2210.04885_ , 2022.
  * Tewel et al. [2023] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon.  Key-locked rank one editing for text-to-image personalization.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 1–11, 2023\.
  * Valevski et al. [2023] Dani Valevski, Matan Kalman, Eyal Molad, Eyal Segalis, Yossi Matias, and Yaniv Leviathan.  Unitune: Text-driven image editing by fine tuning a diffusion model on a single image.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–10, 2023.
  * von Platen et al. [2022] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf.  Diffusers: State-of-the-art diffusion models.  https://github.com/huggingface/diffusers, 2022.
  * Wang et al. [2023] Ruichen Wang, Zekang Chen, Chen Chen, Jian Ma, Haonan Lu, and Xiaodong Lin.  Compositional text-to-image synthesis with attention map control of diffusion models.  _arXiv preprint arXiv:2305.13921_ , 2023.
  * Wang et al. [2022a] Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen.  Pretraining is all you need for image-to-image translation.  _arXiv preprint arXiv:2205.12952_ , 2022a.
  * Wang et al. [2022b] Zhe Wang, Qida Dong, Wei Guo, Dongdong Li, Jing Zhang, and Wenli Du.  Geometric imbalanced deep learning with feature scaling and boundary sample mining.  _Pattern Recognition_ , 126:108564, 2022b.
  * Wei et al. [2023] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  _arXiv preprint arXiv:2302.13848_ , 2023.
  * Yu et al. [2016] Jiahui Yu, Yuning Jiang, Zhangyang Wang, Zhimin Cao, and Thomas Huang.  Unitbox: An advanced object detection network.  In _Proceedings of the 24th ACM international conference on Multimedia_ , pages 516–520, 2016.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2(3):5, 2022\.
  * Zhang et al. [2021] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang.  Cross-modal contrastive learning for text-to-image generation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 833–842, 2021.
  * Zhang et al. [2023a] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023a.
  * Zhang et al. [2023b] Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris N Metaxas, and Jian Ren.  Sine: Single image editing with text-to-image diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6027–6037, 2023b.
  * Zhu et al. [2007] Xiaojin Zhu, Andrew B Goldberg, Mohamed Eldawy, Charles R Dyer, and Bradley Strock.  A text-to-picture synthesis system for augmenting communication.  In _AAAI_ , pages 1590–1595, 2007.

Supplementary Material

##  6 Experiments

Additional qualitative results. Further comparisons, including Textual
Inversion (TI) [12], are illustrated in Figure 11 (independent concepts) and
Figure 12 (combined concepts). Evidently, the concepts synthesized by TI
differ significantly from the input image, affirming the quantitative analysis
in Sec. 4.2.

Detailed quantitative results on ten datasets. As shown in Tab. 1, our method
consistently attains the highest image-alignment across most datasets while
maintaining favorable text-alignment compared to the three baselines.

Attention map visualization of ablation studies. The attention maps for the
component ablations are presented in Fig. 13, encompassing the following
scenarios: (1) Removing the $\mathcal{L}_{\text{bind}}$ loss, (2) removing the
$\mathcal{L}_{\text{s\&s}}$ loss, (3) using $\mathcal{L}_{s}$ (i.e.,
$\mathcal{L}_{\text{separate}}$ in Sec. 3.3) instead of
$\mathcal{L}_{\text{s\&s}}$, (4) removing the suppression strategy, (5)
applying twice suppression, (6) removing the Gaussian filter. Observing Fig.
13 reveals the following insights: (1) Without $\mathcal{L}_{\text{bind}}$,
new modifiers tend to focus on incorrect classes or vague regions; (2) Absence
of $\mathcal{L}_{\text{s\&s}}$ results in interdependence among learned class
tokens, especially the “cat” token; (3) Sole reliance on $\mathcal{L}_{s}$
leads to tiny activation areas for crucial tokens; (4) Removal of the
suppression strategy introduces unnecessary activations for new modifiers,
apart from their corresponding class regions; (5) Applying twice suppression
causes the loss of vital information for new modifiers, (e.g., the attention
of $V_{2}^{*}$ is obviously smaller than the “dog”); (6) The absence of the
Gaussian filter may cause new modifiers to lack specific attributes related to
the concepts, such as the attention on the mouth part for $V_{2}^{*}$ in the
specific dog instance. In summary, our full method generates independent and
comprehensive attention maps for crucial tokens.

##  7 Implementation and Experiment Details

Figure 10: Overview of ten datasets.

Datasets. We present each training image in Fig. 10.

Textual Inversion [12]. We utilized the implementation from [48] with 5000
training steps, a batch size of 4, and a learning rate of 0.0005. The input
prompt, originally “A photo of $V^{*}$” in Textual Inversion, is modified to
“A photo of $V_{1}^{*}$ and $V_{2}^{*}$”. The two new words ($V_{1}^{*}$ and
$V_{2}^{*}$) are initialized with the classes from the input image. For
example, if the image contains a cat and a dog, $V_{1}^{*}$ and $V_{2}^{*}$
token embeddings are initialized as the pre-trained “cat” and “dog” token
embeddings.

DreamBooth [41]. We employ the implementation from [48] with 250 training
steps, a batch size of 2, and a learning rate of $5\times 10^{-6}$. The input
prompt is “$V_{1}^{*}$ [class1] and $V_{2}^{*}$ [class2]”, consistent with our
setting in Sec. 3.1. Additionally, we generate 1000 “a [class1] and a
[class2]” images using the pre-trained model [41]. New modifiers are
initialized as rare token embeddings.

Custom Diffusion [24]. We employ the official implementation with 250 training
steps, a batch size of 8, and a learning rate of $8\times 10^{-5}$. The input
prompt is also “$V_{1}^{*}$ [class1] and $V_{2}^{*}$ [class2]”, and modifiers
are also initialized as rare token embeddings. For regularization, 200 images
are selected using clip-retrieval [3] with the caption “a [class1] and a
[class2]”. We apply the default data augmentation in Custom Diffusion.

DisenDiff (ours). Implementation details are described in Sec. 4.1. For the
total loss in Eq. 6, the weight of $\mathcal{L}_{\text{bind}}$ is set to
$0.01$ in all experiments. The weight of $\mathcal{L}_{\text{s\&s}}$ defaults
to $0.01$ and occasionally adjusts to $0.001$ for specific cases.

| Method | Cat+Dog | Cow+Bird | Man+Woman | Chair+Vase | Chair+Lamp | Dog+Pig | Mother+Child | Woman+Dog | Horse+Dog | Baby+Toy | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---
| Textual Inversion | 0.732 | 0.656 | 0.550 | 0.649 | 0.663 | 0.662 | 0.557 | 0.541 | 0.636 | 0.607 | 0.625
| DreamBooth | 0.732 | 0.701 | 0.606 | 0.815 | 0.784 | 0.701 | 0.601 | 0.625 | 0.708 | 0.689 | 0.696
| Custom Diffusion | 0.808 | 0.777 | 0.719 | 0.811 | 0.798 | 0.771 | 0.705 | 0.706 | 0.747 | 0.775 | 0.762
2-13  Image-alignment (Mean) | Ours | 0.824 | 0.783 | 0.749 | 0.822 | 0.795 | 0.773 | 0.718 | 0.737 | 0.744 | 0.808 | 0.775
| Textual Inversion | 0.802 | 0.815 | 0.739 | 0.814 | 0.834 | 0.834 | 0.764 | 0.776 | 0.811 | 0.767 | 0.796
| DreamBooth | 0.804 | 0.816 | 0.738 | 0.732 | 0.811 | 0.830 | 0.768 | 0.781 | 0.817 | 0.778 | 0.788
| Custom Diffusion | 0.773 | 0.843 | 0.731 | 0.759 | 0.794 | 0.793 | 0.740 | 0.754 | 0.818 | 0.800 | 0.780
2-13  Text-alignment (Mean) | Ours | 0.774 | 0.847 | 0.727 | 0.757 | 0.800 | 0.794 | 0.744 | 0.732 | 0.826 | 0.796 | 0.780
| Textual Inversion | 0.743 | 0.690 | 0.527 | 0.687 | 0.659 | 0.662 | 0.572 | 0.542 | 0.620 | 0.644 | 0.634
| DreamBooth | 0.736 | 0.774 | 0.679 | 0.897 | 0.845 | 0.697 | 0.672 | 0.684 | 0.784 | 0.739 | 0.751
| Custom Diffusion | 0.856 | 0.843 | 0.801 | 0.914 | 0.903 | 0.793 | 0.777 | 0.807 | 0.801 | 0.820 | 0.832
2-13  Image-alignment (Combined) | Ours | 0.865 | 0.855 | 0.828 | 0.909 | 0.883 | 0.794 | 0.795 | 0.835 | 0.792 | 0.870 | 0.843
| Textual Inversion | 0.797 | 0.805 | 0.738 | 0.800 | 0.816 | 0.839 | 0.797 | 0.777 | 0.823 | 0.811 | 0.800
| DreamBooth | 0.780 | 0.823 | 0.762 | 0.705 | 0.799 | 0.824 | 0.815 | 0.821 | 0.843 | 0.823 | 0.799
| Custom Diffusion | 0.736 | 0.882 | 0.719 | 0.698 | 0.747 | 0.749 | 0.735 | 0.729 | 0.826 | 0.792 | 0.761
2-13  Text-alignment (Combined) | Ours | 0.747 | 0.896 | 0.708 | 0.711 | 0.767 | 0.772 | 0.746 | 0.712 | 0.842 | 0.805 | 0.771
| Textual Inversion | 0.756 | 0.688 | 0.527 | 0.671 | 0.669 | 0.682 | 0.501 | 0.463 | 0.647 | 0.648 | 0.625
| DreamBooth | 0.763 | 0.697 | 0.545 | 0.755 | 0.795 | 0.707 | 0.520 | 0.554 | 0.707 | 0.679 | 0.672
| Custom Diffusion | 0.818 | 0.768 | 0.661 | 0.750 | 0.803 | 0.779 | 0.661 | 0.635 | 0.761 | 0.802 | 0.744
2-13  Image-alignment (Concept1) | Ours | 0.837 | 0.766 | 0.674 | 0.766 | 0.804 | 0.771 | 0.651 | 0.678 | 0.752 | 0.808 | 0.751
| Textual Inversion | 0.798 | 0.845 | 0.732 | 0.830 | 0.840 | 0.822 | 0.729 | 0.738 | 0.815 | 0.771 | 0.792
| DreamBooth | 0.823 | 0.800 | 0.722 | 0.768 | 0.792 | 0.818 | 0.724 | 0.715 | 0.819 | 0.775 | 0.779
| Custom Diffusion | 0.776 | 0.856 | 0.733 | 0.812 | 0.805 | 0.777 | 0.688 | 0.718 | 0.819 | 0.831 | 0.781
2-13  Text-alignment (Concept1) | Ours | 0.776 | 0.856 | 0.732 | 0.809 | 0.809 | 0.774 | 0.693 | 0.677 | 0.822 | 0.826 | 0.777
| Textual Inversion | 0.695 | 0.590 | 0.596 | 0.589 | 0.660 | 0.642 | 0.598 | 0.619 | 0.640 | 0.528 | 0.616
| DreamBooth | 0.696 | 0.632 | 0.594 | 0.795 | 0.711 | 0.699 | 0.612 | 0.635 | 0.635 | 0.650 | 0.666
| Custom Diffusion | 0.748 | 0.721 | 0.696 | 0.769 | 0.688 | 0.741 | 0.675 | 0.675 | 0.678 | 0.702 | 0.709
2-13  Image-alignment (Concept2) | Ours | 0.770 | 0.729 | 0.744 | 0.790 | 0.699 | 0.754 | 0.708 | 0.697 | 0.688 | 0.747 | 0.733
| Textual Inversion | 0.812 | 0.796 | 0.747 | 0.817 | 0.847 | 0.842 | 0.766 | 0.812 | 0.794 | 0.719 | 0.795
| DreamBooth | 0.809 | 0.794 | 0.729 | 0.725 | 0.843 | 0.848 | 0.765 | 0.808 | 0.790 | 0.737 | 0.785
| Custom Diffusion | 0.808 | 0.792 | 0.742 | 0.767 | 0.830 | 0.853 | 0.797 | 0.815 | 0.808 | 0.775 | 0.799
2-13  Text-alignment (Concept2) | Ours | 0.799 | 0.787 | 0.741 | 0.752 | 0.823 | 0.836 | 0.792 | 0.807 | 0.814 | 0.757 | 0.791

Table 1: Quantitative comparison on each dataset. Evaluation metrics are
outlined in Section 4.1 (higher is better for both metrics). We report four
types of scores (Mean, Combined, Concept1, Concept2), and the averaged results
across ten datasets are illustrated in Figure 6. The term “Cat+Dog” signifies
the presence of both “Cat” and “Dog” concepts within the dataset. Figure 11:
Qualitative comparison on independent concepts including Textual Inversion.
Figure 12: Qualitative comparison on combined concepts including Textual
Inversion. Figure 13: Attention map visualization of ablations.  Each row
represents the generated image and attention maps for all input tokens by
ablation methods.


# Edge-oriented Implicit Neural Representation with Channel Tuning

Wonjoon Chang Contributed equally Dahee Kwon11footnotemark: 1 Bumjin
Park11footnotemark: 1

###### Abstract

Implicit neural representation, which expresses an image as a continuous
function rather than a discrete grid form, is widely used for image
processing. Despite its outperforming results, there are still remaining
limitations on restoring clear shapes of a given signal such as the edges of
an image. In this paper, we propose Gradient Magnitude Adjustment algorithm
which calculates the gradient of an image for training the implicit
representation. In addition, we propose Edge-oriented Representation Network
(EoREN) that can reconstruct the image with clear edges by fitting gradient
information (Edge-oriented module). Furthermore, we add Channel-tuning module
to adjust the distribution of given signals so that it solves a chronic
problem of fitting gradients. By separating backpropagation paths of the two
modules, EoREN can learn true color of the image without hindering the role
for gradients. We qualitatively show that our model can reconstruct complex
signals and demonstrate general reconstruction ability of our model with
quantitative results.

††footnotetext: Preprint

##  1 Introduction

Implicit Neural Representation learns a function, neural network in most
cases, which takes pixel coordinates as input and output the corresponding RGB
pixel values. The implicit problem formulation takes form:

$F(\mathbf{x},\Phi,\nabla_{\mathbf{x}}\Phi,\nabla^{2}_{\mathbf{x}}\Phi,\cdots)=0,~{}~{}\Phi:\mathbf{x}\mapsto\Phi(\mathbf{x})$
(1)

where $\Phi$ is the function we are interested and $F$ is the relation between
$\mathbf{x}$ and $\Phi$. For example, a single image data can be represented
as $D=\\{(\mathbf{x}_{i},f(\mathbf{x}_{i})\\}$ where $\mathbf{x}_{i}$ is pixel
coordinates associated with their RGB colors $f(\mathbf{x}_{i})$.
$\Phi(\mathbf{x})$ can be enforced to output RGB colors and minimize
$||f(\mathbf{x})-\Phi(\mathbf{x})||^{2}$. The goal of the implicit problem is
to find the function $\Phi$.

Representing image as a continuous function can have benefits in terms of
efficiency and flexibility which makes Implicit Neural Representation be
widely used in image reconstruction and super resolution task. For example,
Sitzmann et al. [18] achieved breakthrough performance on image reconstruction
by using implicit representation with sine activation. However, it does not
imply implicit neural representation methods can reconstruct images perfectly.

Figure 1: First row (top) is an original image and the corresponding gradient
values. Second row shows the result from SIREN model trained with pixel
values. We enlarge some points to clearly show the detailed differences.

In Figure 1, we show some drawbacks of SIREN, proposed on [18]. On the second
row of Figure 1, the overall reconstruction seems quite plausible, but you can
see that local features such as eyes and the logo on the jacket are not well
caught. This phenomenon is more pronounced in the gradient visualization. In
other words, RGB target based learning is lack of edge reconstruction while
edge target based learning is lack of color reconstruction. Yet, the above two
are not trade-off properties but tasks that must be considered simultaneously.
Thus, in this paper, we propose edge-oriented implicit representation with
color-tuning architecture (EoREN) not only detect the edge clearly but also
reproduce the color well. EoREN is supervised in a direction that gradients
trained with implicit function become close to true gradients, which
prioritizes fitting gradients rather than pixel values. In the process, it is
required to approximate the true gradient values because we couldn’t directly
access to the ground truth gradients. Generally, directional filters such as
sobel filter are widely used which amplify the total magnitude of the
gradients. Since the enlargement of gradients makes the model be trained with
wrong target values, we propose gradient magnitude adjustment (GMA) that
allows the model to learn in a more accurate direction. In addition, for more
accurate color reconstruction, channel tuning module that adjusts the scale of
the pixel values are added after gradient fitting.

##  2 Related Work

Implicit neural representation. Implicit neural representation is a task of
learning a function which puts position of the data as inputs and outputs the
signal of the position. Recently, many researchers try to adopt implicit
neural representation into various deap learning fields [6, 13] such as 3D
reconstruction [19, 7, 1]. The Multi Layer Perceptron (MLP) is mostly used to
train the implicit function with ReLU activation. However, Sitzmann et al.
[18] pays attention to the fact that MLP with ReLU activation is ineffective
to represent the signal data and suggests SIREN model which uses periodic
activation function that can learn the representation with image, audio, and
video signal well. Also, instead of training separate neural network for each
image, more generalized way is proposed with feature representation of images
[4]. Chen et al. [3] suggests Local Implicit Image Representation and shows a
way to train continuous representation learning method in super resolution
task. However, none of these approaches examines the effect of the complexity
of images when training implicit representation function. Different from these
works, our work focuses on the capability of the implicit neural
representation.

Gradient-based edge detection. Edge detection is an important step that
becomes the cornerstone of image processing for object detection and
recognition. Traditionally, gradient information was mainly used for edge
detection [17, 10]. The gradient of the image refers to amount of changes from
a low brightness value to a high value or vice versa. Therefore, finding
pixels with large gradients can be seen as finding the boundaries of the
objects in the image. The most commonly used gradient-based edge detection
method is [2] called Canny edge detection, which uses Gaussian filtering and
the Sobel mask filter. Canny detection method is simple and effective, yet has
the disadvantages of time-consuming and parameter adjusting issue. In
addition, processing complex images with only edge lacked information.

Along with these limitations, traditional methods have been overshadowed by
the popular deep learning methods such as CNN for a while. [21] utilizes the
feature maps of intermediate layer for edge detection and [11] uses generative
adversarial network for super-resolution task. Recently, the Neural Implicit
Representation based models [14, 18] boost the performance of image
reconstruction and super-resolution task by considering continuous
representation space. Even though these deep learning based methods could
process images very well with their super-complex structure, there still exist
some challenges to be solved in the point of the exact image reproduction. One
of them is to recognize the edge of objects and restore them clearly. So in
order to overcome the performance bottleneck, many studies applied the
gradient-based edge detection method to deep neural network. [12] uses the
gradient information to improve the performance of the existing super-
resolution method [11]. And in addition, various techniques such as [16] uses
gradient-based edge detection method to improve the performance of image
restoration. We also focus on improving the reconstruction performance of the
existing neural implicit presentation by utilizing the gradient information.

##  3 Background

###  3.1 Signal Fitting with Implicit Representations

Implicit neural representations learn implicitly defined functions to model
many different types of signals such as image, video, and audio processing in
a continuous, memory-efficient way. The following equation is a general form
of loss to train the implicit function $\Phi$ to represent the signal $f$ on
$\Omega$:

$\mathcal{L}=\int_{\Omega}\|\Phi(\mathbf{x})-f(\mathbf{x})\|d\mathbf{x}.$ (2)

It turns out that sinusoidal representation networks (SIRENs) can effectively
learn representing complex natural signals by using periodic activation
functions such as sin activation functions [18]. SIRENs converge significantly
faster than existing networks that use other activation functions as well as
feature high image fidelity. It shows that reconstructed images from SIRENs
have more realistic image properties so that their gradients and laplacians
are close to the ground truth.

###  3.2 Poisson Equation

A supervised neural network is generally trained to provide an output that is
close to the ground truth. Implicit neural representations also focus on
learning implicit functions of given signals. However, reconstructed images
from SIRENs that are trained to fit given images may not reflect complex
patterns such as edges in a logo. It is because there is no supervision to
learn exact edges in images and the network is optimized to fit average color-
level of pixels.

The implicit neural representation can also be supervised solely by
derivatives of a given signal [18]. That is, the model is never provided with
the actual function values. The Poisson equation is the simplest elliptic
partial differential equation (PDE) that requires to fit the first or higher-
order derivatives of a given signal. In this problem, an unknown ground truth
signal $f$ is estimated from discrete samples of either is gradients $\nabla
f$ or Laplacian $\Delta f=\nabla\cdot\nabla f$ as

$\mathcal{L}_{\textrm{grad.}}=\int_{\Omega}\|\nabla_{\mathbf{x}}\Phi(\mathbf{x})-\nabla_{\mathbf{x}}f(\mathbf{x})\|d\mathbf{x},$
(3)

$\mathcal{L}_{\textrm{lapl.}}=\int_{\Omega}\|\Delta_{\mathbf{x}}\Phi(\mathbf{x})-\Delta_{\mathbf{x}}f(\mathbf{x})\|d\mathbf{x}.$
(4)

Solving the Poisson equation enables the reconstruction of images from their
derivatives so that the implicit function represents edges of images much
better. Actually, a way to fit gradients with SIREN for a RGB image has not
been directly described in the original paper. We train SIRENs to minimize the
difference between automatically computed gradients and true gradients
calculated by sobel filters for each RGB channel.

Figure 1 shows one example of the results. SIREN trained to fit gradients can
describe complex patterns such as the logo in the jacket and clearly
reconstruct eyes of the astronaut. However, there is remaining intensity
variations due to the ill-posedness of the problem.

###  3.3 Sobel Filter

In the SIREN paper, the Sobel filter was used to calculate the ground truth
gradient. The Sobel filter is used in image processing and computer vision,
particularly within edge detection algorithms where it creates an image
emphasizing edges. Technically, it is a discrete differentiation operator,
computing an approximation of the gradient of the image intensity function.
The Sobel filter is based on convolving the image with a 3x3 filter in the
horizontal and vertical direction.

$\mathbf{G}_{x}=\begin{bmatrix}-1&0&1\\\ -2&0&2\\\
-1&0&1\end{bmatrix},\quad\mathbf{G}_{y}=\begin{bmatrix}-1&-2&-1\\\ 0&0&0\\\
1&2&1\end{bmatrix}$ (5)

$\mathbf{G}_{x}$ denotes the Sobel filter for the horizontal direction and
$\mathbf{G}_{y}$ denotes the Sobel filter for the vertical direction. The
gradient approximation that the Sobel filter produces is relatively crude, in
particular for high-frequency variations in the image.

Figure 2: Induced problem of the Sobel filter. (a) shows that the gradients
computed by convolving the Sobel filter can be overestimated compared to the
original gradient in the simple example. (b) shows that the reconstructed dog
image from SIREN trained with these estimated gradients has extremely large
differences at the edges of the image.

##  4 Method

Figure 3: The structure of EoREN. Each pixel $(x,y)$ of input image $f$ goes
through 1) Edge-oriented module to learn the edge property of the image and 2)
channel tuning module to learn the distribution of the pixel values. The loss
of Edge-oriented module is computed with the gradient $\nabla f(x,y)$ on pixel
$(x,y)$ of the image. The loss of the channel tuning module is computed with
RGB value $f(x,y)$. The red dashed line indicates backpropagation.

###  4.1 Gradient Magnitude Adjustment

Even though the Sobel filter is widely used as a computationally efficient way
to approximate the gradients of images, it is problematic to feed the
estimated gradients as the ground truth to the model. Figure 2 describes which
problem can be induced from the Sobel filter. Consider the simple case where
the gradient value is a constant $a$ in the horizontal direction. When we
apply convolution with the Sobel filter, the estimated gradient is $8a$. That
is, the Sobel filter overestimate the gradients. To fit the overestimated
gradient values, the model sacrifice the correct pixel values such as the dog
image in Figure 2 (b). Even a little contrast exists in the original input
image, the difference of the values in the edge is overestimated and the
reconstructed image becomes very unrealistic.

To address this issue, we propose to adjust the magnitude of gradients
computed by filter for realistic image reconstruction. We define the magnitude
of the filter as the total sum of the absolute values used in the filter. For
instance, the Sobel filter, described in Figure 2, has magnitude 8 which means
that the estimated gradient values are 8 times larger than the actual gradient
values. Therefore, we need to offset the gap between the estimated gradients
and the actual gradients.

We consider two main components to feed proper gradient values to the model.
The first one is the magnitude of the filter which is introduced previously,
since it leads to overestimation. The second thing is the size of the image.
Let $W^{T},H^{T}$ be the range of the input coordinates and $W,H$ be the width
and height of the image respectively. Note that convolving process assumes
that the size of the interval between adjacent pixels is 1. However, SIREN
uses the different range of input coordinates. As a result, the derivatives of
the implicit function with respect to input may have different scale.
Therefore, we need to multiply the gradients by
$\frac{W}{W_{T}},\frac{H}{H_{T}}$, after normalizing it by filter magnitude.
Then, we can feed adjusted gradient values to the model for solving the
Poisson Equation.

Algorithm 1 Gradient with Magnitude Adjustment

1:procedure GMA($I_{W\times H}$, ${W_{T},H_{T}}$ , $F^{x}$, $F^{y}$)

2: $M_{F}^{x}\leftarrow|F^{x}|$ $\triangleright$ compute filter magnitude

3: $M_{F}^{y}\leftarrow|F^{y}|$

4: $F^{x}\leftarrow F^{x}/M_{F}^{x}$ $\triangleright$ normalize by filter
magnitude

5: $F^{y}\leftarrow F^{y}/M_{F}^{y}$

6: $F^{x}\leftarrow F^{x}\times\frac{W}{W_{T}}$ $\triangleright$ normalize by
image size

7: $F^{y}\leftarrow F^{y}\times\frac{H}{H_{T}}$

8: $grad_{x}\leftarrow F^{x}\circledast I_{W\times H}$ $\triangleright$
convolution

9: $grad_{y}\leftarrow F^{y}\circledast I_{W\times H}$

10: return $(grad_{x},grad_{y})$

Figure 4: The implicit representation of the RGB images in skimage data with
SIREN-pixel, SIREN-grad, EoREN without the channel tuning module and EoREN.

###  4.2 EoREN

We add one more additional module to help our model correctly represent pixel
values while keeping complex signals by gradient adjustment. We propose Edge-
oriented Representation Network (EoREN) that can represent complex patterns in
the signal with avoiding the ill-posedness problem through the channel tuning
module.

$\Phi(\mathbf{x})=(h\circ g)(\mathbf{x})$ (6)

EoREN is composed of two functions $g$ and $h$. $g$ represents original
implicit representation function of the signal and $h$ is an additional shift
transformation for each dimension of outputs.

$\displaystyle g(\mathbf{x})=\mathbf{W}_{n}(g_{n-1}\circ
g_{n-2}\circ\cdots\circ g_{0})(\mathbf{x})+\mathbf{b}_{n}$ (7) $\displaystyle
g_{i}:\mathbf{x}_{i}\mapsto\mathrm{Activation}(\mathbf{W}_{i}\mathbf{x}_{i}+\mathbf{b}_{i})$
(8)

Here, $g_{i}:\mathbb{R}^{M_{i}}\rightarrow\mathbb{R}^{N_{i}}$ is the $i^{th}$
layer of the network with the weight matrix
$\mathbf{W}_{i}\in\mathbb{R}^{N_{i}\times M_{i}}$ and the biases
$\mathbf{b}_{i}\in\mathbb{R}^{N_{i}}$. The function $g$ is a MLP that acts as
an implicit function of the signal. It can be SIREN with the use of sine
activation functions.

The objective of our implicit function is not to optimize the values of the
signal $f$, but to match the gradients of the signal. Our implicit function
$g$ is trained to solve the Poisson equation for fitting true gradients
$\nabla_{\mathbf{x}}f(\mathbf{x})$ for each coordinate $\mathbf{x}\in\Omega$.
That is, it minimizes the loss described in Equation 3.

$\mathcal{L}_{\mathrm{grad.}}=\int_{\Omega}||\nabla_{\mathbf{x}}g(\mathbf{x})-\nabla_{\mathbf{x}}f(\mathbf{x})||d\mathbf{x}$
(9)

We call this implicit function Edge-oriented module. The Edge-oriented module
can be easily supervised to learn complex shapes in a given images. However,
since it is optimized only to fit $\nabla_{\mathbf{x}}f(\mathbf{x})$, $g$ may
not follow the actual signal $f$. In the case of image fitting, $g$ may not
represent true colors of the image $f$. Therefore, we need to calibrate the
distribution of reconstructed images.

We propose an additional shift transformation layer, called channel tuning
module, to adjust differences between the output and the target signal. Figure
3 shows the overall architecture of our model. $\mathbf{\alpha}$ and
$\mathbf{\beta}$ are vectors of trainable parameters.

$h(\mathbf{x})=\alpha*g(\mathbf{x})+\beta$ (10)

It is updated to transform the output of the Edge-oriented module so that it
fit the true values of the signal $f$. The loss to optimize $\alpha$ and
$\beta$ are as follows:

$\tilde{\mathcal{L}}=\int_{\Omega}||f(\mathbf{x})-\Phi(\mathbf{x})||d\mathbf{x}$
(11)

Note that we do not backpropagate the loss in Equation 11 to optimize $g$.
When we set $\alpha=\mathbf{1}$, even if we add the additional channel tuning
module, the derivatives of EoREN do not change due to the chain rule, i.e.,
$\nabla\Phi=\nabla g$. Theoretically, it is enough to set $\alpha=\mathbf{1}$
and only train $\beta$ parameter to fit correct pixel values. However, it has
been empirically observed that training $\alpha$ parameter helps the model to
fit the distribution of pixel values more precisely. Therefore, in our
experiment, we decide to train $\alpha$ in the channel tuning module.
Consequently, EoREN can learn true color of the image without hindering the
representability for gradients.

##  5 Experiment

In this section, we experimentally show the performance of EoREN model
compared to SIREN models with the pixel fitting and the gradient fitting.
First, we show the images reconstructed by three models. Also, we conduct
quantitative analysis for the Set14 [9], DIV2K [20], and MNIST [5] dataset
with image qualifying metrics, PSNR and SSIM score. Furthermore, we show the
result of the image composition task by the EoREN model and demonstrate the
advantage and the weakness of EoREN. We take an experiment on gray scale of an
image because an image with 3 channels requires a bigger size model and more
training time. We trained 1000 epochs for each model equally. In the case of
EoREN , we trained 950 epochs for the edge-oriented module and 50 epochs for
the channel-tuning module.

###  5.1 Signal Reconstruction

We first show the signal reconstruction of our model. We test our model with
three color images in skimage data. We choose three images, the astronaut, the
rocket and the coffee. The results of reconstruction are described in the
Figure 4. The first two columns in Figure 4 are the images from SIREN with
pixel/gradient fitting. Gradient fitting images successfully represent the
edge parts of the image, while they fail to match the RGB values of the image.
The results of SIREN with pixel fitting have plausible reconstructed RGB
values, but fail to distinguish detailed edges on image, such as the patch in
the astronaut image, towers in the rocket image and the background in the
coffee image. The images reconstructed by our EoREN are the 3th and 4th
columns on Figure 4. EoREN without channel tuning module overestimate the
edges, such as shadows in the coffee image and dark parts in the rocket image.
Then, it results in generating high-contrast image compared to the original
images. Meanwhile, EoREN with channel tuning module shows not only clear but
also realistic images with similar color levels compared to the other gradient
fitting models.

Figure 5: The histogram of the pixel values and the gradients for the RGB astronaut image.  Table 1: The result of image reconstruction for the baseline models and the EoREN. |  Set14 |  DIV2K |  MNIST
---|---|---|---
Measure | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
SIREN+Pixel | 73.81 | 0.78 | 74.23 | 0.83 | 83.69 | 0.91
SIREN+Grad | 56.46 | 0.33 | 55.45 | 0.31 | 54.54 | 0.74
EoREN | 66.01 | 0.46 | 64.36 | 0.41 | 93.96 | 0.97

###  5.2 Disentangled Property on Edge and Offset

One advantage of EoREN is the separation of modules for edge and offset. We
show the disentangled property in color and edge construction of our model by
the reconstruction of the edge-oriented module and the channel-tuning module
separately. The distributions of the pixel values and the gradients for three
models are shown in the Figure 5. The SIREN with pixel fitting shows that most
of the pixel values are matched. However, the model underestimates the
gradients and results in the smooth image as shown in the Figure 4. The SIREN
with gradient fitting matches most of the gradients but fails to reconstruct
pixel values. We find that two pixel distributions from the ground truth and
the model output are similar in shape. Therefore, our proposed model EoREN
linearly transforms the pixel values by the channel-tuning module and matches
the pixel distributions.

###  5.3 Benchmark Dataset

Following the previous work in [18], we compare EoREN and SIREN on the
benchmark dataset. We evaluate our model based on the PSNR and SSIM metric [8]
which is widely used metric for image reconstruction. The overall results are
in the Table LABEL:table_psnr_ssim.

####  5.3.1 Set14

Set14 contains 14 images from simple to complex. The PSNR performance of
EoREN, SIREN with pixel fitting, and SIREN with gradient fitting for each
sample is in Figure 6. We observe that SIREN+pixel shows generally better PSNR
scores than EoREN. However, there are some samples on which EoRENshows better
PSNR than SIREN+pixel. We observe that EoRENachieves better performance when
an image contains less complex edges. We shows four examples in Figure 7. The
images of Face and Flower in Set14 contains more edges than the images of
Pepper and PPT3. This implies that EoREN reconstructs an image well when it
contains more clear edges.

Figure 6: The PSNR performance of SIREN+pixel, SIREN+grad, and EoREN for Set14
dataset. EoREN outperforms SIREN+pixel for three images, coastguard, pepper,
and ppt3.  Figure 7: Reconstructed images by EoREN and SIREN with pixel
fitting. the left panel contains the images which SIREN+pixel outperforms
EoREN, whereas the right panel contains the images which EoREN outperforms
SIREN+pixel.

####  5.3.2 DIV2K

DIV2K contains high resolution realistic images and we trained three models
for validation dataset of the high resolution images of DIV2K which constains
100 samples. The image has higher resolution than $256\times 256$ and training
bigger size of an image requires larger model and more computational time.
Therefore, we down-sample the image to $256\times 256$ size. The result in
Table LABEL:table_psnr_ssim shows that EoREN has lower performance than
SIREN+pixel. When we compare the performance of each samples, there is no case
EoREN outperforms SIREN+pixel. We hypothesize that this is due to the fact
that high resolution images contains more complex edge patterns than original
$256\times 256$ image and down-sampling does not contain clear edge
information.

####  5.3.3 MNIST

Finally, we tested three models with MNIST dataset. As the results of Table
LABEL:table_psnr_ssim and the stacked box plot in Figure 8 show, EoREN
significantly outperforms SIREN+pixel and SIREN+grad. We show an
reconstruction example of digit six in Figure 9. When training SIREN with
pixel fitting, there is a watermark on the reconstructed image, whereas, the
reconstructed image of EoRENhas no watermark. When the EoRENis trained without
the channel-tuning module, it is equivalent to the SIREN+grad. As the
performance of SIREN+grad suggests, the channel tuning module is necessary for
the gradient fitting of image.

Figure 8: The performance of EoREN, SIREN with pixel fitting, and SIREN with
gradient fitting for each digits. EoREN outperforms other models for all
digits. We test for 10 samples each.  Figure 9: Reconstruction sample of SIREN
with pixel fitting and EoREN for digit six for 500 and 1000 epochs.

###  5.4 Poisson Image Editing

Two images $f_{1}$ and $f_{2}$ can be composited by the interpolation of them.
The composite image $f_{1,2}$ can be trained using Possion image editing
method [15] which focuses on the edge of two images. Poisson image editing
technique uses addition of the gradients of images and trained with single
parametric function by

$\displaystyle\mathcal{L}_{\mathrm{grad.}}=\int_{\Omega}||\nabla_{\mathbf{x}}\Phi(\mathbf{x})-\nabla_{\mathbf{x}}f_{1,2}||d\mathbf{x}$
(12) $\displaystyle f_{1,2}=\lambda
f_{1}(\mathbf{x})+(1-\lambda)f_{2}(\mathbf{x})$ (13)

When model $\Phi$ is fully trained by Equation 13, we get two edge based
composited image. The proportion of two images is controlled by the constant
$\lambda\in[0,1]$. In the case of EoREN, there are already two hyperparameters
$\gamma_{1}$ and $\gamma_{2}$ in a single image to control the training, the
learning rate of the edge-oriented module and the offset-tuning module. Then,
the composition of two images can be trained by two losses

$\displaystyle\mathcal{L}_{1,2}^{grad}=\lambda\gamma_{1}\mathcal{L}_{\mathrm{grad.}}^{1}+(1-\lambda)\gamma_{1}\mathcal{L}_{\mathrm{grad.}}^{2}$
(14)
$\displaystyle\tilde{\mathcal{L}}_{1,2}=\lambda\gamma_{2}\tilde{\mathcal{L}}^{1}+(1-\lambda)\gamma_{2}\tilde{\mathcal{L}}^{2}$
(15)

where $\mathcal{L}^{i}_{grad}$ is the loss of edge-oriented module of $f_{i}$
and $\tilde{\mathcal{L}}^{i}$ is the loss of offset-tuning module of $f_{i}$.
Therefore, there are total 3 number of independent hyperparameters to train
composition. We show that by setting proper parameter, we can make the
composition image with desired magnitude of edge and color. We test this
Poisson image editing method for EoREN.

We show the result of the image composition in Figure 10. Two images have
different degree of complexity.The first one is an image of a field with
reeds, and the second one is an image of an eagle flying against the sky and
sea. The first image contains more edges than the second image and therefore
more difficult to train with gradients. The result shows that EoREN
successfully combines two images when the portion of the second image is more
than the first image. This implies that the channel tuning depends on the
complexity of the edges.

Figure 10: The result of image composition for $\lambda=0.2,0.6,0.9$. EoREN
reconstruct the composited image successfully when $\lambda$ is large
indicating that EoREN prefers an image with simple patterns.

##  6 Conclusions

In this paper, we propose the Gradient Magnitude Adjustment algorithm to
extract image gradients and the Edge-oriented Representation Network which
disentangles the edge features and the channel-wise value features. We
demonstrate that the implicitly represented image with the gradients adjusted
by the GMA matches the original image well. In addition, we empirically show
that the SIREN model fitted with pixel value makes a smooth image while the
SIREN model fitted with gradients captures the edge parts well. Our proposed
EoREN model is separated into two parts which makes both edges and values of
the image be reconstructed well.
Even though our model improves image reconstruction performance trained with
implicit function, there still exist some challenges to be solved. The
channel-tuning module highly depends on the performance of the edge-oriented
module. Also, the model output is desaturated compared to the original image
even though the distribution of pixel values have no significant difference.
We expect significant improvements on the reconstruction performance by
solving such problems.

## References

  * [1] Matan Atzmon and Yaron Lipman.  Sal: Sign agnostic learning of shapes from raw data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [2] John Canny.  A computational approach to edge detection.  IEEE Transactions on pattern analysis and machine intelligence, (6):679–698, 1986.
  * [3] Yinbo Chen, Sifei Liu, and Xiaolong Wang.  Learning continuous image representation with local implicit image function, 2021.
  * [4] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling, 2019.
  * [5] Li Deng.  The mnist database of handwritten digit images for machine learning research.  IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  * [6] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser.  Local deep implicit functions for 3d shape, 2020.
  * [7] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.  Implicit geometric regularization for learning shapes, 2020.
  * [8] Alain Horé and Djemel Ziou.  Image quality metrics: Psnr vs. ssim.  In 2010 20th International Conference on Pattern Recognition, pages 2366–2369, 2010.
  * [9] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja.  Single image super-resolution from transformed self-exemplars.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5197–5206, 2015.
  * [10] Sunil Kumar Katiyar and PV Arun.  Comparative analysis of common edge detection techniques in context of object extraction.  arXiv preprint arXiv:1405.6132, 2014.
  * [11] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681–4690, 2017.
  * [12] Cheng Ma, Yongming Rao, Yean Cheng, Ce Chen, Jiwen Lu, and Jie Zhou.  Structure-preserving super resolution with gradient guidance.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7769–7778, 2020.
  * [13] Mateusz Michalkiewicz, Jhony K. Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson.  Implicit surface representations as layers in neural networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [14] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European conference on computer vision, pages 405–421. Springer, 2020.
  * [15] Patrick Pérez, Michel Gangnet, and Andrew Blake.  Poisson image editing.  In ACM SIGGRAPH 2003 Papers, pages 313–318. 2003.
  * [16] Takashi Shibata, Masayuki Tanaka, and Masatoshi Okutomi.  Gradient-domain image reconstruction framework with intensity-range and base-structure constraints.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2745–2753, 2016.
  * [17] GT Shrivakshan and Chandramouli Chandrasekar.  A comparison of various edge detection techniques used in image processing.  International Journal of Computer Science Issues (IJCSI), 9(5):269, 2012.
  * [18] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7462–7473. Curran Associates, Inc., 2020.
  * [19] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations.  Curran Associates Inc., Red Hook, NY, USA, 2019.
  * [20] Radu Timofte, Shuhang Gu, Jiqing Wu, Luc Van Gool, Lei Zhang, Ming-Hsuan Yang, Muhammad Haris, et al.  Ntire 2018 challenge on single image super-resolution: Methods and results.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
  * [21] Saining Xie and Zhuowen Tu.  Holistically-nested edge detection.  In Proceedings of the IEEE international conference on computer vision, pages 1395–1403, 2015.