#  $\lambda$-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion
Models
by Leveraging CLIP Latent Space

Maitreya Patel*†, Sangmin Jung*, Chitta Baral, Yezhou Yang
Arizona State University
https://eclipse-t2i.github.io/Lambda-ECLIPSE/

###### Abstract

††* indicates equal contribution, $\dagger$ Corresponding Author:
<EMAIL_ADDRESS>
Despite the recent advances in personalized text-to-image (P-T2I) generative
models, subject-driven T2I remains challenging. The primary bottlenecks
include 1) Intensive training resource requirements, 2) Hyper-parameter
sensitivity leading to inconsistent outputs, and 3) Balancing the intricacies
of novel visual concept and composition alignment. We start by re-iterating
the core philosophy of T2I diffusion models to address the above limitations.
Predominantly, contemporary subject-driven T2I approaches hinge on Latent
Diffusion Models (LDMs), which facilitate T2I mapping through cross-attention
layers. While LDMs offer distinct advantages, P-T2I methods’ reliance on the
latent space of these diffusion models significantly escalates resource
demands, leading to inconsistent results and necessitating numerous iterations
for a single desired image. Recently, ECLIPSE has demonstrated a more
resource-efficient pathway for training UnCLIP-based T2I models, circumventing
the need for diffusion text-to-image priors. Building on this, we introduce
$\lambda$-ECLIPSE111The designation $\lambda$-ECLIPSE is inspired by its
conceptual alignment with the $\lambda$-calculus. In this context, the
$\lambda$-ECLIPSE model functions similarly to a functional abstraction within
$\lambda$-calculus, where it effectively binds variables. These variables, in
our case, represent novel visual concepts that are integrated through
composition prompts.. Our method illustrates that effective P-T2I does not
necessarily depend on the latent space of diffusion models. $\lambda$-ECLIPSE
achieves single, multi-subject, and edge-guided T2I personalization with just
34M parameters and is trained on a mere 74 GPU hours using 1.6M image-text
interleaved data. Through extensive experiments, we also establish that
$\lambda$-ECLIPSE surpasses existing baselines in composition alignment while
preserving concept alignment performance, even with significantly lower
resource utilization.

Figure 1:  Our $\lambda$-ECLIPSE can estimate subject-specific image
embeddings while maintaining the balance between concept and composition
alignment. $\lambda$-ECLIPSE supports multi-concept personalization and
provides additional controls (i.e., edge map) for image generation. In this
work, we show that it is possible to perform subject-driven T2I in the latent
space of a pre-trained CLIP model without depending on the diffusion UNet
models; effectively reducing the heavy resource requirements.

##  1 Introduction

The field of text-to-image (T2I) diffusion models has recently witnessed
remarkable advancements, achieving greater photorealism and enhanced adherence
to textual prompts. This has catalyzed the emergence of diverse applications,
notably in controllable T2I models and personalized T2I (P-T2I) models. P-T2I,
in particular, encompasses the intricate task of learning and reproducing
novel visual concepts or subjects in varied contexts, beyond mere replication
of reference images. This personalization task is challenging, necessitating a
nuanced balance between concept and compositional alignment. The complexity
escalates further when multi-subject personalization is desired.

Early works in this field, such as Textual Inversion [12] and DreamBooth [38],
employed per-concept optimization strategies involving fine-tuning certain
parameters within T2I diffusion models. However, these methods struggle with
generalization challenges and are time-intensive. Contemporary research is
pivoting towards fast personalization techniques that involve training
hypernetworks and integrating new layers or parameters within pre-trained
diffusion UNet models. As summarized in Table 1, despite their relative
parameter efficiency, these methods are still resource-intensive (especially
for multi-concept personalization), largely due to their reliance on the
latent spaces of diffusion models. Additionally, controlling the inherent
randomness in diffusion models remains a significant challenge, often
resulting in inconsistent outputs and necessitating multiple attempts to
generate a single accurate image.

Upon further investigation, we find that most subject-driven T2I approaches
build upon variants of the Latent Diffusion Model (LDM) [37], specifically
Stable Diffusion models. LDMs employ cross-attention layers to condition
diffusion models with text embeddings, necessitating a mapping of target
subject images to latent spaces compatible with the diffusion models, either
as text embeddings or as supplementary model parameters. This process involves
backpropagation through the entire diffusion model, often comprising over a
billion parameters, contributing to the inefficiency of existing P-T2I
methods.

In contrast, the UnCLIP [35] T2I family of LDMs offers an intriguing
alternative, wherein text embeddings are mapped to vision embeddings through a
diffusion transformer prior network before passing the conditioning to the
diffusion UNet image generator. This mechanism, as illustrated in Figure 2,
enables the generator to accurately reproduce original images from their
visual embeddings alone. Consequently, our research pivots towards leveraging
the UnCLIP T2I stack for P-T2I strategies. However, like their predecessors,
these T2I diffusion priors are also parameter-heavy, with around 1 billion
parameters. Until recently, ECLIPSE [31] showed that it is possible to
compress this 1B parameter of the T2I prior model down to 33M by removing the
diffusion modeling.

Table 1: A quick overview of previous works on T2I personalization. Our method
is the first to offer multi-concept personalization without depending on
diffusion UNet models (except for inference). We provide the extended overview
table in the supplementary materials.

Method | Multi | Finetuning | Diffusion | Total opt. | GPU
---|---|---|---|---|---
| Concepts | Free | Free | params | Hours
Textual Inversion [12] | ✗ | ✗ | ✗ | 768 | 1
DreamBooth [38] | ✗ | ✗ | ✗ | 0.9B | 0.5
ELITE [48] | ✗ | ✓ | ✗ | 77M | -
BLIP-Diffusion [21] | ✗ | ✓ | ✗ | 1.5B | 2304
IP-Adapter [50] | ✗ | ✓ | ✗ | 22M | -
Custom Diffusion [20] | ✓ | ✗ | ✗ | 57M | 0.2
Subject-Diffusion [27] | ✓ | ✓ | ✗ | 252M | -
Kosmos-G [29] | ✓ | ✓ | ✗ | 1.9B | 12300
$\lambda$-ECLIPSE (ours) | ✓ | ✓ | ✓ | 34M | 74

Figure 2:  CLIP(vision) features capture the semantics and fine-grained visual
details. UnCLIP models (e.g., Kandinsky v2.2) can effectively leverage this
latent space to reconstruct the image from its embedding.

ECLIPSE posits that text-to-image mapping can be optimized through contrastive
pre-training. It inputs text embeddings and estimates corresponding image
embeddings, ensuring strong alignment with textual features. Building on these
insights, to enhance this framework and deepen the comprehension of novel
visual subjects, we introduce a novel pre-training strategy and propose the
$\lambda$-ECLIPSE T2I prior model for the UnCLIP framework. Specifically, we
propose a subject-driven instruction tuning task involving the image-text
interleaved data. This involves creating 1.6 million high-quality image-text
pairs, where text tokens linked to subjects are substituted with image
embeddings. Subsequently, $\lambda$-ECLIPSE is trained to estimate image
embeddings that not only harmonize with text semantics but also encapsulate
subject representations.

This model facilitates multi-subject-driven image generation using UnCLIP
models without imposing excessive resource demands. $\lambda$-ECLIPSE bypasses
the reliance on diffusion latent space and operates within a pre-trained CLIP
latent space [33]. We leverage Kandinsky v2.2 diffusion UNet model’s [36]
capacity to interpret image structures from CLIP embeddings (refer to Figure
2). Importantly, CLIP image embeddings even accurately capture the details of
the canny edge map [6]. Therefore, we extend $\lambda$-ECLIPSE to incorporate
canny edge maps as auxiliary guides for subject-driven T2I. Our extensive
experimental validation demonstrates that $\lambda$-ECLIPSE achieves
unparalleled performance in single and multi-subject composition alignment
while preserving the integrity of reference subjects, competitively rivaling
Multimodal Large Language Models (MLLM) based methods that require intense
computing resources. At last, we find that $\lambda$-ECLIPSE inherits the
continuous latent space from the CLIP; for the first time allowing the
interpolations between multiple concepts for P-T2I models. Figure 1
illustrates the qualitative capabilities of $\lambda$-ECLIPSE.

In summary, our contributions are: (1) we introduce a straightforward and yet
very effective diffusion-free multi-subject personalization strategy by
estimating subject-aligned image embeddings in CLIP(vision) latent space, (2)
despite significantly reducing resource requirements, our method achieves
state-of-the-art performance, and (3) we demonstrate the potential of
utilizing diffusion image generators as standalone pre-trained entities,
enabling controlled outputs without necessitating explicit fine-tuning.

##  2 Related Works

Figure 3:  This figure illustrates the three stages of the $\lambda$-ECLIPSE
pipeline. 1) Create the image-text interleaved features by utilizing the pre-
trained CLIP encoders. 2) Pre-train the $\lambda$-ECLIPSE (34M parameters)
using Eq. 1, which ensures the mapping to the desired latent space given the
image-text interleaved data. During inference, we just need a single-forward
pass to estimate the corresponding image embedding. 3) During inference, the
Kandinsky v2.2 diffusion UNet model takes the output from the
$\lambda$-ECLIPSE (prior) and generates the corresponding images.

#### Text-to-Image Generative Models.

Pioneering efforts in image generation, notably DALL-E [34] and CogView [11],
leveraged autoregressive models to achieve significant results. Recent
advancements predominantly feature diffusion models, acclaimed for their high
image fidelity and diversity in text-to-image (T2I) generation. A notable
example is Stable Diffusion, which builds upon the Latent Diffusion Model
(LDM) [37] and excels in semantic and conceptual understanding by
transitioning training to latent space. Imagen [40], Pixart-$\alpha$ [8], and
DALL-E 3 [5] propose using a large T5 language model to improve language
understanding. DALL-E 2 [35] along with its UnCLIP variation models such as
Kandinsky [36] and Karlo, uses a diffusion prior and diffusion UNet modules to
generate images using the pre-trained CLIP [33] model.

#### Personalized T2I Methods.

Approaches like Textual Inversion [12], DreamBooth [38], and Custom Diffusion
[20] focus on training specific parameters to encapsulate visual concepts.
LoRA [16] and Perfusion [45] target efficient fine-tuning adjustments,
particularly rank 1 modifications. However, these methods are constrained by
their requirement for concept-specific tuning. ELITE [48] was the first
approach addressing fast personalization for single-subject T2I. BLIP-
Diffusion [21] adapts the BLIP2 encoder [22], training approximately 1.5B
parameters to enable zero-shot, subject-driven image generation. IP-Adapter
[50] introduces a decoupled cross-attention mechanism, negating the need to
train the foundational UNet model by permitting fine-tuning of a reduced
parameter set (22M).

Mix-of-Show [14] and Zip-LoRA [42] train individual concepts and then combine
them to generate multiple subjects. Break-A-Scene [4] shows multi-concept
capability but requires single images containing diverse objects. Subject
Diffusion [27] creates a high-quality dataset and presents the precision
control for fast multi-subject image generation. Kosmos-G [29], akin to
Subject-Diffusion, employs a Multimodal Large Language Model (MLLM) for text-
image embedding alignment, though it necessitates extensive parameter
optimization (1.9B). These multi-subject P-T2I methods are not only demanding
in terms of parameters but also depend on a massive number of frozen
parameters of the diffusion UNet model, increasing training computational
loads. In contrast, our model, $\lambda$-ECLIPSE, forgoes test-time fine-
tuning and training-time reliance on the diffusion UNet model for single and
multi-concept, control-guided P-T2I, positioning it as a resource-efficient
solution.

At last, methods like GLIGEN [23], ControlNet [51], and UniControl [32]
incorporate additional controls (i.e., edge map, depth, segmentations) into
the diffusion model to generate the desired images. BLIP-Diffusion, IP-
Adapter, and Kosmos-G can leverage such pre-trained controls. However, in many
scenarios, these controls are too strong which makes generated images lose
subject-specific details. While $\lambda$-ECLIPSE can be plugged with these
pre-trained methods, we explore the possibility of learning canny edge as the
additional guidance for P-T2I. We show that $\lambda$-ECLIPSE learns to
balance the edge map, subjects, and text composition.

##  3 Method

In this section, we introduce $\lambda$-ECLIPSE, our approach to multi-subject
text-to-image personalization. This method extends the capabilities of the
existing ECLIPSE framework through a novel image-text interleaved pretraining
strategy, notably omitting the need for explicit diffusion modeling. Our
approach mainly capitalizes on the efficient utilization of the CLIP latent
space. Figure 3 outlines the end-to-end framework.

The primary objective of $\lambda$-ECLIPSE is to facilitate both single and
multi-subject T2I generation processes, accommodating controlled conditional
guidance such as edge maps. Initially, we detail the problem formulation and
elaborate on the extended design of the ECLIPSE T2I prior module.
Subsequently, we delve into the image-text interleaved training methodology.
This fine-tuning process enables the ECLIPSE model to harness semantic
correlations between CLIP image and text latent spaces while preserving
subject-specific visual features. Upon attaining the target embedding, the
diffusion UNet model, derived from Kandinsky v2.2, is employed to reconstruct
the image.

###  3.1 Text-to-Image Prior Mapping

In the UnCLIP T2I models, the objective of the text-to-image prior model
($f_{\theta}$) is to establish a proficient text-to-image embedding mapping.
This model is designed to adeptly map textual representations to their
corresponding visual embeddings, denoted as ($f_{\theta}:z_{y}\rightarrow
z_{x}$), where $z_{x/y}$ represent the embeddings for images and text,
respectively. The visual embedding predictions
($\hat{z_{x}}=f_{\theta}(z_{y})$) are then effectively utilized by the
diffusion image generators ($h_{\phi}$), which are inherently conditioned on
these vision embeddings ($h_{\phi}:z_{x}\rightarrow x$).

As illustrated in Figure 2, these diffusion image generators possess a
remarkable capability for precise image reproduction, since $z_{x}$
encompasses comprehensive image information. Our goal is to accurately
estimate the image embedding $\hat{z}_{x}$, incorporating the subject
representations, thereby eliminating reliance on $h_{\phi}$ during training.
Existing LDM-based P-T2I methods are limited by the LDM’s singular module
approach ($h_{\phi}:z_{y}\rightarrow x$). Consequently, mastering the latent
space of $h_{\phi}$ becomes essential for effective P-T2I for the baseline
methodologies.

We propose a mapping function, $f^{\prime}_{\theta}$, which processes text
representations ($z_{y}$) alongside subject ($x_{i}$) specific representations
($z_{x_{i}}$), to derive an image embedding that encapsulates both text
prompts and subject visuals ($\hat{z}_{x}$). The challenge lies in harmonizing
$z_{x_{i}}$ and $z_{y}$ within
$f^{\prime}_{\theta}:(z_{y},z_{x_{i}})\rightarrow\hat{z}_{x}$, ensuring
alignment while preventing overemphasis on either aspect, as this could
compromise composition alignment. To address this, we employ the contrastive
pre-training strategy after [31]:

$\displaystyle\begin{split}\mathcal{L}_{prior}=&\underset{\begin{subarray}{c}\epsilon\sim\mathcal{N}(0,I)\\\
z_{y},z_{x}\end{subarray}}{\operatorname{\mathbb{E}}}\Big{[}||z_{x}-f_{\theta}(\epsilon,z_{y})||_{2}^{2}\Big{]}\\\
&-\frac{\lambda}{N}\sum_{i=0}^{N}\log\frac{\exp(\langle\hat{z}^{i}_{x},z^{i}_{y}\rangle/\tau)}{\sum_{j\in[N]}\exp(\langle\hat{z}^{i}_{x},z^{j}_{y}\rangle/\tau)}.\end{split}$
(1)

Here, $\lambda$ serves as the hyperparameter. The first loss term measures the
mean-squared error between the estimated and actual image embeddings,
primarily ensuring concept alignment. However, our preliminary studies reveal
that exclusive reliance on this term diminishes composition alignment, echoing
observations made by [31]. Therefore, we stick with the contrastive loss
component to bolster compositional generalization, with $\lambda$ balancing
concept and composition alignment.

Figure 4: Qualitative examples of $\lambda$-ECLIPSE. Figure 5:  This figure
illustrates the qualitative comparison of $\lambda$-ECLIPSE w.r.t. the state-
of-the-art approaches for single-subject T2I generations. We choose concepts
and prompts from the Dreambench dataset. For all methods, concepts, and
prompts, we generate the four images and select the best one that represents
the concept and composition more accurately.

#### Additional Control-based T2I Prior Mapping.

Acknowledging the limitations in existing methods, which necessitate learning
the diffusion latent space even for additional control inputs, we endeavor to
achieve a more nuanced balance between subject, text, and supplementary
conditions. Consequently, we have augmented $\lambda$-ECLIPSE to accommodate
an additional modality, a canny edge map, providing more refined control over
subject-driven image generation. This entails modifying the prior model to
accept additional conditions
($f_{\theta}^{\prime}:(z_{y},z_{x_{i}},z_{c})\rightarrow\hat{z}_{x}$, where
$z_{c}$ symbolizes the additional modality embedding). During training, we
drop $z_{c}$ for 1% to improve the unconditional generations.

Incorporating canny edge maps during training enhances stability and broadens
the generalization capabilities of $\lambda$-ECLIPSE, yielding benefits even
in the absence of these controls during inference. Our findings indicate that
$\lambda$-ECLIPSE successfully learns a unified mapping function, effectively
estimating target image representations from a synergy of text, subjects, and
edge map.

###  3.2 Image-text Interleaved Training

Our approach targets developing a versatile prior model capable of processing
diverse inputs to estimate visual outputs. Drawing from earlier methodologies,
a straightforward solution involves concatenating different inputs, like
combining text (“a dog wearing sunglasses”) with respective concept-specific
images. However, preliminary experiments indicated that this method does not
effectively capture the intricate relationships between target text tokens
(e.g., “dog”) and the corresponding concept images.

To address this, we adopt the interleaved pre-training strategy used in
Kosmos-G, but with a notable modification to enhance resource efficiency. We
incorporate pretrained CLIP text and vision encoders for extracting modality-
specific embeddings—separating text-only from subject-specific images. The key
refinement in our process is the substitution of subject token embeddings with
corresponding vision embeddings instead of introducing additional tokens to
handle the image embeddings via resampler [2]. This alteration allows us to
bypass the need to train the prior or MLLMs, significantly improving the
model’s proficiency in handling interleaved data.

For the generation of high-quality training datasets, we carefully selected 2
million high-quality images from the LAION dataset [41], each with a
resolution of 1024x1024. Utilizing BLIP2, we generate captions for these
images and employ SAM [19] for extracting noun or subject-specific
segmentation masks. Given the CLIP model’s requirement for 224x224 resolution
images, we avoid resizing the masks within their original resolutions.
Instead, we opt for cropping the area of interest using DINO [24], followed by
resizing the masked object while preserving its aspect ratio. This technique
is crucial in retaining maximum visual information for each subject during the
training phase. We provide more details about the filters used in the
supplementary material.

In summary, the ECLIPSE prior, trained with our image-text interleaved data
and supplementary conditions, presents an efficient pathway for P-T2I. The
culmination of this process yields the $\lambda$-ECLIPSE model, a refined
version characterized by enhanced effectiveness in handling and interpreting
multimodal data.

##  4 Experiments

In this section, we first introduce the experimental setup, evaluations, and
baselines. Later, we delve into the qualitative and quantitative results.

#### Training and inference details.

We initialize our model, $\lambda$-ECLIPSE, equipped with 34M parameters. We
train our model on an image-text interleaved dataset of 2M instances,
partitioned into 1.6M for training and 0.4M for validation. The model is
specifically tuned for the Kandinsky v2.2 diffusion image decoder. Therefore,
we use pre-trained OpenCLIP-ViT-G/14222https://huggingface.co/laion/CLIP-
ViT-g-14-laion2B-s12B-b42K as the text and vision encoders, ensuring alignment
with Kandinsky v2.2 image embeddings. Training is executed on 2 x A100 GPUs,
leveraging a per-GPU batch size of 512 and a peak learning rate of 0.00005,
across approximately 100,000 iterations, summing up to 74 GPU hours. During
inference, the model employs 50 DDIM steps and 7.5 classifier-free guidance
for the Kandinsky v2.2 diffusion image generator. Adhering to baseline
methodologies, we perform the P-T2I following the baseline papers’ protocols.
For $\lambda$-ECLIPSE, target subject pixel regions in reference images are
segmented before embedding extraction via the CLIP(vision) encoder. We drop
the canny edge map during inference unless specified explicitly.

#### Evaluation setup.

We primarily utilize Dreambench (encompassing 30 unique concepts with 25
prompts per concept) for qualitative and quantitative evaluations using DINO
and CLIP-based metrics [38]. Due to their limitations, we extend our
evaluations on the ConceptBed [30] benchmark (covering 80 diverse imagenet
concepts and a total of 33K composite prompts), where we report performance on
concept replication, concept, and composition alignment using the Concept
Confidence Deviation ($CCD(\downarrow)$) metric.

###  4.1 Results & Analysis

Table 2:  Quantitative comparisons of different methodologies on Dreambench.
The Bold and underline represent the metric-specific first and second-ranked
methods, respectively. * represents that we reproduce the open-source
methodologies. Other results are borrowed from the Subject-Diffusion [27].

Method | Base Model | DINO ($\uparrow$) | CLIP-I ($\uparrow$) | CLIP-T ($\uparrow$)
---|---|---|---|---
Textual Inversion | SDv1.5 | 0.569 | 0.780 | 0.255
DreamBooth | SDv1.5 | 0.668 | 0.803 | 0.305
Custom Diffusion | SDv1.5 | 0.643 | 0.790 | 0.305
Re Imagen | Imagen | 0.600 | 0.740 | 0.270
ELITE | SDv1.4 | 0.621 | 0.771 | 0.293
Subject-Diffusion | SDv1.5 | 0.711 | 0.787 | 0.293
BLIP-Diffusion* | SDv1.5 | 0.603 | 0.793 | 0.291
IP-Adapter* | SDv1.5 | 0.629 | 0.827 | 0.264
IP-Adapter* | SDXL | 0.613 | 0.810 | 0.292
Kosmos-G* | SDv1.5 | 0.618 | 0.822 | 0.250
$\lambda$-ECLIPSE* | Kv2.2 | 0.604 | 0.777 | 0.307

Figure 6:  Multi-subject qualitative examples illustrating the comparison
between $\lambda$-ECLIPSE and other SOTA baselines.  Table 3:  Quantitative
comparisons of different methodologies on ConceptBed. We present results on
$CCD~{}(\downarrow)$ across three evaluation categories. The Bold and
underline represent the metric-specific first and second-ranked methods,
respectively. * represents that we reproduce the open-source methodologies.
Other results are borrowed from the ConceptBed [30].

Method | Base | Concept | Concept | Composition
---|---|---|---|---
| Model | Replication | Alignment | Alignment
Textual Inversion | SDv1.4 | 0.0662 | 0.1163 | 0.1436
Dreambooth | SDv1.4 | 0.0880 | 0.3551 | 0.0360
Custom Diffusion | SDv1.4 | 0.2309 | 0.4882 | 0.0204
ELITE* | SDv1.4 | 0.3195 | 0.4666 | 0.1832
BLIP-DIffusion* | SDv1.5 | 0.3510 | 0.3245 | 0.1589
IP-Adapter* | SDXL | 0.3665 | 0.3571 | 0.0641
$\lambda$-ECLIPSE* | Kv2.2 | 0.2853 | 0.3619 | -0.0200

Figure 7:  Qualitative examples for edge-guided P-T2I showcasing the
comparison between BLIP-Diffusion vs. $\lambda$-ECLIPSE.

#### Quantitative comparison.

The quantitative assessments are detailed in Table 2 and Table 3, focusing on
the single-concept T2I task. For Dreambench, we generate and evaluate four
images per prompt, reporting average performance on three metrics (DINO,
CLIP-I, and CLIP-T). In the case of ConceptBed, we process each of the 33K
prompts to generate a single image. The results, as depicted in these tables,
highlight $\lambda$-ECLIPSE’s superior performance in composition tasks and
its competitive edge over even fine-tuning-based methodologies, while
maintaining robust concept alignment. $\lambda$-ECLIPSE exhibits a notable
proficiency in concept replication, albeit with a marginal trade-off in
concept alignment for enhanced composition fidelity. Comparatively, IP-
Adapter, ELITE, and BLIP-Diffusion prioritize concept alignment, often at the
expense of composition alignment. Noteworthy is $\lambda$-ECLIPSE’s
efficiency, achieved with significantly fewer resources.

#### Qualitative comparisons.

In Figure 5, we present a range of single subject-specific images generated by
various methodologies including BLIP-Diffusion, IP-Adapter, Kosmos-G, and
$\lambda$-ECLIPSE. $\lambda$-ECLIPSE demonstrates exemplary proficiency in
composition while ensuring concept alignment. In contrast, the baselines often
overemphasize reference images (notably in rows 2 and 4) or exhibit concept
dilution (as seen in rows 6 and 8), leading to higher concept alignment but
compromised composition. Additionally, Figure 6 exhibits $\lambda$-ECLIPSE’s
multi-concept generation prowess, in comparison to ZipLoRA (fine-tuning-based
approach) and Kosmos-G (Multimodal LLM-based approach), underscoring its
capability to rival compute-intensive methodologies.

Figure 2 previously demonstrated Kandinsky v2.2’s ability to partially
reconstruct canny edge maps while preserving semantic integrity. Building on
this, Figure 7 offers qualitative insights, comparing BLIP-Diffusion and
$\lambda$-ECLIPSE in edge-guided subject-driven T2I scenarios. Here, BLIP-
Diffusion adheres strictly to the imposed conditions, often at the cost of
concept retention (rows 1, 3, and 4). Conversely, $\lambda$-ECLIPSE not only
complies with these conditions but also adeptly balances the need for high
concept and composition alignment.

Figure 8:  Interpolation between four concepts. Here, we estimate the image
embedding using $\lambda$-ECLIPSE corresponding to each corner and then
interpolate from top to bottom and left to right. At last, we use the
Kandinsky v2.2 diffusion UNet model to generate the images with fixed random
seeds from these sets of image embeddings.  Table 4: Ablation studies w.r.t.
to the key components of $\lambda$-ECLIPSE design. We report the concept and
composition alignment for single-subject T2I using $CCD~{}(\downarrow)$ on the
ConceptBed benchmark.

Model | Concept | Composition
---|---|---
| Alignment | Alignment
Projection loss | 0.394 | 0.008
w/ contrastive loss (lambda=0.5) | 0.435 | -0.043
w/ contrastive loss (lambda=0.2) | 0.402 | -0.026
w/ edge conditions (lambda=0.2) | 0.362 | -0.020

#### Ablations.

We extend our study to evaluate the individual contributions of different
components in $\lambda$-ECLIPSE. Initially, the model’s performance with
solely the projection loss (referenced in Eq.1) is assessed. Subsequent
experiments involve training $\lambda$-ECLIPSE variants with varying
hyperparameters for the contrastive loss, specifically $\lambda$ values of 0.2
and 0.5. A comparative analysis of these baselines is conducted against the
fully equipped $\lambda$-ECLIPSE model, which incorporates
$\mathcal{L}_{prior}$ (Eq.1) with $\lambda=0.2$ and utilizes canny edge maps
during training. As Table 4 illustrates, relying solely on projection loss
results in high concept alignment but compromises compositions. The
contrastive loss variant with $\lambda=0.5$ enhances composition alignment at
the expense of concept alignment, whereas $\lambda=0.2$ achieves a more
balanced performance. Crucially, the integration of canny edge maps during
training optimally balances both alignments and, specifically, improves the
concept alignment. Here, the negative values indicate that the $CCD$ oracle
model performs similarly to real images having the same set of compositions.

#### Multi-subject interpolation.

A key attribute of the CLIP latent space is the ability to perform smooth
interpolation between two sets of embeddings. We conducted experiments to
demonstrate $\lambda$-ECLIPSE’s ability to learn and replicate this smooth
latent space transition. We selected two distinct dog breeds ($<$dog1$>$,
$<$dog2$>$) and two types of hats ($<$hat1$>$, $<$hat2$>$) as the concepts.
$\lambda$-ECLIPSE was then used to estimate image embeddings for all four
possible combinations, each corresponding to the input phrase “a $<$dog$>$
wearing a $<$hat$>$.” Figure 8 showcases a gradual and seamless transition in
the synthesized images from the top left to the bottom right. Unlike current
diffusion models, which often exhibit sensitivity to input variations
necessitating numerous iterations for desired outcomes, $\lambda$-ECLIPSE
inherits CLIP’s smooth latent space. This inheritance not only facilitates
progressive changes in the conceptual domain but also extends the model’s
utility by enabling personalized multi-subject interpolations, a feature that
is not inherently available in previous P-T2I methods.

##  5 Conclusion

In this paper, we have introduced a novel diffusion-free methodology for
personalized text-to-image (T2I) applications, utilizing the latent space of
the pre-trained CLIP model with high efficiency. Our $\lambda$-ECLIPSE model,
trained on an image-text interleaved dataset of superior quality, marks a
significant advancement in the field. This model outperforms existing methods
by its capability to execute single-concept, multi-concept, and edge-guided
controlled P-T2I tasks using a singular model framework, while simultaneously
minimizing resource utilization. Notably, $\lambda$-ECLIPSE sets a new
benchmark in achieving state-of-the-art results in composition alignment,
coupled with exceptional maintenance of subject fidelity. Furthermore, our
research illuminates the potential of $\lambda$-ECLIPSE in exploring and
leveraging the smooth latent space. This capability unlocks new avenues for
interpolating between multiple concepts and their amalgamation, thereby
generating entirely novel concepts. Our findings underscore a promising
pathway where pre-trained diffusion image generators can be effectively
controlled without necessitating extra supervision.

## Acknowledgments

This work was supported by NSF RI grants #1750082, #2132724, and CPS grant
#2038666. The views and opinions of the authors expressed herein do not
necessarily state or reflect those of the funding agencies and employers.

## References

  * Alaluf et al. [2023] Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or.  A neural space-time representation for text-to-image personalization.  _ACM Transactions on Graphics (TOG)_ , 42(6):1–10, 2023.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Arar et al. [2023] Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H. Bermano.  Domain-agnostic tuning-encoder for fast personalization of text-to-image models.  In _SIGGRAPH Asia 2023 Conference Papers_ , pages 1–10, 2023.
  * Avrahami et al. [2023] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski.  Break-a-scene: Extracting multiple concepts from a single image.  _arXiv preprint arXiv:2305.16311_ , 2023.
  * Betker et al. [2023] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.  Improving image generation with better captions.  _Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf_ , 2:3, 2023.
  * Canny [1986] John F. Canny.  A computational approach to edge detection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , PAMI-8:679–698, 1986.
  * Chen et al. [2023a] Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu.  Disenbooth: Disentangled parameter-efficient tuning for subject-driven text-to-image generation.  _arXiv preprint arXiv:2305.03374_ , 2023a.
  * Chen et al. [2023b] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al.  Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.  _arXiv preprint arXiv:2310.00426_ , 2023b.
  * Chen et al. [2022] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen.  Re-imagen: Retrieval-augmented text-to-image generator.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Chen et al. [2023c] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming-Wei Chang, and William W Cohen.  Subject-driven text-to-image generation via apprenticeship learning.  _arXiv preprint arXiv:2304.00186_ , 2023c.
  * Ding et al. [2021] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  _Advances in Neural Information Processing Systems_ , 34:19822–19835, 2021.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Gal et al. [2023] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–13, 2023.
  * Gu et al. [2023] Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al.  Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models.  _arXiv preprint arXiv:2305.18292_ , 2023.
  * Han et al. [2023] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang.  Svdiff: Compact parameter space for diffusion fine-tuning.  _arXiv preprint arXiv:2303.11305_ , 2023.
  * Hu et al. [2021] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2021.
  * Jia et al. [2023] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su.  Taming encoder for zero fine-tuning image customization with text-to-image diffusion models.  _arXiv preprint arXiv:2304.02642_ , 2023.
  * Kim et al. [2023] Changhoon Kim, Kyle Min, Maitreya Patel, Sheng Cheng, and Yezhou Yang.  Wouaf: Weight modulation for user attribution and fingerprinting in text-to-image diffusion models.  _arXiv preprint arXiv:2306.04744_ , 2023.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1931–1941, 2023.
  * Li et al. [2023a] Dongxu Li, Junnan Li, and Steven CH Hoi.  Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing.  _arXiv preprint arXiv:2305.14720_ , 2023a.
  * Li et al. [2023b] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023b.
  * Li et al. [2023c] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22511–22521, 2023c.
  * Liu et al. [2023a] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023a.
  * Liu et al. [2023b] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao.  Cones: Concept neurons in diffusion models for customized generation.  _arXiv preprint arXiv:2303.05125_ , 2023b.
  * Liu et al. [2023c] Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao.  Customizable image synthesis with multiple subjects.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023c.
  * Ma et al. [2023a] Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu.  Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning.  _arXiv preprint arXiv:2307.11410_ , 2023a.
  * Ma et al. [2023b] Yiyang Ma, Huan Yang, Wenjing Wang, Jianlong Fu, and Jiaying Liu.  Unified multi-modal latent diffusion for joint subject and text conditional image generation.  _arXiv preprint arXiv:2303.09319_ , 2023b.
  * Pan et al. [2023] Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei.  Kosmos-g: Generating images in context with multimodal large language models.  _arXiv preprint arXiv:2310.02992_ , 2023.
  * Patel et al. [2023a] Maitreya Patel, Tejas Gokhale, Chitta Baral, and Yezhou Yang.  Conceptbed: Evaluating concept learning abilities of text-to-image diffusion models.  _arXiv preprint arXiv:2306.04695_ , 2023a.
  * Patel et al. [2023b] Maitreya Patel, Changhoon Kim, Sheng Cheng, Chitta Baral, and Yezhou Yang.  Eclipse: A resource-efficient text-to-image prior for image generations.  _arXiv preprint arXiv:2312.04655_ , 2023b.
  * Qin et al. [2023] Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al.  Unicontrol: A unified diffusion model for controllable visual generation in the wild.  _arXiv preprint arXiv:2305.11147_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Razzhigaev et al. [2023] Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, Vladimir Arkhipkin, Igor Pavlov, Ilya Ryabov, Angelina Kuts, Alexander Panchenko, Andrey Kuznetsov, and Denis Dimitrov.  Kandinsky: an improved text-to-image synthesis with image prior and latent diffusion.  _arXiv preprint arXiv:2310.03502_ , 2023.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2023a] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023a.
  * Ruiz et al. [2023b] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman.  Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.  _arXiv preprint arXiv:2307.06949_ , 2023b.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Shah et al. [2023] Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani.  Ziplora: Any subject in any style by effectively merging loras.  _arXiv preprint arXiv:2311.13600_ , 2023.
  * Shi et al. [2023] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung.  Instantbooth: Personalized text-to-image generation without test-time finetuning.  _arXiv preprint arXiv:2304.03411_ , 2023.
  * Smith et al. [2023] James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin.  Continual diffusion: Continual customization of text-to-image diffusion with c-lora.  _arXiv preprint arXiv:2304.06027_ , 2023.
  * Tewel et al. [2023] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon.  Key-locked rank one editing for text-to-image personalization.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 1–11, 2023.
  * Tumanyan et al. [2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1921–1930, 2023.
  * Voynov et al. [2023] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman.  $p+$: Extended textual conditioning in text-to-image generation.  _arXiv preprint arXiv:2303.09522_ , 2023.
  * Wei et al. [2023] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 15897–15907, 2023.
  * Xiao et al. [2023] Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han.  Fastcomposer: Tuning-free multi-subject image generation with localized attention.  _arXiv preprint arXiv:2305.10431_ , 2023.
  * Ye et al. [2023] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  _arXiv preprint arXiv:2308.06721_ , 2023.
  * Zhang et al. [2023a] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023a.
  * Zhang et al. [2023b] Yuxuan Zhang, Jiaming Liu, Yiren Song, Rui Wang, Hao Tang, Jinpeng Yu, Huaxia Li, Xu Tang, Yao Hu, Han Pan, et al.  Ssr-encoder: Encoding selective subject representation for subject-driven generation.  _arXiv preprint arXiv:2312.16272_ , 2023b.
  * Zhao et al. [2023] Ruoyu Zhao, Mingrui Zhu, Shiyin Dong, Nannan Wang, and Xinbo Gao.  Catversion: Concatenating embeddings for diffusion-based text-to-image personalization.  _arXiv preprint arXiv:2311.14631_ , 2023.

##  Appendix A Preliminaries for T2I Diffusion Models

As evidenced in numerous contemporary studies regarding T2I models, Stable
Diffusion (SD) has emerged as a predominant backbone for text-to-image (T2I)
models. SD involves training diffusion models in latent space, reversing a
forward diffusion process that introduces noise into the image. A notable
feature of SD is its integration of cross-attention, facilitating various
conditions like text input. Operating in VQVAE latent space, SD not only
achieves exceptional generative performance surpassing that in pixel space but
also significantly reduces computational demands.

UnCLIP models (such as DALL-E 2) are very similar to the Stable Diffusion. In
contrast, the UnCLIP stack takes the modular approach. UnCLIP first trains the
diffusion text-to-image to the image prior ($f_{\theta}$) to estimate the
image embeddings ($z_{x}$) from the text embeddings ($z_{y}$). In parallel, a
UNet-like diffusion image generator ($h_{\phi}$) is trained to generate images
($x$) conditioned on ground truth vision embeddings ($z_{x}$).

Traditionally, T2I prior is modeled to estimate $x_{0}$-prediction instead of
$\epsilon$-prediction. Given the forward function $z_{x}^{(t)}\sim
q(t,z_{x})$, the goal of $f_{\theta}$ is to directly estimate $z_{x}$ for all
timesteps $t\sim[0,T]$ as:

$\mathcal{L}_{prior}=\underset{\begin{subarray}{c}t\sim[0,T],\\\
z_{x}^{(t)}\sim
q(t,z_{x})\end{subarray}}{\operatorname{\mathbb{E}}}\Big{[}||z_{x}-f_{\theta}(z_{x}^{(t)},t,z_{y})||_{2}^{2}\Big{]}.$
(2)

ECLIPSE presents the contrastive learning strategy (Eq. 1) instead of
minimizing Eq. 2. The diffusion image generator is trained by following
standard $\epsilon$-prediction formulation. Here, $h_{\phi}$ will estimate the
ground truth added Gaussian noise $\epsilon\sim N(0,I)$, given the noise image
$X^{(t)}$ for all timesteps $t\sim[0,T]$ and input conditions (such as
$z_{x}$, $z_{y}$).

$\mathcal{L}_{decoder}=\underset{\begin{subarray}{c}\epsilon\sim N(0,I)\\\
t\sim[0,T],\\\
(z_{x},~{}z_{y})\end{subarray}}{\operatorname{\mathbb{E}}}\Big{[}||\epsilon-
h_{\phi}(x^{(t)},t,z_{x},z_{y})||_{2}^{2}\Big{]}.$ (3)

For models like Kandinsky v2.2, we drop the $z_{y}$ to condition the model on
$z_{x}$. Therefore, $\lambda$-ECLIPSE also only conditions the image
generation with $z_{y}$ in the prior stage.

##  Appendix B Image-Text Interleaved Training Details

#### Dataset Creation

In constructing the dataset, we adhered to the data processing pipeline of
Subject Diffusion [27]. We utilized the LAION-5B High-Res dataset, requiring a
minimum image size of 1024x1024 resolution. Original captions were replaced
with new captions generated by BLIP-2
(flan-t5-xl)333https://huggingface.co/Salesforce/blip2-flan-t5-xl. Subjects
were extracted using Spacy444https://spacy.io. For each subject, we identified
bounding boxes employing Grounding DINO [24], setting both box-threshold and
text-threshold values to 0.2. We retained images with 1 to 8 detected bounding
boxes, discarding the rest. Additionally, captions with multiple instances of
identical objects were filtered, allowing a maximum of 6 identical objects.
Following bounding box detection, individual subject masks were isolated using
Segment-Anything (SAM) [19]. To enhance the efficiency of the training
process, we pre-processed the dataset by pre-extracting features from CLIP
vision and text encoders. During this phase, images predominantly featuring a
background (white portion) exceeding 10% of the total area were excluded. We
preserved bounding boxes with a width-height ratio ranging from 0.08 to 0.7
and logit scores of at least 0.3. Masks constituting less than 40% of the
bounding box area were discarded. For the selection of subjects in images, we
constrained the range to 1-4 subjects per image, excluding those with more
than 4 subjects. At last, the interleaved image-text examples with respective
ground truth images are shown in Figure 9.

Figure 9:  Examples of image-text interleaved training data. Left column shows
the input the prior model and right images shows the ground truth
corresponding images. Note: these examples are generated from
$\lambda$-ECLIPSE for better interpretability.

#### Dataset Statistics

In the final analysis, our dataset comprised a total of 1,990,123 images. The
distribution of subjects per image exhibited a range from 1 to 4, with the
following breakdown: 1,479,785 images featuring one subject, 432,831 images
with two subjects, 65,597 images containing three subjects, and 11,910 images
showcasing four subjects. The overall count of unique subjects acquired from
this dataset amounted to 30,358. We partitioned our dataset into an 80:20
split between training and validation, reserving the remaining 1.6 million
images for training and the rest for validation.

Table 5:  The detailed overview of subject-driven text-to-image generative methodologies. * represents the backbone base models listed are subject to potential updates or modifications.  Method | Multi-Concept | Finetuning Free | Base Model | Image Input
---|---|---|---|---
Re-Imagen [9] | ✗ | ✓ | Imagen | Single
Textual Inversion [12] | ✗ | ✗ | SDv1.4 | Multiple
DreamBooth [38] | ✗ | ✗ | SDv1.4 | Multiple
Custom Diffusion [20] | ✓ | ✗ | SDv1.4 | Multiple
ELITE [48] | ✗ | ✓ | SDv1.4 | Single
E4T [13] | ✗ | ✗ | SD | Single
Cones [25] | ✓ | ✗ | SDv1.4 | Single
SVDiff [15] | ✓ | ✗ | SD | Multiple
UMM-Diffusion [28] | ✗ | ✓ | SDv1.5 | Single
XTI [47] | ✗ | ✗ | SDv1.4 | Multiple
Continual Diffusion [44] | ✓ | ✗ | - | Multiple
InstantBooth [43] | ✗ | ✓ | SDv1.4 | Multiple
SuTi [10] | ✗ | ✓ | Imagen | Multiple
Taming [17] | ✗ | ✓ | Imagen | Single
BLIP-Diffusion [21] | ✗ | ✓ | SDv1.5 | Single
Cones 2 [26] | ✓ | ✗ | SDv2.1 | Single
DisenBooth [7] | ✗ | ✗ | SDv2.1 | Single
FastComposer [49] | ✓ | ✓ | SDv1.5 | Single
Perfusion [45] | ✓ | ✗ | SDv1.5 | Multiple
Mix-of-Show [14] | ✓ | ✗ | Chilloutmix | Multiple
NeTI [1] | ✗ | ✗ | SDv1.4 | Mulitple
Break-A-Scene [4] | ✓ | ✗ | SDv2.1 |  Single*
ViCo [46] | ✗ | ✗ | SDv1.4 | Mulitple
Domain-Agnostic [3] | ✗ | ✗ | - | Single
Subject-Diffusion [27] | ✓ | ✓ | SDv2 | Single
HyperDreamBooth [39] | ✗ | ✗ | SDv1.5 | Single
IP-Adapter [50] | ✗ | ✓ | SDv1.5 | Single
Kosmos-G [29] | ✓ | ✓ | SDv1.5 | Single
Zip-LoRA [42] | ✓ | ✗ | sdxl | Multiple
CatVersion [53] | ✗ | ✗ | SDv1.5 | Multiple
SSR-Encoder [52] | ✓ | ✓ | SDv1.5 | Single
$\lambda$-ECLIPSE (ours) | ✓ | ✓ | Kv2.2 | Single

##  Appendix C Implementation Details

The $\lambda$-ECLIPSE transformer prior architecture is significantly more
compact compared to other Text-to-Image (T2I) methodologies. Our model employs
a configuration of 16 Attention Heads, each with a dimension size of 32,
alongside a total of 10 layers. Additionally, the embedding dimension size for
our model is set at 1280, supplemented by 4 auxiliary embeddings (including,
one for canny edge map). As $\lambda$-ECLIPSE is not a diffusion prior model,
we do not keep time embedding layers. Overall, the $\lambda$-ECLIPSE model
comprises approximately 34 million parameters, establishing it as a
streamlined yet effective solution for Personalaized-T2I. Notably, the
standard UnCLIP T2I priors contain 1 billion parameters.

##  Appendix D Extended P-T2I Baselines Comparison

We further expand our comparative analysis of P-T2I methods encompassing a
total of 32 approaches including ours and parallel works. Table 5 summarizes
them into four crucial aspects: 1) multi-concept support, 2) fine-tuning free,
3) base model types, and 4) the required number of input images. To summarize,
$\lambda$-ECLIPSE is the only methodology built on top of the UnCLIP models
while supporting multi-concept personalization, fine-tuning free, and only
requires a single reference concept image. We detail the comparison below:

#### Multi-Concept Generation.

Multi-concept generation enables users to integrate multiple personal subjects
to generate an image that follows the text prompts and aligns with all the
concept visuals. Of the 32 methods analyzed, 14 offer this capability. Ten
methods require separate training for each concept and then an additional
fusing step for combining the learned concepts. (Mix-of-Show). While only four
methods support fast multi-concept personalization. Only a few can learn
auxiliary guided information such as canny edge, depth maps, or open-pose and
adapt style variation.

#### Fine-tuning Free (Fast Personalization).

Many models require test-time fine-tuning. Each varies on which part
alteration occurs, as early models tend to modify the whole UNet. In contrast,
recent models tune a small portion of the cross-attention layers or introduce
additional layers performing as adapters. In our analysis of P-T2I models, 13
out of 32 models employ a fine-tuning free approach which enables fast
personalization.

#### Diffusion Independent.

A majority of the reviewed models utilize diffusion models, with Stable
Diffusion being the predominant choice, spanning versions 1.4, 1.5, 2.1, and
XL. Few adapt Imagen (SuTi, Taming) and interestingly Mix-of-show employs
ChillOutMix as their pre-trained model, known for its adeptness at preserving
realistic concepts like human faces. A unique outlier in this landscape is our
$\lambda$-ECLIPSE, the only one that eschews the use of any diffusion prior
model.

#### Easiness to Use (Input and Output).

Ease of use, particularly regarding model input and output, significantly
impacts user convenience in subject-driven generalization. A more user-
friendly model typically requires a single input image, as opposed to multiple
images of the same subject. In our study, 18 models offer P-T2I capabilities
with just one input image. In contrast, others often require 4 to 5 images of
the subject. Additionally, some models necessitate storage space for learned
concepts, ranging from a few hundred kilobytes (e.g., Perfusion,
HyperDreamBooth) to several megabytes (e.g., Zip-LoRA). Our model stands out
by eliminating the need for individual concept pre-learning or storing any
artifacts for P-T2I utilization, offering a streamlined, efficient user
experience.

##  Appendix E More Qualitative Results

In this section, we present an array of extended examples of P-T2I generation,
which presents complex compositions that are hard to achieve in
$\lambda$-ECLIPSE as well as other models. Figure 10 showcases the complexity
level that notably escalates as it goes from top to bottom. As the complexity
of the visual concepts increases, we observe that all methodologies perform
poorly in maintaining the subject-fidelity. This applies to $\lambda$-ECLIPSE
as well. Interestingly, $\lambda$-ECLIPSE retains the compositional
understanding while baselines perform poorly in all scenarios.

In addition, we demonstrate some examples of how P-T2I methods exhibit
inconsistency in their generated results depending on each trial. Figure 11
exhibits consistency in single and multiple concept generation across both
models. However, Kosmos-G exhibits inconsistency in generating multiple
concepts, sometimes placing the Ironman suit in various parts of the dog or
omitting it entirely. This also means that $\lambda$-ECLIPSE reduces the
diversity of the generated images to improve the consistency. However, this
behavior is observed across the UnCLIP family of models.

In our final exploration, we examine the preservation of actual human facial
characteristics in target scenarios, combined with varying captions. While
each model attempts to retain the original face, none can precisely replicate
the exact personal facial features. These examples generally fail to
accurately represent the intended compositions, except in one instance in IP-
Adapter FaceID.

##  Appendix F Limitations

Our work represents a pioneering effort in harnessing a method that utilizes
the latent space of pre-trained CLIP models for P-T2I generation, it is
important to acknowledge certain limitations. Firstly, CLIP, despite its
capabilities, does not perfectly capture hierarchical representations, and
this limitation can occasionally result in suboptimal outcomes. Consequently,
the CLIP contrastive loss can cause the model to deviate from the original
subject features, making performing P-T2I on complex concepts, such as human
faces, more challenging. We anticipate that the enhancement of CLIP
representations will correspondingly improve the performance of our framework
in the realm of P-T2I mapping. The $\lambda$-ECLIPSE model has been trained
with 34 million parameters and 1.6 million images. However, increasing the
data quality and parameters might lead to improvements in the results.

##  Appendix G Broader Impact

The recent retraction of the LAION-5B dataset, prompted by concerns over the
presence of data samples potentially breaching ethical guidelines, has
significantly affected various research initiatives. This situation
underscores the crucial importance of careful dataset selection and raises a
cautionary note for users about the ethical dimensions involved. In line with
this, the community needs caution in the application of associated
methodologies as well. Subject-driven image generation or Personalized Text-
to-Image (P-T2I) methods have the potential to be a transformative tool in
numerous domains.

For their positive influence, they enable users to effortlessly generate,
modify, and synthesize original subjects into diverse environments, thereby
enriching creative expression. On the other hand, the ease of altering and
creating images raises concerns about the responsible use of this technology
which requires significant ethical and legal considerations. We recommend
developers provide a more secure way (such as image attribution) for end-users
to ensure accountability for misuse of such models [18]. Users must be acutely
aware of being able to infringe intellectual property rights and create
misleading or harmful content. As such, those employing subject-driven image-
generation techniques should exercise careful judgment, ensuring that their
work adheres to ethical standards and legal boundaries. It is imperative that
the broader implications of this technology are considered, and that a
commitment to responsible and conscientious use guides its application.

Figure 10:  Qualitative examples of increasing complexity of novel visual
concepts as we move from top to bottom.  Figure 11:  Qualitative examples of
showcasing the consistency comparisons between Kosmos-G and $\lambda$-ECLIPSE.
Figure 12:  Qualitative examples of showcasing the failure cases on human
faces on Kosmos-G, IP-Adapter (SDXL), IP-Adapter (FaceID), and
$\lambda$-ECLIPSE.


# Anomalous conductivity due to relativistic Landau quantization

Gert Brodin  Department of Physics, Umeå University, SE–901 87 Umeå, Sweden
<EMAIL_ADDRESS>Haidar Al-Naseri  Stanford PULSE Institute, SLAC National
Accelerator Laboratory, Menlo Park, California 94025, USA
<EMAIL_ADDRESS>
###### Abstract

We use a recently developed kinetic model derived from the Dirac equation, in
order to study electromagnetic wave propagation in superstrong magnetic
fields, such as in magnetars, where relativistic Landau quantization is
prominent. The leading contribution to the conductivity tensor in such a
plasma is calculated. It is found that the electron Hall current has an
anomalous contribution, in the quantum relativistic regime, where the
effective particle energy has a significant contribution from the diamagnetic
and Zeeman energy. As a result, a new quantum resonance frequency appears, and
the dispersion relation for the left- and right-hand polarized modes are
strongly modified for long and moderate wavelengths. The implications for
magnetar physics are discussed.

###### pacs:

52.25.Dg, 52.27.Ny, 52.25.Xz, 03.50.De, 03.65.Sq, 03.30.+p

##  I Introduction

As is well-known, letting the electromagnetic field strength approach the
critical field, opens for a host of new phenomena. For example, aided by a
strong electric field, the Schwinger mechanism allows virtual electron-
positron pairs to tunnel out of the Dirac sea and become real [1, 2]. It may
be noted, however, that field strengths of this magnitude cannot yet be
produced in the laboratory. In an astrophysical context, on the other hand,
magnetic fields of the order of the critical field, and sometimes larger, are
known to exist in magnetars [3, 4, 5]. Also in this case, physics induced by
the virtual particles can be crucial, as the vacuum polarization induced by
the ultra-strong magnetic fields is believed to be the reason for the observed
polarization of electromagnetic radiation emitted from magnetars.
Specifically, for photons propagating perpendicular to the magnetic field, the
vacuum polarization leads to so-called photon-splitting, implying that photons
in one polarization state can decay into photons with a lower frequency
belonging to the opposite polarization state [6, 3]

With real particles present and affected by the magnetic field, either at the
magnetar surface or in a magnetar accretion disc, the ultrastrong magnetic
field will lead to relativistic Landau quantization [7, 4, 8, 9, 10] of
electrons. In a laboratory context, Landau quantization is known to be a key
feature in the quantum hall effect of the two-dimensional electron gas [11],
where the transverse resistivity becomes quantized. However, for the case
where electrons are subject to relativistic Landau quantization, less is
known. While many works have treated certain aspects of relativistic Landau
quantization [5, 8, 9], e.g. computed the energy and effective mass of
electrons in different Landau states, rigorous treatments of the current
response due to an electric field have not been made. On the other hand, when
studying electromagnetic wave propagation in magnetar environments, several
authors (e.g. [12, 13, 14, 15, 16]) have stressed the significance of the
strong field vacuum polarization, but at the same time evaluated the current
response of the plasma based on classical theory [12, 15, 14], or semi-
classical models [13, 16], not fully accounting for the dynamics of the
perturbed eigenstates. There is a reasonable justification for this approach
since a strong magnetic field mainly influences the perpendicular current
response, which tends to be rather effectively suppressed for electrons due to
their shorter Larmor radius. Hence there is room for a classical ion response
to dominate the perpendicular current. Moreover, the parallel electron current
tends to be classical as a rough approximation, since it is only marginally
affected by the magnetic field. While there is a fair bit of merit to this
description, nevertheless we will show that the picture given above is an
over-simplification

In the present work, we will use a recently developed kinetic approach [4] in
order to evaluate the electron conductivity in a relativistically Landau
quantized plasma, of particular relevance for magnetar environments. We find
that the quantum relativistic features induce crucial deviations from the
classical conductivity. For certain cases, the idea that relativistically
Landau quantized electrons behave almost like classical particles, but with an
effective mass dependent on the energy state [5, 8, 9], can be confirmed.
However, for other equally common cases, e.g., electromagnetic waves
propagating parallel to the magnetic field, the deviations from classical
theory turn out to be dramatic. Specifically, both the right and left-hand
circular polarized modes behave much differently from classical theory. In
this case, one of the wave frequencies approaches zero as the wavenumber
$k\rightarrow 0$, and the other approaches a new type of quantum resonance
frequency $\omega_{\rm res}$ given by $\omega_{\rm
res}\sim(\hbar\omega_{c}/mc^{2})(\hbar\omega_{p}/mc^{2})\omega_{p}$, where
$\hbar$ is the reduced Planck constant, $m$ is the electron mass, $c$ is the
speed of light in vacuum, $\omega_{c}$ is the electron cyclotron frequency,
and $\omega_{p}$ is the electron plasma frequency. The present results are a
prerequisite for understanding the dynamics of relativistically Landau
quantized states, as well as for interpreting spectra emitted from magnetars.
[17, 5, 18, 19, 20, 21, 22, 7, 14]

##  II Basic equations

In this letter, our starting point will be the kinetic theory derived in Ref.
[4]

$\partial_{t}W_{\pm}+\frac{1}{\epsilon}\mathbf{p}\cdot\nabla_{r}W_{\pm}+q\mathbf{E}\cdot\nabla_{p}W_{\pm}+\frac{1}{\epsilon}\mathbf{p}\times\mathbf{B}\cdot\nabla_{p}W_{\pm}$
(1)

This evolution equation is very similar to the usual relativistic Vlasov
equation. However, in order to account for relativistic Landau quantization,
the particle energy $\epsilon$ is generalized to be a momentum operator with

$\epsilon=\sqrt{m^{2}+p^{2}\pm
2\mu_{B}B_{0}-\mu_{B}^{2}B_{0}^{2}(\mathbf{\hat{z}}\times\nabla_{p})^{2}}$ (2)

in units where $c=1$. Here the index $\pm$ on the distribution function
(Wigner function [23]) refers to the particle spin state, up or down relative
to the external magnetic field $\mathbf{B}_{0}$, and $\mu_{B}=e\hbar/2m$ is
the Bohr magneton The kinetic evolution equation is combined with Maxwell’s
equations, where the current and charge densities are given as

$\displaystyle{\bf j}=\sum_{\pm}q\int\frac{1}{\epsilon}({\bf p}W_{\pm})d^{3}p$
(3)

and

$\displaystyle\rho=\sum_{\pm}q\int W_{\pm}d^{3}p$ (4)

The above kinetic theory is based on a Foldy-Wouthuysen transformation [24,
25] of the Dirac Hamiltonian, separating the positive and negative energy
states. The model is applicable for an ultrastrong and homogeneous background
magnetic field, $\mathbf{B}_{0}=B_{0}\mathbf{\hat{z}}$ , allowing for magnetic
fields $B_{0}\sim B_{cr}$ that can be found in magnetars, where the critical
field strength is $B_{cr}=m^{2}c^{3}/|q|\hbar$. Perturbations of the
electromagnetic field added to $\mathbf{B}_{0}$ must still be small compared
to the critical field, though, as the electron and positron states cannot be
separated otherwise. The model is designed to fully include the effect of
relativistic Landau quantization, but ignore other quantum effects, except
degeneracy, that might be included by simply picking a degenerate initial
state. Ignoring dynamical quantum effects, besides those due to Landau
quantization, is possible if the relevant dimensionless parameters involving
$\hbar$ are small, excluding $\mu_{B}B_{0}/m\sim 1\,$ which is allowed. To be
concrete, we thus assume $\hbar k\ll p_{th}$(where $p_{th}$ is characteristic
thermal momentum such that particle dispersive effects can be ignored), $\hbar
k^{2}/m\omega\ll 1$ (making the magnetic dipole force due to the spin small),
together with $\hbar k\ll m$ and $\hbar\omega\ll m$ such that spin-orbit
coupling [26, 27]and other quantum relativistic effects, except those due to
the strong Landau quantization can be dropped. In the above, $\omega$ and $k$
represent characteristic frequency and wavenumber scales of the macroscopic
variables.

The physics behind the model is that in a superstrong magnetic field, the
magnetic dipole energy due to the spin and the energy associated with the
orbital angular momentum gives a significant contribution to the particle
energy and thereby to the effective gamma factor $\gamma=\epsilon/m$. This is
accompanied by a relativistic momentum spread in the energy eigenstates. In
the expression (2), the term $\pm 2\mu_{B}B_{0}$ is the energy contribution
from the Zeeman energy due to the spin, whereas the term
$\mu_{B}^{2}B_{0}^{2}(\mathbf{\hat{z}}\times\nabla_{p})^{2}$ gives the orbital
diamagnetic energy contribution. When the distribution function is in a Landau
quantized energy eigenstate, $W_{\pm}=W_{\pm n}$, we have $\epsilon W_{\pm
n}=$ $\sqrt{m^{2}+p_{z}^{2}+\mu_{B}B_{0}(\pm 1+n)}$ $W_{\pm n}$ (see Eqs.
(34)-(35) of Ref. [4] for the expression for the energy eigenstate $W_{\pm
n}$). However, this particular expression is of use mostly to evaluate
$\epsilon$ when acting on the time-independent background, that can be written
as a sum of energy eigenstates. For a perturbation of the background, which in
general depends on the azimuthal momentum coordinate, using cylindrical
coordinates in momentum space, the distribution function cannot be written as
a sum over energy eigenstates. To evaluate the operator $\epsilon$ in this
case, we must use the defining expression, where the root in the energy
expression is Taylor expanded to infinite order,

$\epsilon=m(1+\frac{1}{2}\frac{p^{2}\pm
2\mu_{B}B_{0}-\mu_{B}^{2}B_{0}^{2}(\mathbf{\hat{z}}\times\nabla_{p})^{2}}{m}+...).$
(5)

Naturally, the same definition of $\epsilon$ applies also when acting on the
energy eigenstates, but in this case, the infinite series sums up to a simple
energy eigenvalue. In all equations written, note that $\epsilon\ $is acting
on all momentum-dependence that stands to the right, e.g. in the last term of
(1), $\epsilon$ does not act only on $\nabla_{p}W_{\pm}$, but also on the
momentum in the cross-product $\mathbf{p}\times\mathbf{B}$, and similarly for
the energy operator $\epsilon$ that appears in the current density (3) to be
used in Ampere’s law.

##  III Linear theory

Next, we use Eq. 1 to study linear waves in a magnetized plasma. We divide the
variables according to $W=W_{0}(\mathbf{p})+W_{1}(\mathbf{r},\mathbf{p},t)$
and $\mathbf{B}=\mathbf{B}_{0}+\mathbf{B}_{1}(\mathbf{r},t)$. Furthermore, we
use a plane wave ansatz
$W_{1}(\mathbf{r},\mathbf{p},t)=\tilde{W}_{1}(\mathbf{p})e^{i(\mathbf{k}\cdot\mathbf{r}-\omega
t)}$. Using the ansatz and linearizing Eq. 1, we obtain

$\Big{[}-i\omega+\frac{\mathbf{k}}{\epsilon}\cdot\mathbf{p}+\frac{qB_{0}}{\epsilon}\Big{]}\tilde{W}_{1}=-q\Big{[}\mathbf{E}+\frac{1}{\epsilon}\mathbf{p}\times\mathbf{B}_{1}\Big{]}\cdot\nabla_{p}W_{0}$
(6)

Due to the energy $\epsilon$ being a momentum operator, the standard
techniques that work for the linearized Vlasov equation are not directly
applicable. However, the equation can be solved using an expansion of
$\tilde{W}_{1}$ according to

$\tilde{W}_{1}=\sum_{n=-\infty}^{\infty}\sum_{m=0}^{\infty}g_{nm}(p_{\bot},p_{z})e^{in\varphi_{p}}\Big{(}\frac{k_{\bot}p_{\bot}}{m\omega_{c}}\Big{)}$
(7)

In principle, we can include terms to an arbitrary order in the summation over
$n$ and $m$. However, in the strongly magnetized regime that we consider
(where all other frequencies are small compared to the electron cyclotron
frequency), only the low values of $n$ and $m$ will give a significant
contribution. Moreover, without loss of generality we let the wavevector be
directed in the $xy$-plane, i.e. ${\bf k}=k_{x}{\hat{\bf x}}+k_{z}{\hat{\bf
z}}$, such that $k_{\perp}=k_{x}$. Using the expansion (7), we solve for
$\tilde{W}_{1}$ to leading order, including terms up to $n=\pm 1$ and $m=1$.
Combining the solution for $\tilde{W}_{1}$ with Ampere’s law, we obtain the
dispersion relation for electromagnetic wave propagation (including also the
electrostatic limiting cases) in a magnetized plasma, extending previous
classical results [28] to account for relativistic Landau quantization. The
dispersion relation can be written as $\det D_{ij}=0$, where the matrix is
written as

$D_{ij}=\delta_{ij}\bigg{(}1-\frac{k^{2}c^{2}}{\omega^{2}}\bigg{)}+\frac{k_{i}k_{j}c^{2}}{\omega^{2}}+\chi_{ij}$
(8)

where $\delta_{ij}$ comes from the displacement current in Ampere’s law, and
the terms proportional to the wavenumber comes from the curl of the magnetic
field. The last term, $\chi_{ij}\,,$ is the plasma susceptibility and is
related to the plasma currents as
$\chi_{ij}=\sigma_{ij}/i\omega\varepsilon_{0}$, where $\sigma_{ij}$ is the
conductivity tensor determined from the kinetic evolution equation. With the
wavevector in the xz-plane, the expression for $D_{ij}$ becomes:

$D_{ij}=\left[\begin{array}[]{ccc}1-\frac{k_{z}^{2}c^{2}}{\omega^{2}}+\chi_{xx}&\chi_{yx}&\frac{k_{x}k_{z}c^{2}}{\omega^{2}}+\chi_{zx}\\\
\chi_{yx}^{\ast}&1-\frac{k^{2}c^{2}}{\omega^{2}}+\chi_{yy}&\chi_{yz}\\\
\frac{k_{x}k_{z}c^{2}}{\omega^{2}}+\chi_{zx}^{\ast}&\chi_{yz}^{\ast}&1-\frac{k_{x}^{2}c^{2}}{\omega^{2}}+\chi_{zz}\end{array}\right]$
(9)

where

$\displaystyle\chi_{xy}=\chi_{yx}^{*}$ $\displaystyle=-\frac{i\omega
q2\pi}{4}\int d^{2}p\frac{1}{\epsilon}\frac{\pm p_{\bot}}{\omega-
kp_{z}/\epsilon^{\prime}\mp qB_{0}/\epsilon^{\prime}}\bigg{[}\frac{\partial
f_{0}}{\partial
p_{\bot}}-\frac{k_{z}}{\omega}\frac{1}{\epsilon^{\prime}}\Big{(}p_{z}\frac{\partial
W_{0}}{\partial p_{\bot}}-p_{\bot}\frac{\partial W_{0}}{\partial
p_{z}}\Big{)}\bigg{]}$ (10) $\displaystyle\chi_{yz}=\chi_{zy}^{*}$
$\displaystyle=\frac{i\omega q2\pi}{4}\int d^{2}p\frac{1}{\epsilon}\frac{\mp
p_{\bot}}{\omega-kp_{z}/\epsilon^{\prime}\mp
qB_{0}/\epsilon^{\prime}}\bigg{[}\frac{k_{z}}{\omega}\frac{1}{\epsilon^{\prime}}\Big{(}p_{z}\frac{\partial
W_{0}}{\partial p_{\bot}}-p_{\bot}\frac{\partial W_{0}}{\partial
p_{z}}\Big{)}\bigg{]}$ (11)

The given components for the electron susceptibility constitute the dominating
contributions for electrons in the regime $\omega\ll\omega_{c}=qB_{0}/m\sim
qB_{0}/\epsilon^{\prime}$. Accordingly, we should replace the denominators in
the expressions above according to $\omega-kp_{z}/\epsilon^{\prime}\mp
qB_{0}/\epsilon^{\prime}\rightarrow qB_{0}/\epsilon^{\prime}$. For consistency
with other approximations, this should be done whenever applying the
expressions. The only reason to show the formula without this simplification
is to emphasize the similarity of the quantum result with previous well-known
classical formulas, when the arguments of the Bessel functions are small [28].
The expression for $\chi_{zz}$ is not given above, but we note that this term
has been computed previously, see Ref. [4]. Contrary to the terms given here,
the component for $\chi_{zz}$ can essentially be written as for a classical
plasma, except that each Landau-quantized energy eigenstate behaves like a
separate particle species, which contributes to the plasma frequency with its
own effective mass, see Ref. [4] for details.

The remaining susceptibility components for electrons are approximated with
zero, since in the regime $\omega\ll\omega_{c}$ studied here, those components
will always be negligible compared to the ion contribution. While the ion
contribution has not been written out, a contribution from the ions can always
be added to give the total susceptibility. The ions susceptibility will
typically be given by the classical textbook results (see e.g. Ref. [28]),
since even for magnetar field strengths $\hbar\omega_{ci}\ll k_{B}T$, where
$\omega_{ci}$ is the ion cyklotron frequency, $k_{B}$ is the Boltzmann
constant and $T$ is the temperature (the condition for neglecting ion Landau
quantization) tend to apply.

##  IV The linear Dispersion relation

Let us next consider the dispersion relation for electromagnetic waves
propagating parallel to the external magnetic field. Although we are
interested in the case of ultra-strong magnetic fields, approaching the
critical field, in order to obtain simple analytical results, we will make an
expansion in the parameter $\mu_{B}B_{0}/m\ll 1$. For $\omega/\omega_{ce}\ll
1\,$, we keep terms to first order in $\omega/\omega_{ce}$, but allow for
$\omega/\omega_{ci}\sim 1$. Moreover, we keep correction terms to second order
in $\mu_{B}B_{0}/m$. By considering an electron-ion plasma, and evaluating the
ion current in the classical cold limit, we find that the total (including
electrons and ions) susceptibility is given by
$\chi_{yx}=\chi_{yx,e}+\chi_{yx,i}$ with the electron contribution

$\chi_{yx,e}=\frac{\omega_{pe}^{2}}{\omega_{ce}\omega}\left[1+\frac{3}{2}\left(\frac{\mu_{B}B_{0}}{m}\right)^{2}\right]$
(12)

and the ion contribution

$\chi_{yx,i}=\frac{\omega_{pi}^{2}\omega_{ci}}{\omega_{ci}^{2}-\omega^{2}}$
(13)

such that the total susceptibility is

$\displaystyle\chi_{yx}$ $\displaystyle=$
$\displaystyle\frac{\omega_{pe}^{2}}{\omega_{ce}\omega}\left[\frac{\omega^{2}}{\omega^{2}-\omega_{ci}^{2}}+\frac{3}{2}\left(\frac{\mu_{B}B_{0}}{m}\right)^{2}\right]$
(14)

Consequently, for reasonably low frequencies, $\omega\ll\omega_{ci},$ the
second term $\propto(\mu_{B}B_{0}/m)^{2}$, which is the quantum contribution
associated with the electron Landau quantization, will dominate for magnetic
fields not too much smaller than the critical field. Calculating the
determinant of the dispersion matrix $D_{ij}$ for $k_{\bot}=0$, we obtain the
dispersion relation

$\left(1-\frac{k^{2}c^{2}}{\omega^{2}}-\frac{\omega_{pi}^{2}}{\omega^{2}-\omega_{ci}^{2}}\right)=\pm\frac{3}{2}\frac{\omega_{pe}^{2}}{\omega_{ce}\omega}\left(\frac{\mu_{B}B_{0}}{m}\right)^{2}$

where the two signs correspond to the left and right hand circularly polarized
modes, respectively. Naturally, the dispersion relation can be studied in all
generality. However, to illustrate the main new features we limit ourselves to
frequencies well below the ion-cyclotron frequency, in which case we obtain

$\left(1-\frac{k^{2}c^{2}}{\omega^{2}}+\frac{\omega_{pi}^{2}}{\omega_{ci}^{2}}\right)=\pm\frac{3}{2}\frac{\omega_{pe}^{2}}{\omega_{ce}\omega}\left(\frac{\mu_{B}B_{0}}{m}\right)^{2}$
(15)

In the short-wavelength limit, the quantum term becomes negligible, and we get

$\omega^{2}=\frac{k^{2}c^{2}}{1+\omega_{pi}^{2}/\omega_{ci}^{2}}$ (16)

for both the left and right hand circularly polarized mode. For $\omega$
approaching $\omega_{ci}$, naturally we should avoid the low-frequency
simplification and the left and right hand modes will have different
frequencies also in the classical limit. The solution to 15 is

Figure 1: The frequency $\omega$ is plotted as a function of the wave vector
$k$ for $\mu_{B}B_{0}/m=0.2$ and $\omega_{c}/\omega_{p}=10$.

$\displaystyle\omega=-\frac{3\omega_{pe}^{2}}{4\omega_{ce}\left(1+\omega_{pi}^{2}/\omega_{ci}^{2}\right)}\left(\frac{\mu_{B}B_{0}}{m}\right)^{2}$
$\displaystyle\pm\sqrt{\left[\frac{3\omega_{pe}^{2}}{4\omega_{ce}\left(1+\frac{\omega_{pi}^{2}}{\omega_{ci}^{2}}\right)}\left(\frac{\mu_{B}B_{0}}{m}\right)^{2}\right]^{2}+\frac{k^{2}c^{2}}{\left(1+\frac{\omega_{pi}^{2}}{\omega_{ci}^{2}}\right)}}$
(17)

Thus in the long wavelength limit, we have a new type of quantum resonance
frequency $\omega_{q}$, such that when $k\rightarrow 0$, for the left hand
circularly polarized mode we get $\omega\rightarrow\omega_{q}$ with

$\omega_{q}=-\frac{3\omega_{pe}^{2}}{2\omega_{ce}\left(1+\omega_{pi}^{2}/\omega_{ci}^{2}\right)}\left(\frac{\mu_{B}B_{0}}{m}\right)^{2}$
(18)

whereas the right-hand circularly polarized mode fulfills

$\omega\simeq\frac{k^{2}c^{2}}{4\omega_{q}\left(1+\omega_{pi}^{2}/\omega_{ci}^{2}\right)}$
(19)

for long wavelengths fulfilling
$\frac{k^{2}c^{2}}{\left(1+\omega_{pi}^{2}/\omega_{ci}^{2}\right)}\ll\omega_{q}^{2}$.
We thus see that for a sufficiently strong magnetization of the plasma, when
relativistic Landau quantization is significant, both the left and right- hand
circularly polarized modes show a distinct quantum behavior for long
wavelengths, see Fig. 1 where the left- and right-handed solutions to Eq. 15
are compared with the classical limit. For shorter wavelengths,
$\frac{k^{2}c^{2}}{\left(1+\omega_{pi}^{2}/\omega_{ci}^{2}\right)}\gg\omega_{q}^{2}$
not covered in Fig. 1 , the quantum behavior is suppressed, and we recover the
classical limit. The wavelength where the transition from quantum to classical
behavior takes place depends on the plasma density. For the high densities at
the surface of magnetars, the plasma frequency is of the order
$\omega_{pe}\sim 10^{19-20}s^{-1}$. In this case, for magnetars with surface
magnetic fields slightly below the critical field, the quantum limits of the
dispersion relation shown above apply for wavelengths in the UV region and
longer.

##  V Discussion and conclusion

As is known from condensed matter physics, and seen in the quantum Hall
effect, the effect of Landau quantization may induce peculiar properties of
the conductivity. In condensed matter systems, this may happen already in a
non-relativistic description. In that case, besides Landau quantization, the
special quantum features are dependent on the system being a 2D electron gas.
In the present theory, the electrons are not limited to two dimensions, but
instead, the relativistic aspects of Landau quantization are crucial,
accounting for the contribution to the electron’s effective mass through the
diamagnetic and Zeeman energies. At first glance, the extra quantum
contributions to the Hall term of the electron susceptibility (leading to the
quantum contribution in the total susceptibility (14), i.e., the second term
$\propto(\mu_{B}B_{0}/m)^{2}$) may be expected. Why would the susceptibility
(conductivity) be unaffected when the Zeeman energy becomes significant?
However, looking more carefully at the result, it appears to challenge a key
principle of physics, namely Lorentz invariance.

Normally, in the limit when $k\rightarrow 0$ and $\omega\rightarrow 0$, the
total current density of all species cancels. The reason is as follows: For an
electromagnetic field that is effectively static and homogenous, in a
reference system moving with the $\mathbf{E\times B}-$drift, where
$\mathbf{v}_{E}=\mathbf{E\times B/}B^{2}$, particles will only feel a static
magnetic field. Hence, independent of species, there will only be a gyration
and no net-drift in this system. This corresponds to all species $s$ having
the current density $q_{s}n_{0s}\mathbf{v}_{E}$ in the original system. Due to
the background neutrality of all species, this implies a vanishing total
current density. Fundamentally, a vanishing current density in the limit
$k\rightarrow 0$ and $\omega\rightarrow 0$ is a classical result that follows
from Lorentz invariance. In the present case, while our governing equations do
not show Lorentz invariance directly (since they are built around a preferred
reference system subject to a strong magnetic field $B_{0}$), we should still
demand our results to be consistent with Lorentz invariance. Clearly, since
the model is derived from the Dirac equation and Maxwell’s equations, we
should not have a theory that is in conflict with the basic principles of
special relativity. Thus the question arises, how can the effective electron
drift velocity deviate from the $\mathbf{E\times B}-$drift in the limit
$k\rightarrow 0$ and $\omega\rightarrow 0$, without contradicting Lorentz
invariance?

A first observation is that, contrary to classical theory, it is no longer
meaningful to think of individual velocities for different phase space
elements, represented by $\mathbf{p}/\varepsilon$. In the limit where this
would be accurate, the quantum contribution to $\chi_{xy}$ vanish, and the
current density for electrons indeed become $q_{e}n_{0e}\mathbf{v}_{E}$. A
complication of the present theory, where the energy $\varepsilon$ is a
momentum-operator, is that the effective velocity of the theory depends on the
behavior in a region of momentum space, rather than in a single point.
Moreover, the effective velocity averaged over all momentum for energy
transport ($\mathbf{v}_{en}=\int\frac{1}{\varepsilon}(\mathbf{p}\varepsilon
f)d^{3}p/\int\varepsilon fd^{3}p$) and for particle transport
($\mathbf{v}_{p}=\int\frac{1}{\varepsilon}(\mathbf{p}f)d^{3}p/\int f$ ) only
coincide if the distribution function $f$ is an energy eigenstate, in general
$\mathbf{v}_{en}\neq$ $\mathbf{v}_{p}$. However, even when this ambiguity
regarding what constitutes the velocity is noted, one can argue that in a
reference system with a static magnetic field, there should be no net current
density. If this were true, one would obtain the same cancellation of the
total current density as in the classical case.

This argument leaves out a key aspect of the present theory, though. When a
Landau quantized eigenstate in a pure magnetic field is exposed to an electric
field, for the particle to move at all, the momentum distribution of the
particles need to be modified in a way that prevents the particle state from
being written as a sum of energy eigenstates (this is not possible, as we
break the angular symmetry and get a dependence on $\varphi_{p}$). Breaking
the original symmetry contributes to the energy by a positive and quantized
amount, since for the perturbed distribution function we must let
$B_{0}^{2}\left(\frac{\partial^{2}}{\partial\rho^{2}}+\frac{1}{\rho}\frac{\partial}{\partial\rho}\right)\rightarrow
B_{0}^{2}\left(\frac{\partial^{2}}{\partial\rho^{2}}+\frac{1}{\rho}\frac{\partial}{\partial\rho}+\frac{1}{\rho^{2}}\right)$
in the orbital magnetic energy part of the energy operator. This quantized
change of energy when modifying the particle state has no classical
correspondence and is what determines the momentum-velocity relation when
calculating the magnitude of the perturbed distribution function. Importantly,
the effective velocity obtained in this step is not the same as the effective
velocity that we get when calculating the current density. Here, for the part
surviving the momentum integration, the energy operator acts on a function of
momentum that is angular symmetric, which leads to a different expression of
what can be interpreted as the effective velocity. Ultimately, in the
relativistically Landau quantized regime, what constitutes ”particle velocity”
depends on the details of the definition. As a result, a ratio of different
particle energies, which would simply be unity in a classical calculation, is
what induces a relativistically quantum-corrected Hall current.

The main conclusion from our analysis is that the anomalous conductivity for
relativistically Landau quantized states can modify the electromagnetic wave
propagation properties in magnetar environments in a distinct way. However, it
is a challenge to relate the properties of the plasma susceptibility tensor to
observational magnetar data. Nevertheless, a specific possibility might be to
look for the quantum resonance given by Eq. 18 in the observed spectra.

In the present work, we have focused on the simplest aspects of the linearized
theory of electromagnetic waves. A more complete study of the linear
susceptibility and generalizations to cover also the nonlinear regime are
projects for future work. It may then be of interest to include also the
physics associated with the magnetic dipole force due to the electron spin
[10, 26], and the vacuum polarization associated with strong magnetic fields,
see e.g. Refs. [29, 13, 30, 5]

##  VI acknowledgement

We are grateful to Frederico Fiuza and Christopher Thompson for valuable
discussions. H. A-N also like to acknowledge support by the Knut and Alice
Wallenberg Foundation through the fellowship KAW 2022.0361.

## References

  * Fedotov  _et al._ [2023] A. Fedotov, A. Ilderton, F. Karbstein, B. King, D. Seipt, H. Taya, and G. Torgrimsson, Advances in qed with intense background fields, Physics Reports 1010, 1 (2023).
  * Di Piazza  _et al._ [2012] A. Di Piazza, C. Müller, K. Hatsagortsyan, and C. H. Keitel, Extremely high-intensity laser interactions with fundamental quantum systems, Reviews of Modern Physics 84, 1177 (2012).
  * Turolla  _et al._ [2015] R. Turolla, S. Zane, and A. Watts, Magnetars: the physics behind observations. a review, Reports on Progress in Physics 78, 116901 (2015).
  * Al-Naseri  _et al._ [2020] H. Al-Naseri, J. Zamanian, R. Ekman, and G. Brodin, Kinetic theory for spin-1/2 particles in ultrastrong magnetic fields, Phys. Rev. E 102, 043203 (2020).
  * Uzdensky and Rightley [2014] D. A. Uzdensky and S. Rightley, Plasma physics of extreme astrophysical environments, Reports on Progress in Physics 77, 036902 (2014).
  * Baring and Harding [2001] M. G. Baring and A. K. Harding, Photon splitting and pair creation in highly magnetized pulsars, The Astrophysical Journal 547, 929 (2001).
  * Sheng  _et al._ [2018] X.-l. Sheng, D. H. Rischke, D. Vasak, and Q. Wang, Wigner functions for fermions in strong magnetic fields, The European Physical Journal A 54, 21 (2018).
  * Zare  _et al._ [2022] S. Zare, H. Hassanabadi, and A. Guvendi, Relativistic landau quantization for a composite system in the spiral dislocation spacetime, The European Physical Journal Plus 137, 589 (2022).
  * Melrose and Weise [2012] D. B. Melrose and J. I. Weise, Spin-dependent relativistic quantum magnetized electron gas, Journal of Physics A: Mathematical and Theoretical 45, 395501 (2012).
  * Brodin and Zamanian [2022] G. Brodin and J. Zamanian, Quantum kinetic theory of plasmas, Reviews of Modern Plasma Physics 6, 4 (2022).
  * Cage  _et al._ [2012] M. E. Cage, K. Klitzing, A. Chang, F. Duncan, M. Haldane, R. B. Laughlin, A. Pruisken, and D. Thouless,  _The quantum Hall effect_ (Springer Science & Business Media, 2012).
  * Lai and Ho [2002] D. Lai and W. C. Ho, Resonant conversion of photon modes due to vacuum polarization in a magnetized plasma: implications for x-ray emission from magnetars, The Astrophysical Journal 566, 373 (2002).
  * Lundin [2009] J. Lundin, An effective action approach to photon propagation on a magnetized background, Europhysics letters 87, 31001 (2009).
  * Medvedev [2023] M. V. Medvedev, Plasma modes in qed super-strong magnetic fields of magnetars and laser plasmas, Physics of Plasmas 30 (2023).
  * Wang and Lai [2007] C. Wang and D. Lai, Wave modes in the magnetospheres of pulsars and magnetars, Monthly Notices of the Royal Astronomical Society 377, 1095 (2007).
  * Ghosh  _et al._ [2002] S. Ghosh, K. Goswami, S. Chakrabarty, A. Goyal, and S. Ghosh, Electrical conductivity at the core of a magnetar, International Journal of Modern Physics D 11, 843 (2002).
  * Iqbal  _et al._ [2018] M. Iqbal, H. Shah, W. Masood, and N. Tsintsadze, Nonlinear ion acoustic waves in a relativistic degenerate plasma with landau diamagnetism and electron trapping, The European Physical Journal D 72, 1 (2018).
  * Hussain and Akhtar [2020] S. Hussain and N. Akhtar, The influence of landau quantization on the propagation of solitary structures in collisional plasmas, Communications in Theoretical Physics 72, 085503 (2020).
  * Hussain  _et al._ [2023] A. Hussain, A. Majeed, M. Ayub, G. Murtaza, and Z. Iqbal, Oblique bernstein wave propagation in electron-ion plasma with electron quantization effects, Contributions to Plasma Physics 63, e202200160 (2023).
  * Areeb  _et al._ [2023] F. Areeb, I. Ahsan, A. Rasheed, P. Sumera, and M. Jamil, Magnetosonics with landau levels in gaas semiconductor systems, Physica B: Condensed Matter 663, 414968 (2023).
  * Jahangir and Ali [2021] R. Jahangir and S. Ali, Nonlinear ion-acoustic waves in degenerate plasma with landau quantized trapped electrons, Frontiers in Physics 9, 622820 (2021).
  * Eliezer  _et al._ [2005] S. Eliezer, P. Norreys, J. T. Mendonça, and K. Lancaster, Effects of landau quantization on the equations of state in intense laser plasma interactions with strong magnetic fields, Physics of plasmas 12 (2005).
  * [23] Formally, since the derivation uses a Wigner-transform, the function $W$ is a Wigner-function, rather than a distribution function. As a result, it can also be negative in limited regions of momentum space. However, due to the similarity with the Vlasov equation, for most practical purposes, it makes sense to think of $W$ as a distribution function, and we will therefore refer to it as such. .
  * Foldy and Wouthuysen [1950] L. L. Foldy and S. A. Wouthuysen, On the dirac theory of spin 1/2 particles and its non-relativistic limit, Phys. Rev. 78, 29 (1950).
  * Silenko [2008] A. J. Silenko, Foldy-wouthyusen transformation and semiclassical limit for relativistic particles in strong external fields, Phys. Rev. A 77, 012116 (2008).
  * Asenjo  _et al._ [2012] F. A. Asenjo, J. Zamanian, M. Marklund, G. Brodin, and P. Johansson, Semi-relativistic effects in spin-1/2 quantum plasmas, New Journal of Physics 14, 073042 (2012).
  * Hussain  _et al._ [2014] A. Hussain, M. Stefan, and G. Brodin, Weakly relativistic quantum kinetic theory for electrostatic wave modes in magnetized plasmas, Physics of Plasmas 21 (2014).
  * Stix [1992] T. H. Stix,  _Waves in plasmas_ (Springer Science & Business Media, 1992).
  * Haas  _et al._ [2010] F. Haas, M. Marklund, G. Brodin, and J. Zamanian, Fluid moment hierarchy equations derived from quantum kinetic theory, Physics Letters. A 374 (2010).
  * Eriksson  _et al._ [2004] D. Eriksson, G. Brodin, M. Marklund, and L. Stenflo, Possibility to measure elastic photon-photon scattering in vacuum, Physical Review A 70, 013808 (2004).