# Subject-driven Text-to-Image Generation via Preference-based Reinforcement
Learning

Yanting Miao
University of Waterloo
Vector Institute
<EMAIL_ADDRESS>
&William Loh
University of Waterloo
Vector Institute
<EMAIL_ADDRESS>
&Suraj Kothawade
Google
<EMAIL_ADDRESS>&Pascal Poupart
University of Waterloo
Vector Institute
<EMAIL_ADDRESS>
&Abdullah Rashwan
Google
<EMAIL_ADDRESS>
&Yeqing Li
Google
<EMAIL_ADDRESS>This paper was completed during work at Google

###### Abstract

Text-to-image generative models have recently attracted considerable interest,
enabling the synthesis of high-quality images from textual prompts. However,
these models often lack the capability to generate specific subjects from
given reference images or to synthesize novel renditions under varying
conditions. Methods like DreamBooth and Subject-driven Text-to-Image (SuTI)
have made significant progress in this area. Yet, both approaches primarily
focus on enhancing similarity to reference images and require expensive
setups, often overlooking the need for efficient training and avoiding
overfitting to the reference images. In this work, we present the
$\lambda$-Harmonic reward function, which provides a reliable reward signal
and enables early stopping for faster training and effective regularization.
By combining the Bradley-Terry preference model, the $\lambda$-Harmonic reward
function also provides preference labels for subject-driven generation tasks.
We propose Reward Preference Optimization (RPO), which offers a simpler setup
(requiring only $3\%$ of the negative samples used by DreamBooth) and fewer
gradient steps for fine-tuning. Unlike most existing methods, our approach
does not require training a text encoder or optimizing text embeddings and
achieves text-image alignment by fine-tuning only the U-Net component.
Empirically, $\lambda$-Harmonic proves to be a reliable approach for model
selection in subject-driven generation tasks. Based on preference labels and
early stopping validation from the $\lambda$-Harmonic reward function, our
algorithm achieves a state-of-the-art CLIP-I score of 0.833 and a CLIP-T score
of 0.314 on DreamBench.

Figure 1: We illustrate the $\lambda$-Harmonic reward function applied to the
subject-driven generation task. Leveraging preference labels produced by the
$\lambda$-Harmonic reward function, alongside a few reference images, our
preference-based algorithm efficiently generates unseen scenes that are both
faithful to the reference images and the textual prompts.

##  1 Introduction

In the evolving field of generative AI, text-to-image diffusion models [25,
12, 23, 26, 24, 21, 22] have demonstrated remarkable abilities in rendering
scenes that are both imaginative and contextually appropriate. However, these
models often struggle with tasks that require the portrayal of specific
subjects within text prompts. For instance, if provided with a photo of your
cat, current diffusion models are unable to generate an image of your cat
situated in the castle of your childhood dreams. This challenge necessitates a
deep understanding of subject identity. Consequently, subject-driven text-to-
image generation has attracted considerable interest within the community.
Chen et al. [7] have noted that this task requires complex transformations of
reference images. Additionally, Ruiz et al. [21] have highlighted that
detailed and descriptive prompts about specific objects can lead to varied
appearances in subjects. Thus, traditional image editing approaches and
existing text-to-image models are ill-suited for subject-driven tasks.

Current subject-driven text-to-image generation methods are less expressive
and expensive. Textual Inversion [11] performs poorly due to the limited
expressiveness of frozen diffusion models. Imagic [14] is both time-consuming
and resource-intensive during the fine-tuning phase. It requires text-
embedding optimization for each prompt, fine-tuning of diffusion models, and
interpolation between optimized and target prompts. The training process is
complex and slow. These text-based methods require 30 to 70 minutes to fine-
tune their models, which is not scalable for real applications. SuTI [7]
proposes an in-context learning method for subject-driven tasks. However, SuTI
demands half a million expert models for each different subject, making it
prohibitively expensive. Although SuTI can perform in-context learning during
inference, the setup of expert models remains costly. DreamBooth [21] provides
a simpler method for handling subject-driven tasks. Nevertheless, DreamBooth
requires approximately 1000 negative samples and 1000 gradient steps, and also
needs fine-tuning of the text encoder to achieve state-of-the-art performance.
Therefore, it is worthwhile to explore more efficient training methods: the
setup should be as simple as possible, the efficient training should not
include multiple optimization phases; second, by learning text-to-image
alignment using only the UNet component without text-embedding for each prompt
or text-encoder optimization; third, by providing a model-selection approach
that enables early stopping for faster evaluation and regularization.

In this paper, we propose a $\lambda$-Harmonic reward function that enables
early stopping and accelerate training. In addition, we incorporate the
Bradley-Terry preference model to generate preference labels and utilize
preference-based reinforcement learning algorithms to finetune pre-trained
diffusion models and achieve text-to-image alignment without optimizing any
text-encoder or text-embedding. The whole finetuning process including setup,
training, validation, and model saving only takes 5 to 20 minutes on Cloud TPU
V4. Our method, Reward Preference Optimization (RPO), only requires a few
input reference images and the finetuned diffusion model can generate images
that preserve the identity of a specific subject and align well with textual
prompts (Figure 1).

To show the effectiveness of our $\lambda$-Harmonic reward function, we
evaluate RPO on diverse subjects and text prompts on DreamBench [21] and we
report the DINO and CLIP-I/CLIP-T of RPO’s generated images on this benchmark
and compare them with existing methods. Surprisingly, our method requires a
simple setup ($3\%$ of DreamBooth configuration) and with a small number of
gradient steps, but the experimental results outperform or match SOTA.

In summary, our contributions are as follows:

  * •

We introduce the $\lambda$-Harmonic reward function, which permits early-
stopping to alleviate overfitting in subject-driven generation tasks and to
accelerate the finetuning process.

  * •

By combining the $\lambda$-Harmonic reward function and a preference model, we
present RPO, which only require a cheap setup but still can provide high
quality results.

  * •

We evaluate RPO and show the effectiveness of the $\lambda$-Harmonic function
with diverse subjects and various prompts on DreamBench. We achieve results
comparable to SOTA.

##  2 Related Works

Ruiz et al. [21] formulated a class of problems called subject-driven
generation, which refers to preserving the appearance of a subject
contextualized in different settings. DreamBooth [21] solves the issue of
preserving the subject by binding it in textual space with a unique identifier
for the subject in the reference images, and simultaneously generating diverse
backgrounds by leveraging prior class-specific information previously learned.
A related work that could possibly perform the same task is textual inversion
[11]. However, its original objective is to produce a modification of the
subject or property marked by a unique token in the text. While it can be used
to preserve the subject and change the background or setting, the performance
is underwhelming compared to DreamBooth in various metrics [21].

The prevalent issue in DreamBooth and textual inversion is the training times
[21, 11] since gradient-based optimization has to be performed on their
respective models for each subject. Subject-driven text-to-image generator
(SuTI) by [7] aims to alleviate this issue by employing apprenticeship
learning. By scraping millions of images online, many expert models are
trained for each cluster which allows the apprentice to learn quickly from the
experts [7]. However, this is an incredibly intensive task with massive
computational overhead during training time.

In the field of natural language processing, direct preference optimization
has found great success in large language models (LLM) [19]. By bypassing
reinforcement learning from human feedback and directly maximizing likelihoods
using preference data, LLMs benefit from more stable training and reduced
dependency on an external reward model. Subsequently, this inspired Diffusion-
DPO by [27] which applies a similar technique onto the domain of diffusion
models. However, this relies on a preference labelled dataset, which can be
expensive to collect or not publicly available for legal reasons.

Fortunately, there are reward models that can serve as functional substitutes
such as CLIP [17] and ALIGN [13]. ALIGN has a dual encoder architecture that
was trained on a large dataset. The encoders can produce text and image
embeddings, which allows us to obtain pairwise similarity scores by computing
cosine similarity. There are also diffusion modeleling techniques that can
leverage reward models. An example is denoising diffusion policy optimization
(DDPO) by Black et al. [3] that uses a policy gradient reinforcement learning
method to encourage generations that leads to higher rewards.

##  3 Preliminary

In this section, we introduce notations and some key concepts about text-to-
image diffusion models and reinforcement learning.

#### Text-to-Image Diffusion Models.

Diffusion models [12, 23, 25, 26, 24] are a family of latent variable models
of the form
$p_{\bm{\theta}}(\mathbf{x}_{0})=\int_{\mathcal{X}}p_{\bm{\theta}}(\mathbf{x}_{0:T})d\mathbf{x}_{1:T}$,
where the $\mathbf{x}_{1},\dots,\mathbf{x}_{T}$ are noised latent variables of
the same dimensionality as the input data $\mathbf{x}_{0}\sim
q(\mathbf{x}_{0})$. The diffusion or forward process is often a Markov chain
that gradually adds Gaussian noise to the input data and each intermediate
sample $\mathbf{x}_{t}$ can be written as

$\displaystyle\mathbf{x}_{t}=\sqrt{\alpha_{t}}\mathbf{x}_{0}+\sqrt{1-\alpha_{t}}\bm{\epsilon}_{t},\quad\text{for
all }t\in\\{1,\dots,T\\},$ (1)

where $\alpha_{t}$ refers to the variance schedule and
$\bm{\epsilon}_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. Given a
conditioning tensor $\mathbf{c}$, the core premise of text-to-image diffusion
models is to use a neural network
$\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}_{t},\mathbf{c},t)$ that iteratively
refines the current noised sample $\mathbf{x}_{t}$ to obtain the previous step
sample $\mathbf{x}_{t-1}$, This network can be trained by optimizing a simple
denoising objective function, which is the time coefficient weighted mean
squared error:

$\displaystyle\mathbb{E}_{\mathbf{x}_{0},\mathbf{c},t,\bm{\epsilon}_{t}}\left[\omega(t)\lVert\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}_{t},\mathbf{c},t)-\bm{\epsilon}_{t}\rVert^{2}_{2}\right],$
(2)

where $t$ is uniformly sampled from $\\{1,\dots,T\\}$ and $\omega(t)$ can be
simplified as 1 according to [12, 24, 20].

#### Reinforcement Learning and Diffusion DPO

Reinforcement Learning for diffusion models [3, 10, 27] aims to solve the
following optimization problem:

$\displaystyle\mathbb{E}_{\mathbf{x}_{0:T}\sim
p_{\bm{\theta}}(\mathbf{x}_{0:T}\mid\mathbf{c})}\left[\sum_{t=1}^{T}R(\mathbf{x}_{t},\mathbf{x}_{t-1},\mathbf{c})-\beta\mathbb{D}_{\text{KL}}(p_{\bm{\theta}}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{c})\|p_{\text{base}}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{c}))\right],$
(3)

where $\beta$ is a hyperparameter controlling the KL-divergence between the
finetune model $p_{\bm{\theta}}$ and the pre-trained base model
$p_{\text{base}}$. In Equation (14) from Diffusion-DPO [27], the optimal
$p_{\bm{\theta}}$ can be approximated by minimizing the negative log-
likelihood:

$\displaystyle\mathbb{E}_{\mathbf{x}^{+}_{0},\mathbf{x}^{-}_{0},t,\mathbf{x}^{+}_{t},\mathbf{x}^{-}_{t}}$
$\displaystyle\Bigg{[}-\log\sigma\bigg{(}\beta\big{(}\lVert\bm{\epsilon}_{\text{base}}(\mathbf{x}^{+}_{t},\mathbf{c},t)-\bm{\epsilon}^{+}_{t}\rVert^{2}_{2}-\lVert\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}^{+}_{t},\mathbf{c},t)-\bm{\epsilon}^{+}\rVert^{2}_{2}$
$\displaystyle-(\lVert\bm{\epsilon}_{\text{base}}(\mathbf{x}^{-}_{t},\mathbf{c},t)-\bm{\epsilon}^{-}_{t}\rVert^{2}_{2}-\lVert\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}^{-}_{t},\mathbf{c},t)-\bm{\epsilon}^{-}\rVert^{2}_{2})\big{)}\bigg{)}\Bigg{]},$
(4)

where $\\{\mathbf{x}^{+}_{t}\\}_{t=0}^{T}$ represents the preference
trajectory, i.e.,
$r(\mathbf{x}^{+}_{0},\mathbf{c})>r(\mathbf{x}^{-}_{0},\mathbf{c})$, and
$\bm{\epsilon}^{+}$ and $\bm{\epsilon}^{-}$ are independent samples from a
Gaussian distribution. A detailed description is given in Appendix A.1.

#### Additional notations.

We use $\mathbf{x}_{\text{ref}}$ and $\mathbf{x}_{\text{gen}}$ to represent
the reference image and generated image, respectively.
$\mathcal{I}_{\text{ref}}$ denotes the set of reference images, and
$\mathcal{I}_{\text{gen}}$ is the set of generated images.
$\mathbb{P}(\mathbf{x}\succ\tilde{\mathbf{x}})$ represents the probability
that $\mathbf{x}$ is more preferred than $\tilde{\mathbf{x}}$.

##  4 Method

We present our $\lambda$-Harmonic reward function that provides reward signals
for subject-driven tasks to reduce the learned model to be overfitted to the
reference images. Based on this reward function, we use Bradley-Terry model to
sample preference labels and a preference algorithm to finetune the diffusion
model by optimizing both the similarity loss and the preference loss.

###  4.1 Reward Preference Optimization

In contrast to other fine-tuning applications [27, 16, 19, 18], there is no
human feedback in the subject-driven text-to-image generation task. The model
only receives a few reference images and a prompt with a specific subject.
Hence, we first propose the $\lambda$-Harmonic reward function that can
leverage the ALIGN model [13] to provide valid feedback based on the generated
image fidelity: similarity to the given reference images and faithfulness to
the text prompts.

####  $\lambda$-Harmonic Reward Function.

The normalized ALIGN-I and ALIGN-T scores can be denoted as

$\displaystyle\text{ALIGN-I}(\mathbf{x},\mathcal{I}_{\text{ref}})\coloneq\frac{1}{|\mathcal{I}_{\text{ref}}|}\sum_{\tilde{\mathbf{x}}\in\mathcal{I}_{\text{ref}}}\frac{\text{CosSim}(f_{\bm{\phi}}(\mathbf{x}),f_{\bm{\phi}}(\tilde{\mathbf{x}}))+1}{2}$
$\displaystyle\text{ALIGN-T}(\mathbf{x},\mathbf{c})\coloneq\frac{\text{CosSim}(f_{\bm{\phi}}(\mathbf{x}),g_{\bm{\phi}}(\mathbf{c}))+1}{2},$

where $f_{\bm{\phi}}(\mathbf{x})$ is the image feature extractor and
$g_{\bm{\phi}}(\mathbf{c})$ is the text encoder in the ALIGN model. Given a
reference image set $\mathcal{I}_{\text{ref}}$, the $\lambda$-Harmonic reward
function can be defined by a weighted harmonic mean of the ALIGN-I and ALIGN-T
scores,

$\displaystyle
r(\mathbf{x},\mathbf{c};\lambda,\mathcal{I}_{\text{ref}})\coloneq\dfrac{1}{\dfrac{\lambda}{\text{ALIGN-I}(\mathbf{x},\mathcal{I}_{\text{ref}})}+\dfrac{1-\lambda}{\text{ALIGN-T}(\mathbf{x},\mathbf{c})}}.$
(5)

Compared to the arithmetic mean, there are two advantages to using the
harmonic mean: (1) according to AM-GM-HM inequalities [9], the harmonic mean
is a lower bound of the arithmetic mean and maximizing this “pessimistic”
reward can also improve the arithmetic mean of ALIGN-I and ALIGN-T scores; (2)
the harmonic mean is more sensitive to the smaller of the two scores, i.e., a
larger reward is only achieved when both scores are relatively large.

For a simple example, consider $\lambda=0.5$. If there are two images,
$\mathbf{x}$ and $\tilde{\mathbf{x}}$, where the first image achieves an
ALIGN-I score of 0.9 and an ALIGN-T score of 0.01, and the second image
receives an ALIGN-I score of 0.7 and an ALIGN-T score of 0.21, we may prefer
the second image because it has high similarity to the reference images and is
faithful to the text prompts. However, using the arithmetic mean would assign
both images the same reward of 0.455. In contrast, the harmonic mean would
assign the first image a reward of 0.020 and the second image a reward of
0.323, aligning with our preferences. During training, we set
$\lambda_{\text{train}}=0$, which means the reward model will focus solely on
text-to-image alignment because the objective function consists only of a loss
for image-to-image alignment. Note that we set $\lambda_{\text{val}}$ to a
different value for validation, which evaluates the fidelity of the subject
and faithfulness of the prompt. Details can be found in Section 5.

#### Dataset.

The set of images for subject-driven generative tasks can usually be
represented as
$\mathcal{I}=\mathcal{I}_{\text{ref}}\cup\mathcal{I}_{\text{gen}}$, where
$\mathcal{I}_{\text{gen}}$ is the image set generated by the base model.
DreamBooth [21] requires two different prompts, $\mathbf{c}$ and
$\mathbf{c}_{\text{pr}}$, for each subject during the training phase. For
example, $\mathbf{c}$ is ‘‘a photo of [V] dog’’ and $\mathbf{c}_{\text{pr}}$
is ‘‘a photo of dog’’, where ‘‘[V]’’ is a unique token to specify the subject.
DreamBooth then uses $\mathbf{c}_{\text{pr}}$ to generate
$\mathcal{I}_{\text{gen}}$ for prior preservation loss. Typically, the size of
the set of generated images is around 1000, i.e.,
$\lvert\mathcal{I}_{\text{gen}}\rvert=1000$, which is time-consuming and
space-intensive in real applications. However, the diffusion model can only
maximize the similarity score and still receives a high reward based on this
uninformative prompt $\mathbf{c}_{\text{pr}}$. Our method aims to balance the
trade-off between similarity and faithfulness. Thus, for efficiency, we
introduce 8 novel training prompts, $\mathbf{c}_{\text{pr}}$, which can be
pre-specified or generated by other Large Language Models111SuTI [7] utilizes
PaLM [8] to generate unseen prompts during training. These training prompts
include artistic style transfer, re-contextualization, and accessorization.
The full list of training prompts is provided in the supplementary material.
For example, $\mathbf{c}_{\text{pr}}$ can be ‘‘a [V] dog is on the Moon’’. We
feed these training prompts to the base model and generate 4 images for each
training prompt, i.e., $\lvert\mathcal{I}_{\text{gen}}\rvert=32$.

Once we obtain reward signals, we adopt the Bradley-Terry model [4] to
generate preference labels. In particular, given a tuple
$(\mathbf{x}_{\text{ref}},\mathbf{x}_{\text{gen}},\mathbf{c}_{\text{pr}})$, we
sample preference labels $y$ from the following probability model:

$\displaystyle\mathbb{P}(\mathbf{x}_{\text{ref}}\succ\mathbf{x}_{\text{gen}})\coloneq\frac{\exp(r(\mathbf{x}_{\text{ref}},\mathbf{c}_{\text{pr}};\lambda,\mathcal{I}_{\text{ref}}))}{\exp(r(\mathbf{x}_{\text{ref}},\mathbf{c}_{\text{pr}};\lambda,\mathcal{I}_{\text{ref}}))+\exp(r(\mathbf{x}_{\text{gen}},\mathbf{c}_{\text{pr}};\lambda,\mathcal{I}_{\text{ref}}))}.$
(6)

#### Learning.

The learning objective function consist of two parts — similarity loss and
preference loss. The similarity loss is designed to minimize the KL divergence
between the distribution of reference images and the learned distribution
$p_{\bm{\theta}}(\mathbf{x})$, which is equivalent to minimizing:

$\displaystyle\mathcal{L}_{\text{sim}}(\bm{\theta})\coloneq\mathbb{E}_{\mathbf{x}_{\text{ref}},\mathbf{c},t,\bm{\epsilon}_{\text{ref}}}\left[\lVert\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}_{\text{ref},t},\mathbf{c},t)-\bm{\epsilon}_{\text{ref}}\rVert^{2}_{2}\right],\quad
t\sim\mathcal{U}\\{1,\dots,T\\},\quad\bm{\epsilon}_{\text{ref}}\sim\mathcal{N}(\mathbf{0},\mathbf{I}).$
(7)

The preference loss aims to capture the preference signals and fit the
preference model, Eq. (6). Therefore, we use the binary cross-entropy as the
objective function for preference loss, combining DPO objective function Eq. 3
the loss function can be written as the following function:

$\displaystyle\mathcal{L}_{\text{pre}}(\bm{\theta})$
$\displaystyle\coloneq\mathbb{E}_{\mathbf{x}_{\text{ref}},\mathbf{x}_{\text{gen}},\mathbf{c}_{\text{pr}},y,t,\bm{\epsilon}_{\text{ref}},\bm{\epsilon}_{\text{gen}}}\Bigg{[}y\log\sigma\bigg{(}\beta\ell_{\bm{\theta}}(\mathbf{x}_{\text{ref}},\mathbf{x}_{\text{gen}},\mathbf{c}_{\text{pr}},y,t,\bm{\epsilon}_{\text{ref}},\bm{\epsilon}_{\text{gen}})\bigg{)}$
$\displaystyle+(1-y)\log\sigma\bigg{(}-\beta\ell_{\bm{\theta}}(\mathbf{x}_{\text{ref}},\mathbf{x}_{\text{gen}},\mathbf{c}_{\text{pr}},y,t,\bm{\epsilon}_{\text{ref}},\bm{\epsilon}_{\text{gen}})\bigg{)}\Bigg{]},$
(8)

where

$\displaystyle\ell_{\bm{\theta}}(\mathbf{x}_{\text{ref}},\mathbf{x}_{\text{gen}},\
\mathbf{c}_{\text{pr}},y,t,\bm{\epsilon}_{\text{ref}},\bm{\epsilon}_{\text{gen}})$
$\displaystyle\coloneq\lVert\bm{\epsilon}_{\text{base}}(\mathbf{x}_{\text{ref},t},\mathbf{c}_{\text{pr}},t)-\bm{\epsilon}_{\text{ref}}\rVert^{2}_{2}-\lVert\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}_{\text{ref},t},\mathbf{c}_{\text{pr}},t)-\bm{\epsilon}_{\text{ref}}\rVert^{2}_{2}$
$\displaystyle-(\lVert\bm{\epsilon}_{\text{base}}(\mathbf{x}_{\text{gen},t},\mathbf{c}_{\text{pr}},t)-\bm{\epsilon}_{\text{gen}}\rVert^{2}_{2}-\lVert\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}_{\text{gen},t},\mathbf{c}_{\text{pr}},t)-\bm{\epsilon}_{\text{gen}}\rVert^{2}_{2}),$

and $t\sim\mathcal{U}\\{1,\dots,T\\}$ and
$\bm{\epsilon}_{\text{ref}},\bm{\epsilon}_{\text{gen}}\sim\mathcal{N}(\mathbf{0},\mathbf{I}).$
Combining these two loss functions together, the objective function for
finetuning is written as

$\displaystyle\mathcal{L}(\bm{\theta})=\mathcal{L}_{\text{sim}}(\bm{\theta})+\mathcal{L}_{\text{pre}}(\bm{\theta})$
(9)

Figure 2: Overview of the finetuning phase for RPO. First, the base diffusion
model generates a few images based on novel training prompts. Second, we
compute the rewards for both reference and generated images using Equation
(5). Then, preference labels are sampled according to the preference
distribution, as defined in Equation (6). Finally, the diffusion model is
trained by minimizing both the similarity loss (Equation (7)) and preference
loss (Equation (4.1)).

Figure 2 presents an overview of the training method, which includes the base
model generated samples, the align reward model, and the preference loss. Note
that $\mathcal{L}_{\text{pre}}$ serves as a regularizer for approximating the
text-to-image alignment policy. Conversely, DreamBooth [21] adopts
$\mathbb{D}_{\text{KL}}(p_{\text{base}}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{c})|p_{\bm{\theta}}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{c}))$
as its regularizer, which cannot guarantee faithfulness to the text-prompt.
Based on this loss function and preference model, we only need a few hundred
iterations and a small set size of $\lvert\mathcal{I}_{\text{gen}}\rvert$ to
achieve results that are comparable to, or even better than, the state of the
art. The fine-tuning process, which includes generating images, training, and
validation, takes about 5 to 20 minutes on a single Google Cloud Platform
TPUv4-8 (32GB) for Stable Diffusion.

##  5 Experiments

In this section, we present the experimental results demonstrated by RPO. We
investigate three primary questions. First, can our algorithm learn to
generate images that are faithful both to the reference images and to the
textual prompts, according to preference labels? Second, if RPO can generate
high-quality images, which part is the key component of RPO: the reference
loss or the early stopping by the $\lambda$-Harmonic reward function? Third,
how do different $\lambda_{\text{val}}$ values affect validation and cause
performance differences for RPO? We refer readers to Appendix A.2 for details
on the experimental setup, Appendix A.7 for the skill set of RPO, Appendix A.4
for the limitations of the RPO algorithm, and Appendix A.5 for future work
involving RPO.

###  5.1 Dataset and Evaluation

#### DreamBench.

In this work, we use the DreamBench dataset proposed by DreamBooth [21]. This
dataset contains 30 different subject images including backpacks, sneakers,
boots, cats, dogs, and toy, etc. DreamBench also provides 25 various prompt
templates for each subject and these prompts are requiring the learned models
to have such abilities: re-contextualization, accessorization, property
modification, and attribute editing.

#### Evaluation Metrics.

We follow DreamBooth [21] and SuTI [7] to report DINO [5] and CLIP-I [17] for
evaluating image-to-image similarity score and CLIP-T [17] for evaluating the
text-to-image similarity score. We also use our $\lambda$-Harmonic reward as a
evaluation metric for the overall fidelity and the default value of
$\lambda=0.3$. For evaluation, we follow DreamBooth [21] and SuTI [7] to
generate 4 images per prompt, 3000 images in total, which provides robust
evaluation.

#### Baseline algorithms.

DreamBooth [21]: This algorithm requires approximately
$\lvert\mathcal{I}_{\text{gen}}\rvert=1000$ and 1000 gradient steps to
finetune the UNet and text-encoder components. SuTI [7]: A pre-trained method
that requires half a million expert models and introduces cross-attention
layers into the original diffusion models. Textual Inversion [11]: A text-
based method that optimizes the text embedding but freezes the diffusion
models. Re-Imagen [6]: An information retrieval-based algorithm that modifies
the backbone network architectures and introduces cross-attention layers into
the original diffusion models.

Figure 3: Qualitative comparison with other subject-driven text-to-image
methods, adapted from [7]

###  5.2 Comparisons

#### Quantitative Comparisons.

We begin by addressing the first question. We use a quantitative evaluation to
compare RPO with other existing methods on three metrics (DINO, CLIP-I,
CLIP-T) in DreamBench to validate the effectiveness of RPO. The experimental
results on DreamBench is shown in Table 1. We observe that RPO can perform
better or on par with SuTI and DreamBooth on all three metrics. Compared to
DreamBooth, RPO only requires $3\%$ of the negative samples, but RPO can
outperform DreamBooth on the CLIP-I and CLIP-T scores by $3\%$ given the same
backbone. Our method outperforms all baseline algorithms in the CLIP-T score,
establishing a new SOTA result. This demonstrates that RPO, by solely
optimizing UNet through preference labels from the $\lambda$-Harmonic reward
function, can generate images that are faithful to the input prompts.
Similarly, our CLIP-I score is also the highest, which indicates that RPO can
generate images that preserve the subject’s visual features. In terms of the
DINO score, our method is almost the same as DreamBooth when using the same
backbone. We conjecture that the reason RPO achieves higher CLIP scores and
lower DINO score is that the $\lambda$-Harmonic reward function prefers to
select images that are semantically similar to the textual prompt, which may
result in the loss of some unique features in the pixel space.

Table 1: Quantitative comparison for the number of iterations, subject fidelity and prompt fidelity. Method | Backbone | Iterations $\downarrow$ | DINO $\uparrow$ | CLIP-I $\uparrow$ | CLIP-T $\uparrow$
---|---|---|---|---|---
Reference Images | N/A | N/A | $0.774$ | $0.885$ | N/A
DreamBooth [21] | Imagen [22] | $1000$ | $0.696$ | $0.812$ | $0.306$
DreamBooth [21] | SD [20] | $1000$ | $0.668$ | $0.803$ | $0.305$
Textual inversion [11] | SD [20] | $5000$ | $0.569$ | $0.780$ | $0.255$
SuTI [7] | Imagen [22] | $1.5\times 10^{5}$ | $\bm{0.741}$ | $0.819$ | $0.304$
Re-Imagen [6] | Imagen [22] | $2\times 10^{5}$ | $0.600$ | $0.740$ | $0.270$
Ours: RPO | SD [20] | $\bm{400}$ | $0.652$ | $\bm{0.833}$ | $\bm{0.314}$

#### Qualitative Comparisons.

We use the same prompt as SuTI [7], and the generated images are shown in
Figure 3. RPO generates images that are faithful to both reference images and
textual prompts. We noticed a grammatical mistake in the first prompt used by
SuTI [7]; it should be A dog eating a cherry from a bowl. This incorrect
prompt caused the RPO-trained model to become confused during the inference
phase. However, RPO still preserves the unique appearance of the bowl. For
instance, while the text on the bowl is incorrect or blurred in the SuTI and
DreamBooth results, RPO accurately retains the words Bon Appetit from the
reference bowl images. We also observed that RPO is the only method that
generates thin-neck vases. Although existing methods can produce images highly
faithful to the reference images, they may not align as well with the textual
prompts. We also provide an example in Appendix A.3 that shows how RPO can
handle the failure case observed in DreamBooth and SuTI.

Table 2: Ablation study of regularization for evaluating fidelity over multiple subjects and prompts. Method | DINO $\uparrow$ | CLIP-I $\uparrow$ | CLIP-T $\uparrow$ |  $0.3$-Harmonic $\uparrow$
---|---|---|---|---
Pure $\mathcal{L}_{\text{sim}}$ | $\bm{0.695\pm 0.077}$ | $\bm{0.852\pm 0.043}$ | $0.285\pm 0.027$ | $0.660\pm 0.016$
$\mathcal{L}_{\text{ref}}$ w/o early-stopping | $0.688\pm 0.082$ | $0.845\pm 0.042$ | $0.296\pm 0.027$ | $0.663\pm 0.014$
Early-stopping w/o $\mathcal{L}_{\text{ref}}$ | $0.575\pm 0.124$ | $0.799\pm 0.052$ | $0.323\pm 0.025$ | $0.672\pm 0.013$
RPO ($\lambda_{\text{val}}=0.3$) | $0.581\pm 0.113$ | $0.798\pm 0.039$ | $\bm{0.329\pm 0.021}$ | $\bm{0.673\pm 0.013}$
Table 3: Different validation $\lambda_{\text{val}}$-Harmonic reward comparison for evaluating fidelity over multiple subjects and prompts. Configuration | DINO $\uparrow$ | CLIP-I $\uparrow$ | CLIP-T $\uparrow$ |  $0.3$-Harmonic $\uparrow$
---|---|---|---|---
$\lambda_{\text{val}}=0.3$ | $0.581\pm 0.113$ | $0.798\pm 0.039$ | $\bm{0.329\pm 0.021}$ | $\bm{0.673\pm 0.013}$
$\lambda_{\text{val}}=0.5$ | $0.652\pm 0.082$ | $0.833\pm 0.041$ | $0.314\pm 0.022$ | $0.671\pm 0.008$
$\lambda_{\text{val}}=0.7$ | $\bm{0.679\pm 0.085}$ | $\bm{0.850\pm 0.045}$ | $0.304\pm 0.023$ | $0.667\pm 0.011$
Figure 4: Changes in the $0.3$-Harmonic reward value during RPO training
process. Figure 5: Different $\lambda_{\text{val}}$ will lead to different
results. A small $\lambda_{\text{val}}$ assign a higher a weight for text-to-
image alignment and lead to diverse generation. A large $\lambda_{\text{val}}$
may also cause overfitting.

###  5.3 Ablation Study and Method Analysis

#### Preference Loss and $\lambda$-Harmonic Ablation.

We investigate the second primary question through an ablation study. Two
regularization components are introduced into RPO: reference loss as a
regularizer and early stopping by $\lambda_{\text{val}}$-Harmonic reward
function. Consequently, we compare four methods: (1) Pure
$\mathcal{L}_{\text{sim}}$, which only minimizes the image-to-image similarity
loss $\mathcal{L}_{\text{sim}}$; (2) $\mathcal{L}_{\text{ref}}$ w/o early
stopping, which employs $\mathcal{L}_{\text{ref}}$ as a regularizer but omits
early stopping by $\lambda_{\text{val}}$-Harmonic reward function; (3) Early
stopping w/o $\mathcal{L}_{\text{ref}}$, which uses
$\lambda_{\text{val}}$-Harmonic reward function as a regularization method but
excludes $\mathcal{L}_{\text{ref}}$; (4) Full RPO, which utilizes both
$\mathcal{L}_{\text{ref}}$ and early stopping by the
$\lambda_{\text{val}}$-Harmonic reward function. We choose the default value
$\lambda_{\text{val}}=0.3$ in this ablation study.

Table 2 lists the evaluation results of these four methods on DreamBench. We
observe that without early stopping, $\mathcal{L}_{\text{ref}}$ can still
prevent overfitting to the reference images and improve text-to-image
alignment, though the regularization effect is weak. Specifically, the
$0.3$-Harmonic only shows a marginal improvement of 0.003 over pure
$\mathcal{L}_{\text{sim}}$ and 0.001 over early stopping without
$\mathcal{L}_{\text{ref}}$. The early stopping facilitated by the
$\lambda_{\text{val}}$-Harmonic reward function plays a crucial role in
counteracting overfitting, helping the diffusion models retain the ability to
generate high-quality images aligned with textual prompts. To provide a deeper
understanding of the $\lambda$-Harmonic reward validation, we present two
examples from during training in Figure 4, covering both objects and live
subjects. We found that the model tends to overfit at a very early stage,
i.e., within 200 gradient steps, where $\lambda$-Harmonic can provide correct
reward signals for the generated images. For the backpack subject, the
generated image receives a low reward at gradient step 80 due to its lack of
fidelity to the reference images. However, at gradient step 400, the image is
overfitted to the reference images, and the model fails to align well with the
input text, resulting in another low reward. $\lambda$-Harmonic assigns a high
reward to images that are faithful to both the reference image and textual
prompts.

#### Impact of $\lambda_{\text{val}}$.

We examine the third primary question by selecting different
$\lambda_{\text{val}}$ values from the set $\\{0.3,0.5,0.7\\}$ as the
validation parameters for the $\lambda$-Harmonic reward. According to Equation
5, we believe that as $\lambda_{\text{val}}$ increases, the $\lambda$-Harmonic
reward function will give higher weight to the image-to-image similarity
score. This will make the generated images more closely resemble the reference
images, however, there is also a risk of overfitting. Table 3 shows us the
results of three different $\lambda_{\text{val}}$ values on DreamBench. As we
expected, a larger $\lambda_{\text{val}}$ makes the images better preserve the
characteristics of the reference images, but it also reduces the text-to-image
alignment score. Figure 5 shows us an example. In this example, different
$\lambda_{\text{val}}$ values lead to different outcomes due to varying
strengths of regularization. A smaller $\lambda_{\text{val}}=0.3$ can generate
more varied results, but seems somewhat off from the reference images.
$\lambda_{\text{val}}=0.5$ preserves some characteristics beyond the original
subject, such as the sofa, but also maintains alignment between text and
image. However, when $\lambda_{\text{val}}=0.7$ is chosen as an excessively
large value, the model actually overfits to the reference images, ignoring the
prompts. We have more comparisons in Appendix A.3.

##  6 Conclusion

We introduce the $\lambda$-Harmonic reward function to derive preference
labels and employ RPO to finetune the diffusion model for subject-driven text-
to-image generation tasks. Additionally, the $\lambda$-Harmonic reward
function serves as a validation method, enabling early stopping to mitigate
overfitting to reference images and speeding up the finetuning process.

## Acknowledgement

We thank Shixin Luo and Hongliang Fei for providing constructive feedback.
This work is supported by a Google grant and with Cloud TPUs from Google’s TPU
Research Cloud (TRC).

## References

  * Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022.
  * Bird et al. [2023] Charlotte Bird, Eddie Ungless, and Atoosa Kasirzadeh.  Typology of risks of generative text-to-image models.  In _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 396–410, 2023.
  * Black et al. [2023] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine.  Training diffusion models with reinforcement learning.  _arXiv preprint arXiv:2305.13301_ , 2023.
  * Bradley and Terry [1952] Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Chen et al. [2022] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen.  Re-imagen: Retrieval-augmented text-to-image generator.  _arXiv preprint arXiv:2209.14491_ , 2022.
  * Chen et al. [2024] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen.  Subject-driven text-to-image generation via apprenticeship learning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Chowdhery et al. [2023] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ , 24(240):1–113, 2023.
  * Djukić et al. [2006] Dušan Djukić, Vladimir Janković, Ivan Matić, and Nikola Petrović.  _The IMO compendium: a collection of problems suggested for the International Mathematical Olympiads: 1959-2004_ , volume 119.  Springer, 2006.
  * Fan et al. [2023] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee.  Reinforcement learning for fine-tuning text-to-image diffusion models.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=8OTPepXzeh.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pages 4904–4916. PMLR, 2021.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6007–6017, 2023.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744, 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rafailov et al. [2024a] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn.  From $r$ to $q^{*}$: Your language model is secretly a q-function.  _arXiv preprint arXiv:2404.12358_ , 2024a.
  * Rafailov et al. [2024b] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36, 2024b.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in neural information processing systems_ , 35:36479–36494, 2022.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International conference on machine learning_ , pages 2256–2265. PMLR, 2015.
  * Song et al. [2021] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=St1giarCHLP.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Song et al. [2020] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020.
  * Wallace et al. [2023] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik.  Diffusion model alignment using direct preference optimization.  _arXiv preprint arXiv:2311.12908_ , 2023.

##  Appendix A Appendix

###  A.1 Background

#### Reinforcement Learning.

In Reinforcement Learning (RL), the environment can be formalized as a Markov
Decision Process (MDP). An MDP is defined by a tuple
$(\mathcal{S},\mathcal{A},P,R,\rho_{0},T)$, where $\mathcal{S}$ is the state
space, $\mathcal{A}$ is the action space, $P$ is the transition function, $R$
is the reward function, $\rho_{0}$ is the distribution over initial states,
and $T$ is the time horizon. At each timestep $t$, the agent observes a state
$\mathbf{s}_{t}$ and selects an action $\mathbf{a}_{t}$ according to a policy
$\pi(\mathbf{a}_{t}|\mathbf{s}_{t})$, and obtains a reward
$R(\mathbf{s}_{t},\mathbf{a}_{t})$, and then transit to a next state
$\mathbf{s}_{t+1}\sim P(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}).$ As
the agent interacts with the MDP, it produces a sequence of states and
actions, which is denoted as a trajectory
$\bm{\tau}=(\mathbf{s}_{0},\mathbf{a}_{0},\mathbf{s}_{1},\mathbf{a}_{1},\dots,\mathbf{s}_{T-1},\mathbf{a}_{T-1})$.
The RL objective is to maximize the expected value of cumulative reward over
the trajectories sample from its policy:

$\displaystyle\mathbb{E}_{\bm{\tau}\sim
p^{\pi}(\bm{\tau})}\left[\sum_{t=0}^{T-1}R(\mathbf{s}_{t},\mathbf{a}_{t})\right]$

#### Diffusion MDP

We formalize the denoising process as the following Diffusion MDP:

$\displaystyle\mathbf{s}_{T-t}=(\mathbf{x}_{t},\mathbf{c}),\quad\mathbf{a}_{T-t}=\mathbf{x}_{t-1},\quad\pi_{\bm{\theta}}(\mathbf{a}_{T-t}\mid\mathbf{s}_{T-t})=p_{\bm{\theta}}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{c}),$
$\displaystyle\rho_{0}=p(\mathbf{c})\times\mathcal{N}(\mathbf{0},\mathbf{I}),\quad
R(\mathbf{s}_{T-t},\mathbf{a}_{T-t})=R(\mathbf{x}_{t},\mathbf{x}_{t-1},\mathbf{c})=\begin{cases}r(\mathbf{x}_{0},\mathbf{c})&\text{
if }t=1,\\\ 0&\text{otherwise}\end{cases}$

where $r(\mathbf{x}_{0},\mathbf{c})$ can be a reward signal for the denoised
image. The transition kernel is deterministic, i.e.,
$P(\mathbf{s}_{T-t+1}\mid\mathbf{s}_{T-t},\mathbf{a}_{T-t})=(\mathbf{a}_{T-t},\mathbf{c})=(\mathbf{x}_{t-1},\mathbf{c}).$
For brevity, the trajectory $\bm{\tau}$ is defined by
$(\mathbf{x}_{0},\mathbf{x}_{1},\dots,\mathbf{x}_{T}).$ Hence, the trajectory
distribution for given diffusion models can be denoted as the joint
distribution $p_{\bm{\theta}}(\mathbf{x}_{0:T}\mid\mathbf{c})$. In particular,
the RL objective function for finetuning diffusion models can be re-written as
the following optimization problem:

$\displaystyle\mathbb{E}_{\mathbf{x}_{0:T}\sim
p_{\bm{\theta}}(\mathbf{x}_{0:T}\mid\mathbf{c})}\left[\sum_{t=1}^{T}R(\mathbf{x}_{t},\mathbf{x}_{t-1},\mathbf{c})-\beta\mathbb{D}_{\text{KL}}(p_{\bm{\theta}}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{c})\|p_{\text{base}}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{c}))\right],$

where $\beta$ is a hyperparameter controlling the KL-divergence between the
finetune model $p_{\bm{\theta}}$ and the base model $p_{\text{base}}$. This
constraint prevents the learned model from losing generation diversity and
falling into ’mode collapse’ due to a single high cumulative reward result. In
practice, this KL-divergence has become a standard constraint in large
language model finetuning [1, 16, 19].

Given preference labels and following the Direct Preference Optimization (DPO)
framework [27, 19, 18], we can approximate the optimal policy $p^{*}$ by
minimizing the upper bound:

$\displaystyle\mathbb{E}_{\mathbf{x}^{+}_{0},\mathbf{x}^{-}_{0},t,\mathbf{x}^{+}_{t},\mathbf{x}^{-}_{t}}$
$\displaystyle\Bigg{[}-\log\sigma\bigg{(}\beta\big{(}\lVert\bm{\epsilon}_{\text{base}}(\mathbf{x}^{+}_{t},\mathbf{c},t)-\bm{\epsilon}^{+}_{t}\rVert^{2}_{2}-\lVert\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}^{+}_{t},\mathbf{c},t)-\bm{\epsilon}^{+}\rVert^{2}_{2}$
$\displaystyle-(\lVert\bm{\epsilon}_{\text{base}}(\mathbf{x}^{-}_{t},\mathbf{c},t)-\bm{\epsilon}^{-}_{t}\rVert^{2}_{2}-\lVert\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}^{-}_{t},\mathbf{c},t)-\bm{\epsilon}^{-}\rVert^{2}_{2})\big{)}\bigg{)}\Bigg{]},$

where $\\{\mathbf{x}^{+}_{t}\\}_{t=0}^{T}$ represents the preference
trajectory, i.e.,
$r(\mathbf{x}^{+}_{0},\mathbf{c})>r(\mathbf{x}^{-}_{0},\mathbf{c})$, and
$\bm{\epsilon}^{+}$ and $\bm{\epsilon}^{-}$ are independent samples from
Gaussian distribution. The detailed derivation can be found in [27].

###  A.2 Experimental Details

#### Training Prompts.

We collect 8 training prompts: 6 re-contextualization, 1 property modification
and 1 artistic style transfer for objects. 5 re-contextualization, 1 attribute
editing, 1 artistic style transfer and 1 accessorization for live subjects.
The trainig prompts are shown in Figure 6

Figure 6: Training prompts for objects and live subjects.

#### Experimental Setup

During training we use $\lambda_{\text{train}}=0$ for the $\lambda$-Harmonic
reward function to generate the preference labels. We evaluate the model
performance by $\lambda_{\text{val}}$-Harmonic per 40 gradient steps during
training time and save the checkpoint that achieve the highest validation
reward. Table 4 lists the common hyperparameters used in the generating skill
set and the $\lambda_{\text{val}}$ used in the default setting.

Table 4: RPO Hyperparameters Parameter | Value
---|---
Optimization |
optimizer | AdamW [15]
learning rate | $5\times 10^{-6}$
weight decay | $0.01$
gradient clip norm | $1.0$
RPO |
regularizer weight, $\beta$ | $1.0$
gradient steps | $400$
training preference weight, $\lambda_{\text{train}}$ | $0.0$
validation preference weight (default), $\lambda_{\text{val}}$ | $0.3$

###  A.3 Additional Comparisons

We observe RPO that can be faithful to both reference images and the input
prompt. To investigate whether RPO can provide better quality than DreamBooth
and SuTI, we follow SuTI paper and pick the robot toy as an example to
demonstrate the effectiveness of RPO (Figure 7). In this example, DreamBooth
is faithful to the reference image but it does not provide a good text-to-
image alignment. SuTI provides an result that is fidelty to textual prompt.
However, SuTI lacks fidelity to the reference image, i.e., the robot should
stand with its wheels instead of legs. [7] use DreamBooth to finetune SuTI
(Dream-SuTI) further to solve this failure case. Instead, RPO can generate an
image not only faithful to the reference images but also align well with the
input prompts.

We have also added more samples for comparison of different
$\lambda_{\text{val}}$ values (see Figure 8). We find that
$\lambda_{\text{val}}=0.5$ encourages the learned model to retain output
diversity while still aligning with textual prompts. However, the generated
images invariably contain a sofa, which is unrelated to the subject images.
This occurs because every training image includes a sofa. A large
$\lambda_{\text{val}}$ weakens the regularization strength and leads to
overfitting. Nevertheless, a small value of $\lambda_{\text{val}}$ can
potentially eliminate background bias. We highlight that this small
$\lambda_{\text{val}}$ not only encourages diversity but also mitigates
background bias in identity preservation and enables the model to focus on the
subject.

Figure 7: Comparison between DreamBooth, SuTI, Dream-SuTI, and RPO, adapted
from [7] Figure 8: Additional samples of different $\lambda_{\text{val}}$. A
small $\lambda_{\text{val}}$ leads to more diverse generated images.

###  A.4 Limitations

Figure 9 illustrates some failure examples of RPO. The first issue is context-
appearance entanglement. In Figure 9, the learned model correctly understands
the keyword blue house; however, the appearance of the subject is also altered
by this context, e.g. the color of the backpack has changed, and there is a
house pattern on the backpack. The second issue is incorrect contextual
integration. We conjecture that this failure is due to the rarity of the
textual prompt. For instance, imagining a cross between a chow chow and a
tiger is challenging, even for humans. Third, although RPO provides
regularization, it still cannot guarantee the avoidance of overfitting. As
shown in Figure 9, this may be because, to some extent, the visual appearance
of sand and bed sheets is similar, which has led to overfitting issues in the
model.

###  A.5 Future Work

The overfitting failure case leads to a future work direction: can online RL
improve regularization and avoid overfitting? The second direction for future
work involves implementing the LoRA version for RPO and comparing it to LoRA
DreamBooth. Last but not least, we aim to identify or construct open-source,
subject-driven datasets for comparison. Currently, DreamBench is the only
open-source dataset we can access and evaluate for model performance.
Nevertheless, we should create a larger dataset that includes more diverse
subjects to verify the effectiveness of different algorithms.

Figure 9: RPO’s failure modes include: (1) The context and the appearance of
the subject becoming entangled. (2) The trained model failing to generate an
image with respect to the input prompt. (3) The trained model still
overfitting to the training set.

###  A.6 Broader Impacts

The nature of generative image models is inherently subjected to criticism on
the issue of privacy, security and ethics in the presence of nefarious actors.
However, the core of this paper remains purely on an academic mission to
extend the boundaries of generative models. The societal consequences of
democratizing such powerful generative models is more thoroughly discussed in
other papers. For example, Bird et al. outlines the classes of risks in text-
to-image models [2] in greater detail and should be directed to such papers.
Nevertheless, we play our part in the management of such risks by avoiding the
use of identifiable parts of humans in the reference sets.

###  A.7 Skill Set

The skill set of the RPO-trained model is varied and includes re-
contextualization (Figure 10), artistic style generation (Figure 11),
expression modification (Figure 12), subject accessorization (Figure 13),
color editing (Figure 14), multi-view rendering (Figure 15), and novel hybrid
synthesis (Figure 16).

Figure 10: Re-contextualization samples from the RPO algorithm. RPO is able to
generate images of specific subjects in unseen environments while preserving
the identity and details of the subjects and be faithful to the input prompts.
Reference images are shown on the left. We display the generated images along
with their textual prompts on the right. Figure 11: Artistic style rendition
samples from the RPO algorithm. These art renditions can be applied to a dog
given a prompt of ‘‘a [painter] styled painting of a [V] dog’’. The identity
of the subject is preserved and faithfully imitates the style of famous
painters. Figure 12: Expression modification samples from the RPO algorithm.
RPO can integrate subject with various unseen expressions in the reference
images. We also notice the pose of generated images, e.g., a [V] sleepy dog,
were not displayed in the training set. Figure 13: Accessories samples from
the RPO algorithm. Conditioned the prompts, ‘‘a [V] chow chow wearing a
[target outfit]’’, the generated images retains the unique features of the
reference images, e.g., the hair color and breed of the subject dog. The
interaction between subject dog and the outfits is realistic. Figure 14: Color
editing samples from the RPO algorithm. We display color modifications using
prompts such as ‘‘a [target color] [V] monster toy’’. The identity of the
subject is preserved. Figure 15: Multi-view samples from the RPO algorithm. We
generate images from specified viewpoints of the subject. For the top and
bottom views, we use the prompts ‘‘a [V] cat looking up/down at the camera’’.
For the back and side views, we apply the prompts ‘‘a back/side view of a [V]
cat’’. Figure 16: Novel hybrid synthesis samples from the RPO algorithm. We
apply the prompts ‘‘a cross of a [V] chow chow and [target species]’’ to the
RPO-trained model to generate these images. We highlight that RPO can
synthesize new hybrids that retain the identity of the subject chow chow and
perform property modifications.


††thanks: Corresponding author

# Magnetic-field and pressure phase diagram of the triangular lattice
antiferromagnet CsCuCl3 clarified by magnetic susceptibility measured with a
proximity detector oscillator

K. Nihongi  T. Kida  Y. Narumi  Center for Advanced High Magnetic Field
Science (AHMF), Graduate School of Science, Osaka University, Toyonaka, Osaka
560-0043, Japan  J. Zaccaro  Grenoble Alpes University, CNRS, Grenoble INP,
Institut N$\acute{e}$el, 38042 Grenoble, France  Y. Kousaka  Department of
Physics and Electronics, Osaka Prefecture University, Sakai, Osaka 599-8531,
Japan  K. Inoue  Graduate School of Science, Hiroshima University, Hiroshima
739-8526, Japan  Center for Chiral Science, Hiroshima University, Hiroshima
739-8526, Japan  K. Kindo  Y. Uwatoko  The Institute for Solid State Physics,
The University of Tokyo, Kashiwa, Chiba 277-8581, Japan  M. Hagiwara  Center
for Advanced High Magnetic Field Science (AHMF), Graduate School of Science,
Osaka University, Toyonaka, Osaka 560-0043, Japan  Center for Chiral Science,
Hiroshima University, Hiroshima 739-8526, Japan
<EMAIL_ADDRESS>
###### Abstract

The effect of pressure ($P$) on magnetic susceptibility of CsCuCl3 was
examined in magnetic fields ($\rm\mu_{0}$$H$) of up to 51 T using a proximity
detector oscillator (PDO), and the $H$-$P$ phase diagram of CsCuCl3 was
constructed over the saturation field ($H_{\rm sat}$). We found that, with
increasing $P$, $H_{\rm sat}$ increases and the uud-phase that appeared at $P$
= 0.7 GPa widened. Based on comparison between the experimental and calculated
$H$-$P$ phase diagrams, the Y-phase was predicted to appear above 1.7 GPa. The
interchain antiferromagnetic exchange interaction in the $ab$-plane was
evaluated and found to increase with increasing $P$, which is consistent with
a previous study under high pressure [D. Yamamoto et al., Nat. Commun. 12,
4263 (2021).]. Moreover, an anomaly was observed below $P$ = 0.6 GPa just
below $H_{\rm sat}$ and might be a new phase transition derived from nonlinear
response caused by the PDO technique.

##  I Introduction

Geometrically frustrated magnets have been studied over the past several
decades owing to a rich variety of exotic physical phenomena Anderson (1973);
Balents (2010). One of the most typical geometrically frustrated magnets is
the two-dimensional triangular lattice antiferromagnet (TLA) Starykh (2015).
The ground state infinitely degenerates in magnetic fields, and a small
perturbation, such as easy-plane anisotropy, lifts the degeneracy and
stabilizes the one-energy state Yamamoto  _et al._ (2014), for example, an
umbrella spin structure in low-magnetic fields normal to the easy plane. For
classical spin systems, extensive theoretical studies have been conducted
Kawamura and Miyashita (1985); Chubukov and Golosov (1991); Seabra  _et al._
(2011), and one of the experimental realizations is RbFe(MoO4)2 Inami  _et
al._ (1996); Smirnov  _et al._ (2007). When spins are small, quantum
fluctuations play an important role in lifting degeneracy in some cases. For
TLAs, the up-up-down (uud) state, which corresponds to a magnetization plateau
at one-third of the saturation magnetization, is an exotic ground state
Kawamura and Miyashita (1985); Chubukov and Golosov (1991); Seabra  _et al._
(2011); Sakai and Nakano (2011); Coletta  _et al._ (2016) caused by the order-
by-disorder mechanism Henley (1989) in magnetic fields. To date, Cs2CuBr4
Tanaka  _et al._ (2002); Ono  _et al._ (2003, 2004) (not equilateral) and
Ba3CoSb2O9 Shirata  _et al._ (2012); Susuki  _et al._ (2013) are known to
exhibit the uud ground state of $\it S$ = 1/2 TLAs in magnetic fields.

CsCuCl3 has been studied for a long time as one of ABX3 compounds Achiwa
(1969). It is a TLA compound formed by ferromagnetic chains and exhibits a
quantum phase transition in magnetic fields Nojiri  _et al._ (1988); Nikuni
and Shiba (1993). The crystal structure of CsCuCl3 at room temperature is
hexagonal with a space group $\it P$6122 or $\it P$6522 Wells (1947);
Schlueter  _et al._ (1966). Magnetic Cu2+ ions of CsCuCl3 with $\it S$ = 1/2
form a helical chain structure with a sixfold periodicity along the
$c$-direction and triangular lattices in the $ab$-plane. The spin Hamiltonian
for CsCuCl3 in magnetic fields Nikuni and Shiba (1993) is given by,

$\displaystyle\mathcal{H}=$ $\displaystyle-$ $\displaystyle
2J_{0}\sum_{i,n}(\mbox{\boldmath$S$}_{i,n}\cdot\mbox{\boldmath$S$}_{i,n+1}+\eta(S_{i,n}^{x}S_{i,n+1}^{x}+S_{i,n}^{y}S_{i,n+1}^{y}))$
(1) $\displaystyle+$ $\displaystyle
2J_{1}\sum_{<i,j>,n}(\mbox{\boldmath$S$}_{i,n}\cdot\mbox{\boldmath$S$}_{i,n+1})-\sum_{i,n}\mbox{\boldmath$D$}_{n,n+1}\cdot(\mbox{\boldmath$S$}_{i,n}\times\mbox{\boldmath$S$}_{i,n+1})$
$\displaystyle-$ $\displaystyle g{\rm\mu_{B}}H\sum_{i,n}S_{i,n}^{z},$

where $S_{i,n}$ represents a spin located at the $i$-th site in the $n$-th
$ab$-plane, and the summation $\langle$$ij$$\rangle_{n}$ is taken over all
nearest-neighbor pairs in the $n$-th $ab$-plane. The $z$-axis is assumed to be
parallel to the $c$-axis. The first and second terms are the intrachain
ferromagnetic exchange (exchange constant $J_{0}>0$) and interchain
antiferromagnetic exchange (exchange constant $J_{1}>0$) interactions,
respectively. $\eta$ ($>$ 0) represent a weak anisotropic exchange ratio for
the easy plane. The third term represents the Dzyaloshinskii-Moriya (DM)
interaction where $D$n,n+1 represents the DM vector between adjacent spins in
the $n$-th and ($n$+1)-th $c$-planes. The fourth term is the Zeeman energy.
$g$ and $\rm\mu_{\rm B}$ are the $g$-factors for the $z$-axis and Bohr
magneton, respectively.

The intrachain and interchain exchange constants were evaluated based on the
following analyses: $J_{0}$/kB = 24 $\pm$ 3 K and $J_{1}$/kB = 3.8 $\pm$ 1 K
from magnetic susceptibilities Tazuke  _et al._ (1981), $J_{0}$/kB = 28 K and
$J_{1}$/kB = 4.9 K from electron spin resonance (ESR) excitation modes for
$H\parallel c$ Tanaka  _et al._ (1992), and $J_{0}$/kB = 28 K (2.4 meV) and
$J_{1}$/kB = 4.6 K (0.4 meV) from inelastic neutron scattering data Mekata
_et al._ (1995). The Cu spins exhibited a 120∘ spin structure in the
$ab$-plane below the N$\acute{e}$el temperature $T_{\rm N}$ = 10.7 K Adachi
_et al._ (1980); Weber  _et al._ (1996). The Cu chain structure along the
$c$-axis allows DM interaction with a uniform $\bm{D}$ vector pointing in the
chain direction ($D$/kB = 5.1 K Tanaka  _et al._ (1992) or 5.8 K Mekata  _et
al._ (1995)) owing to the chiral crystal structure Adachi  _et al._ (1980);
Kousaka  _et al._ (2009). Below $T_{\rm N}$, the nearest-neighbor spins along
the $c$-axis were rotated with a pitch angle of $\theta$ = 5.1 $\pm$0.1∘ in
the $ab$-plane Adachi  _et al._ (1980). In the following analysis of our
experimental data, we used the parameter values obtained by the analysis of
ESR excitation modes Tanaka  _et al._ (1992): $J_{0}$/kB = 28 K and $J_{1}$/kB
= 4.9 K, and $D$/kB = 5.1 K .

The magnetization ($M$) of CsCuCl3 for $H\parallel c$ at 1.1 K increased
almost linearly and showed a jump at 12.5 T, then increased again and
saturated at 31 T Nojiri  _et al._ (1988). Nikuni and Shiba interpreted the
magnetization process of CsCuCl3 for $H\parallel c$ Nikuni and Shiba (1993)
using the spin Hamiltonian (Eq.(1)). They eliminated the asymmetric part from
the Hamiltonian by rotating the $xy$-plane with pitch $q$ = tan-1
($D$/2$J_{0}$ (1 + $\eta$)). Thereafter, the spin Hamiltonian can be simply
rewritten as:

$\displaystyle\mathcal{H}=$ $\displaystyle-$
$\displaystyle\sum_{i,n}[2\tilde{J}_{0}(S_{i,n}^{x}S_{i,n+1}^{x}+S_{i,n}^{y}S_{i,n+1}^{y})+2J_{0}S_{in}^{z}S_{in+1}^{z}]$
(2) $\displaystyle+$ $\displaystyle
2J_{1}\sum_{<ij>n}\mbox{\boldmath$S$}_{in}\cdot\mbox{\boldmath$S$}_{jn}-g{\rm\mu_{B}}H\sum_{i,n}S_{i,n}^{z},$

where $\tilde{J}_{0}$ = $J_{0}$$\sqrt{(1+\eta)^{2}+(D/2J_{0})^{2}}$. They
defined the easy-plane anisotropy as,

$\Delta\equiv(\tilde{J}_{0}-J_{0})/J_{1}.$ (3)

Thereafter, the value of saturation field is given by,

$H_{\rm sat}=(18+4\Delta)J_{1}S/g\rm\mu_{B}.$ (4)

They interpreted that the jump in $M$ of CsCuCl3 for $H$ $\parallel$ $c$ was
caused by the quantum phase transition from the umbrella to 2-1 coplanar
phase. Their theoretical predictions were confirmed by neutron diffraction
Mino  _et al._ (1994) in magnetic fields.

The application of high pressure is a powerful method that can alter the
magnetic properties of a material by shrinking its crystal lattice. As for
TLAs, magnetic anisotropy and the exchange interaction between magnetic ions
vary through the change in distance and angle between the neighboring magnetic
ions through intervening nonmagnetic ions by applying pressure. For the
anisotropic TLA Cs2CuCl4, the appearance of a number of field-induced magnetic
phase transitions upon pressure application have been reported, accompanied by
an increase in the exchange coupling ratio $J^{\prime}$/$J$, where $J$ and
$J^{\prime}$ are the exchange interactions on the horizontal and diagonal
bonds in the spatially anisotropic triangular lattice, respectively Zvyagin
_et al._ (2019).

Sera et al. Sera  _et al._ (2017) performed magnetization measurements of
CsCuCl3 at 1.4 K in magnetic fields ($\rm\mu$0$H$ up to 15 T) along the
$c$-axis under high pressure ($P$ up to 0.9 GPa) and observed the
magnetization plateau with 0.34 $\rm\mu$B/Cu2+ under $\it P$ $\geq$ 0.68 GPa,
which corresponds to the uud phase in $M$. Hosoi et al. tried to explain the
magnetization under high pressure by assuming that $\Delta$ was sensitive to
pressure Hosoi  _et al._ (2018). They constructed the magnetic phase diagram
of CsCuCl3 at 0 K under pressure by calculating the spin Hamiltonian based on
spin-wave theory with $\Delta$ ($\Delta$ = 0.07 at ambient pressure) as a
variable of pressure. $\Delta$ decreased with increasing pressure, and the uud
and Y coplanar phases under high pressure ($\Delta$ $\leq$ 0.06) were
predicted to appear with the fixed parameters of $J_{0}$ and $\tilde{J}_{0}$.
Yamamoto et al. demonstrated theoretically that the ratio of $J_{0}$ to
$J_{1}$ reduces with pressure, and then the uud and Y coplanar phases appeared
under $P\geq$ 0.75 GPa, corresponding to the results of magnetic measurement
of CsCuCl3 under high pressure up to 1.21 GPa in magnetic fields of up to 5 T
Yamamoto  _et al._ (2021). In both theories, the change in the ratio of
$J_{0}$ to $J_{1}$ was important for the appearance of the field- and
pressure-induced magnetic phases. A quantitative comparison of the $H$-$P$
phase diagram of CsCuCl3 requires $J_{1}$ under high pressure which was
directly estimated from $H_{\rm sat}$. Therefore, in this study, we
investigated the pressure dependence of the entire magnetization process of
CsCuCl3 over the saturation field using a proximity detector oscillator (PDO)
system.

The remainder of this study is organized as follows. In Section 2, the sample
preparation of single crystals of CsCuCl3, experimental methods and conditions
are described. The experimental results of the pressure dependence of magnetic
susceptibility up to 1.72 GPa and the magnetic-field ($H$) versus pressure
($P$) phase diagram obtained experimentally are presented in Section 3. In
Section 4, the constructed $H$-$P$ phase diagram is compared with the
calculated phase diagram by Hosoi et al. Hosoi  _et al._ (2018). The
similarities and differences between them are discussed. Finally, the origin
of the anomaly observed immediately below the saturation field in the
$c$-direction is discussed, followed by a summary of this study.

##  II Experimental

Single crystals of CsCuCl3 were grown by evaporating an aqueous solution
containing equimolar amounts of CsCl and CuCl2$\cdot$2H2O. Pulsed magnetic
fields of up to 51 T with a pulse duration of 35 ms were generated using a
nondestructive pulse magnet at AHMF at Osaka University. High-field
magnetization measurements were performed using an induction method with a
pick-up coil at temperatures down to 1.4 K at ambient pressure. A PDO system
Altarawneh  _et al._ (2009) was utilized to measure the change in the PDO
resonance frequency ($\Delta$$f$), which corresponded to the field change of
the magnetization ($\Delta$$M$/$\Delta$$H$), namely magnetic susceptibility,
in a magnetic insulator, as explained below, under high pressures of up to
1.72 GPa. When the magnetic insulator was placed inside the sensor coil of the
LC tank circuit, the change in $\Delta$$f$ in the magnetic field was
proportional to $\Delta$$M$/$\Delta$$H$ Ghannadzadeh  _et al._ (2011). In this
study, we used a NiCrAl piston cylinder-type pressure cell (PCC) with the
inner and outer diameters of 2.0 and 8.6 mm, respectively, and the pressure
medium Daphne 7373 oil (Idemitsu Kosan Co.,Ltd.). The pressure in the sample
space was calibrated using the change in the superconducting transition
temperature of Sn Smith and Chu (1967). A large field sweep rate of pulsed
magnetic fields induced the Joule heating caused by eddy currents generated in
a conducting material. Based on our preliminary experiments, we found that the
temperature in the sample space changed slightly until approximately 6.5 ms
(approximately 40 T in the ascending process when the maximum field was 51 T)
from the start of magnetic-field generation.

##  III Experimental Results

Figure 1: (a) Magnetization ($M$) and d$M$/d$H$ curves at 1.4 K observed in
CsCuCl3 for $H$ parallel to the $c$-axis. The change in resonance frequencies
( $\Delta$$f$sub) against applied magnetic field for the $c$-axis of CsCuCl3
at (b) 1.4 K and (c) various temperatures. $\Delta$$f$sub is the frequency
difference obtained by subtracting the frequency at $T$ = 20 K as background
from that at the measurement temperature. (In (a)-(c), solid and dotted lines
are field ascending and descending processes, respectively.) Figure 2: (a) $M$
($H$) and d$M$/d$H$ curves for $H\perp c$ at 1.4 K. (b) The change in
resonance frequency ($\Delta$$f$sub) for $H\perp c$ at 1.4 K. $\Delta$$f$sub
is the same definition as in the caption of Fig. 1.

Figure 1(a) shows the magnetization ($M$) and its field derivative (d$M$/d$H$)
of CsCuCl3 for $H\parallel c$ at 1.4 K under ambient pressure. A small jump at
$H_{\rm c1}$ = 12.5 T in $M$, which corresponds to a peak in d$M$/d$H$,
indicates the phase transition from the umbrella to 2-1 coplanar phase. A
small hysteresis in $M$ and d$M$/d$H$ was observed around $H_{\rm c1}$,
indicating a first-order phase transition, and $M$ was saturated at $H_{\rm
sat}$ = 30.2 T. These features were consistent with those in a previous study
Nojiri  _et al._ (1988). Figure 1(b) shows the change in PDO resonance
frequency versus the applied field ($\Delta$$f$sub-$H$ ) curves of CsCuCl3 for
$H\parallel c$ at 1.4 K under ambient pressure. The $\Delta$$f_{\rm sub}$
value was obtained by subtracting the PDO frequency at $T$ = 20 K as the
background from that at each temperature. The $\Delta$$f_{\rm sub}$-$H$ curve
at 1.4 K under ambient pressure indicates shoulders at $H_{\rm c1}$ and
$H_{\rm sat}$, and a sharp peak at $H^{\prime}{}_{\rm sat}$ = 29.8 T, just
below $H_{\rm sat}$ (Fig.1 (b) inset). An anomaly between 18 and 24 T in the
field-ascending process was observed below 4.2 K, when the applied magnetic
field went beyond $H_{\rm c1}$, but the field range of the anomaly depends on
the maximum applied field. The reason for this observation is currently
unclear. The temperature dependence of the $\Delta$$f$sub-$H$ curve under
ambient pressure is shown in Fig.1 (c). The shoulder at $H_{\rm c1}$ shifted
to a lower field with increasing temperature and disappeared above $T_{\rm N}$
(=10.7 K). With increasing temperature, the shoulder at $H_{\rm sat}$ and the
sharp peak at $H^{\prime}{}_{\rm sat}$ shifted together to a lower magnetic
field, merged above 6.0 K and disappeared above $T_{\rm N}$.
$H^{\prime}{}_{\rm sat}$ may appear because of a magnetic phase transition
caused by the AC response (MHz region) in the PDO measurements. In comparison,
pulsed-field magnetization measurement was the low-frequency AC response at
approximately 25 Hz and no sharp peak at $H^{\prime}{}_{\rm sat}$ in d$M$/d$H$
was observed with the conventional method.

In the $M$ and d$M$/d$H$ curves for $H\perp c$ at 1.4 K under ambient pressure
(Fig. 2 (a)), a plateau-like anomaly in $M$, which corresponds to a dip in
d$M$/d$H$, was observed between 10 and 14 T, and the $M$ was saturated at
$H_{\rm sat}$ = 29.3 T. Figure 2(b) exhibits the $\Delta$$f$sub-$H$ curve for
$H\perp c$ at 1.4 K under ambient pressure. A dip between 10 and 14 T and a
peak at $H_{\rm sat}$ were observed. The sharp peak just below $H_{\rm sat}$
seen in the $\Delta$$f$sub-$H$ curves for $H\parallel c$ was not observed in
the $\Delta$$f$sub-$H$ curve for $H\perp c$.

Figure 3:  Changes in resonance frequencies $\Delta$$f$sub ) of CsCuCl3 for
$H\parallel c$ at indicated pressures at 1.4 K. $\Delta$$f$sub is the same
definition as in the caption of Fig. 1. Each datum was taken from the field
ascending process and shifts arbitrarily making it easier to see except for
the ambient data. The small red circle at 1.67 GPa near $H_{\rm c1}$ is an
anomaly which may correspond to the transition to the Y-phase. The data at
ambient pressure were taken without pressure cell. Figure 4: Magnetic field
($\rm\mu_{0}$$H$) versus pressure ($P$) phase diagram of CsCuCl3 for
$H\parallel c$ at 1.4 K constructed by the PDO measurements.  Figure 5:
Pressure ($P$) dependence of the interchain (in-plane) exchange constant
$J_{1}$ and the anisotropy parameter $\Delta$, when the intrachain exchange
constant $J_{0}$ is assumed to be constant under high pressure. Figure 6: (a)
$H$/$H_{\rm sat}$ versus $P$ phase diagram of CsCuCl3 for $H\parallel c$ at
1.4 K. Solid, broken, and dot-dashed lines are the phase boundary between the
umbrella and 2-1 coplanar phases, between the umbrella and uud phases, and
between the uud and 2-1 coplanar phases, respectively. (b) Calculated
$H$/$H_{\rm sat}$ versus $\Delta$ phase diagram constructed by reversing the
horizontal axis from the phase diagram in Ref. [Sera  _et al._ , 2017]. The
smaller $\Delta$ means the higher $P$, and $\Delta$ = 0.07 corresponds to
ambient pressure. By comparison, $\Delta$ = 0.055 may correspond to
$P$$\sim$1.7 GPa.

The pressure dependence of the $\Delta$$f$sub-$H$ curves of CsCuCl3 for
$H\parallel c$ at 1.4 K is shown in Fig. 3. The position of $H_{\rm c1}$ did
not change significantly under $P$ $<$ 0.90 GPa. However, the shoulder at
$H_{\rm c1}$ changed its structure to a dip with two shoulders when $P\geq$
0.90 GPa. We define the low and high transition fields as $H_{\rm c1}$ and
$H_{\rm c2}$, respectively. $H_{\rm c1}$ was nearly constant with increasing
pressure even above 0.90 GPa. Contrarily, $H_{\rm c2}$ shifted to a higher
magnetic field with increasing pressure above 0.90 GPa. The dip between
$H_{\rm c1}$ and $H_{\rm c2}$ is suggested to be the uud phase reported in a
previous study Nojiri  _et al._ (1988). The anomaly between 18 and 24 T at
ambient pressure shifted to a lower magnetic field with increasing pressure
and disappeared under $P\geq$ 0.62 GPa. The $H^{\prime}{}_{\rm sat}$ and
$H_{\rm sat}$ shifted together to a higher magnetic field with increasing
pressure, and merged into one peak or broaden to form one peak at $P\geq$ 0.90
GPa. The increase in $H_{\rm sat}$ indicates an increase in the
antiferromagnetic interaction in the $ab$-plane of CsCuCl3, which may be
caused by the shrinkage of the lattice in the $ab$-plane of CsCuCl3 under
pressure Christy  _et al._ (1994).

The magnetic field ($\rm\mu_{0}$$H$) versus the pressure phase diagram of
CsCuCl3 for $H\parallel c$ at 1.4 K is shown in Fig. 4. The pressure
dependence of $H_{\rm sat}$ is well-fitted by the equation $H_{\rm sat}$ =
30.2 + 2.14$P$ \+ 2.14$P^{2}$. $H_{\rm sat}$ is mainly determined by the
antiferromagnetic exchange interaction in the $ab$-plane, as shown in Eq. (4).
The field range of the uud phase between $H_{\rm c1}$ and $H_{\rm c2}$
expanded to a high-magnetic-field side with increasing pressure. The dashed
lines extrapolated to the lower side from $H_{\rm c1}$ and $H_{\rm c2}$
intersect at approximately 0.7 GPa, which is in good agreement with the
appearance of a magnetic plateau in $M$ above 0.68 GPa in statice fields
observed by Sera et al. Sera  _et al._ (2017). The green magnetic phase near
the saturation field in Fig. 4 is discussed in the next section.

##  IV Discussion

###  IV.1 Magnetic field ($\rm\mu_{0}$$H$) vs. pressure ($P$) phase diagram

We determined the interchain exchange constant $J_{1}$ and anisotropy
parameter $\Delta$ from Eqs. (3) and (4), where the ferromagnetic intrachain
exchange constant $J_{0}$/kB = 28 K and $\tilde{J}_{0}$ = 1.012 $J_{0}$ were
fixed under pressure, and $J_{1}$ ($>$ 0) is the interchain antiferromagnetic
exchange constant, as defined in the introduction. Figure 5 shows the pressure
dependence of $J_{1}$ and $\Delta$. The interchain exchange constant
($J_{1}$/kB = 4.8 K at ambient pressure) was obtained from the saturation
field, which is almost the same as that in Ref. [Tanaka  _et al._ , 1992].
$J_{1}$ increased with increasing pressure, whereas $\Delta$ decreased. The
$J_{1}$/kB and $\Delta$ were evaluated to be 6.4 K and 0.053 under the maximum
pressure $P$ = 1.72 GPa, respectively. In the pressure range of this study,
$\Delta$ decreased from 0.070 to 0.053 (4$\Delta$ $\ll$ 18). $J_{1}$ depended
strongly on $H_{\rm sat}$, as shown in Eq. (4), because the change in $\Delta$
under pressure was quite small. Therefore, the pressure dependence of $J_{1}$
did not change much in both cases of $J_{0}$/kB = constant and $J_{0}$/kB =
28.45 - 10.49$P$ reported in Ref. [Yamamoto  _et al._ , 2021] . The magnetic-
field normalized by the $H_{\rm sat}$ versus pressure phase diagram of CsCuCl3
is shown in Fig. 6(a). The similar calculated phase diagram by Hosoi et al.
Hosoi  _et al._ (2018) is depicted in Fig. 6(b).The pressure dependences of
$H_{\rm c1}$/$H_{\rm sat}$ and $H_{\rm c2}$/$H_{\rm sat}$ qualitatively agree
with the calculation Hosoi  _et al._ (2018), and similar results were obtained
by Yamamoto et al. Yamamoto  _et al._ (2021). The emergence of the Y coplanar
phase was expected at $\Delta$ $\leq$ 0.05 and $H_{\rm c1}$/$H_{\rm sat}$
$\leq$ 0.3, as shown in Fig. 6 (b). In the $\Delta$$f$sub-$H$ curves for
$H\parallel c$ at 1.4 K under $P$ = 1.67 GPa, a small peak indicated by a red
circle at $H_{\rm c1}$ corresponds to the transition field from the umbrella
to predicted Y coplanar phase Hosoi  _et al._ (2018). The Y phase was expected
to appear below $H_{\rm c1}$/$H_{\rm sat}$ = 0.3 in Ref. [Hosoi  _et al._ ,
2018] (Fig. 6 (b)), but the experimentally observed $H_{\rm c1}$/$H_{\rm sat}$
was approximately 0.3 at the maximum pressure in this study. Therefore, we
plan to perform PDO measurements at pressures above 1.72 GPa. A crystal
structural phase transition in CsCuCl3, however, was predicted at pressures
between 1.65 and 3.1 GPa Christy  _et al._ (1994), and this transition may
hinder the appearance of the Y-phase.

###  IV.2 The anomaly just below the saturation field

Figure 7:  Angular ($\theta$) dependence of $H^{\prime}{}_{\rm sat}$ and
$H_{\rm sat}$ of CsCuCl3 at 1.4 K. The solid lines are the fitting curves of
$H_{\rm sat}$ and $H^{\prime}{}_{\rm sat}$ with $H_{\rm sat}$ =
$A$cos2$\theta$ \+ $B$cos4$\theta$ \+ $C$ and $H^{\prime}{}_{\rm sat}$ =
$A$′cos2$\theta$ \+ $B$′cos4$\theta$ \+ $C$′ ($A$, $B$, $C$, $A$′, $B$′, and
$C$′ are fitting parameters).

The peak at $H^{\prime}{}_{\rm sat}$ observed in the $\Delta$$f$sub-$H$ curve
for $H\parallel c$ did not appear in the case of $H\perp c$. The angular
dependences of $H_{\rm sat}$ and $H^{\prime}{}_{\rm sat}$ are shown in Fig. 7,
where $\theta$ is the angle between the applied magnetic field and $c$-axis.
$H_{\rm sat}$ shifted gradually to a lower magnetic field with increasing.
$\theta$, but $H^{\prime}{}_{\rm sat}$ shifted abruptly to a lower magnetic
field with increasing $\theta$ until $\theta$ = 45∘, and then disappeared at
$\theta$ $>$ 45∘. The angle dependences of $H_{\rm sat}$ and
$H^{\prime}{}_{\rm sat}$ were fitted by the equations $H_{\rm sat}$ =
$A$cos2$\theta$ \+ $B$cos4$\theta$ \+ $C$ and $H^{\prime}{}_{sat}$ =
$A$′cos2$\theta$ \+ $B$′cos4$\theta$ \+ $C$′($A$, $B$, $C$, $A$′, $B$′ and
$C$′ are fitting parameters). The following parameter values were obtained
from the fittings: $A$ = 0.448, $B$ = 0.078, $C$ = 29.8, $A$′ = 1.69, $B$′ =
-0.280, and $C$′ = 28.4. In both cases, the coefficients of cos2$\theta$ were
dominant, although their magnitudes were largely different.

In the d$M$/d$H$ curve of the triangular lattice antiferromagnet
NiBr2Katsumata  _et al._ (1983), a sharp peak, which corresponds to the
antiferro-fan transition, was observed just below the saturation field. If the
peak at $H^{\prime}{}_{\rm sat}$ indicates a magnetic transition from the 2-1
coplanar to fan phase, it should be observed in conventional magnetization
measurements of CsCuCl3. However, no anomaly at $H^{\prime}{}_{\rm sat}$ was
observed in the $M$ and d$M$/d$H$ curves (Fig. 1(a)).

Why was the $H^{\prime}{}_{\rm sat}$ anomaly observed in the PDO measurements?
In the PDO measurements, $\Delta M$/$\Delta H$ in the insulating sample
CsCuCl3 was taken as $\Delta$$f$ in high-frequency response (of the order of
ten mega-hertz). The high-frequency response in magnetic fields may give rise
to a nonlinear magnetic response owing to the third-harmonic component of
$\chi_{3}$ in the AC magnetic susceptibility measurements ($M$ in the AC
magnetic measurements is given by $M$ ($H$) = $M$0 +$\chi_{1}$$H$
+$\chi_{3}$$H^{3}$ \+ $\cdot\cdot$). To clarify this interpretation, AC
magnetic susceptibility measurements of CsCuCl3 in high magnetic fields are
required.

##  V Summary

We experimentally performed magnetic measurements of single-crystal samples of
CsCuCl3 in pulsed-high magnetic fields under high pressure using the PDO
technique. We succeeded in measuring the shift of the PDO frequency that
corresponds to the field derivation of magnetization over the saturation
field, and clarified the pressure dependence of $J_{0}$ and $\Delta$. The
magnetic field versus pressure phase diagram of CsCuCl3 constructed from the
experimental results agrees qualitatively with that obtained by theoretical
calculations Hosoi  _et al._ (2018); Yamamoto  _et al._ (2021).The anomaly
just below the saturation field for $H\parallel c$ at 1.4 - 4.2 K was observed
in the PDO measurements and not in the conventional magnetization
measurements. This anomaly may correspond to the transition field caused by
the nonlinear AC magnetic response in PDO measurements.

###### Acknowledgements.

We thank T. Sakurai, D. Yamamoto and T. Takeuchi for the fruitful discussion
on the magnetism of CsCuCl3 under pressure. We would like to thank Editage
(www. editage.com) for English language editing. This study was financially
supported by the Sasakawa Scientific Research Grant from The Japan Science
Society and Osaka University Fellowship: “Super Hierarchical Materials Science
Program”.This study was also supported by JSPS KAKENHI (Grant Nos JP17H06137,
JP17K18758, JP21H01035, JP19H00648, 25220803) and JSPS Core-to-Core Program,
A. Advanced Research Networks.

## References

  * Anderson (1973) P. W. Anderson, Materials Research Bulletin 8, 153 (1973).
  * Balents (2010) L. Balents, Nature 464, 199 (2010).
  * Starykh (2015) O. A. Starykh, Rep. Prog. Phys. 78, 052502 (2015).
  * Yamamoto  _et al._ (2014) D. Yamamoto, G. Marmorini, and I. Danshita, Phys. Rev. Lett. 112, 127203 (2014).
  * Kawamura and Miyashita (1985) H. Kawamura and S. Miyashita, J. Phys. Soc. Jpn. 54, 4530 (1985).
  * Chubukov and Golosov (1991) A. V. Chubukov and D. I. Golosov, J. Phys.: Condens. Matter 3, 69 (1991).
  * Seabra  _et al._ (2011) L. Seabra, T. Momoi, P. Sindzingre, and N. Shannon, Phys. Rev. B 84, 214418 (2011).
  * Inami  _et al._ (1996) T. Inami, Y. Ajiro, and T. Goto, J. Phys. Soc. Jpn. 65, 2374 (1996).
  * Smirnov  _et al._ (2007) A. I. Smirnov, H. Yashiro, S. Kimura, M. Hagiwara, Y. Narumi, K. Kindo, A. Kikkawa, K. Katsumata, A. Y. Shapiro, and L. N. Demianets, Phys. Rev. B 75, 134412 (2007).
  * Sakai and Nakano (2011) T. Sakai and H. Nakano, Phys. Rev. B 83, 100405 (2011).
  * Coletta  _et al._ (2016) T. Coletta, T. A. Tóth, K. Penc, and F. Mila, Phys. Rev. B 94, 075136 (2016).
  * Henley (1989) C. L. Henley, Phys. Rev. Lett. 62, 2056 (1989).
  * Tanaka  _et al._ (2002) H. Tanaka, T. Ono, H. A. Katori, H. Mitamura, F. Ishikawa, and T. Goto, Prog. Theor. Phys. Suppl. 145, 101 (2002).
  * Ono  _et al._ (2003) T. Ono, H. Tanaka, H. A. Katori, F. Ishikawa, H. Mitamura, and T. Goto, Phys. Rev. B 67, 104431 (2003).
  * Ono  _et al._ (2004) T. Ono, H. Tanaka, O. Kolomiyets, H. Mitamura, T. Goto, K. Nakajima, A. Oosawa, Y. Koike, K. Kakurai, J. Klenke, P. Smeibidle, and M. Meißner, J. Phys.: Condens. Matter 16, S773 (2004).
  * Shirata  _et al._ (2012) Y. Shirata, H. Tanaka, A. Matsuo, and K. Kindo, Phys. Rev. Lett. 108, 057205 (2012).
  * Susuki  _et al._ (2013) T. Susuki, N. Kurita, T. Tanaka, H. Nojiri, A. Matsuo, K. Kindo, and H. Tanaka, Phys. Rev. Lett. 110, 267201 (2013).
  * Achiwa (1969) N. Achiwa, J. Phys. Soc. Jpn. 27, 561 (1969).
  * Nojiri  _et al._ (1988) H. Nojiri, Y. Tokunaga, and M. Motokawa, J. Phys. Colloques 49, C8 (1988).
  * Nikuni and Shiba (1993) T. Nikuni and H. Shiba, J. Phys. Soc. Jpn. 62, 3268 (1993).
  * Wells (1947) A. F. Wells, J. Chem. Soc. , 1662 (1947).
  * Schlueter  _et al._ (1966) A. W. Schlueter, R. A. Jacobson, and R. E. Rundle, Inorg. Chem. 5, 277 (1966).
  * Tazuke  _et al._ (1981) Y. Tazuke, H. Tanaka, K. Iio, and K. Nagata, J. Phys. Soc. Jpn. 50, 3919 (1981).
  * Tanaka  _et al._ (1992) H. Tanaka, U. Schotte, and K. D. Schotte, J. Phys. Soc. Jpn. 61, 1344 (1992).
  * Mekata  _et al._ (1995) M. Mekata, Y. Ajiro, T. Sugino, A. Oohara, K. Ohara, S. Yasuda, Y. Oohara, and H. Yoshizawa, J. Magn. Magn. Matter 140, 1987 (1995).
  * Adachi  _et al._ (1980) K. Adachi, N. Achiwa, and M. Mekata, J. Phys. Soc. Jpn. 49, 545 (1980).
  * Weber  _et al._ (1996) H. B. Weber, T. Werner, J. Wosnitza, H. v. Löhneysen, and U. Schotte, Phys. Rev. B 54, 15924 (1996).
  * Kousaka  _et al._ (2009) Y. Kousaka, H. Ohsumi, T. Komesu, T. Arima, M. Takata, S. Sakai, M. Akita, K. Inoue, T. Yokobori, Y. Nakao, E. Kaya, and J. Akimitsu, J. Phys. Soc. Jpn. 78, 123601 (2009).
  * Mino  _et al._ (1994) M. Mino, K. Ubukata, T. Bokui, M. Arai, H. Tanaka, and M. Motokawa, Physica B: Condens. Matter 201, 213 (1994).
  * Zvyagin  _et al._ (2019) S. A. Zvyagin, D. Graf, T. Sakurai, S. Kimura, H. Nojiri, J. Wosnitza, H. Ohta, T. Ono, and H. Tanaka, Nat. Commun. 10, 1064 (2019).
  * Sera  _et al._ (2017) A. Sera, Y. Kousaka, J. Akimitsu, M. Sera, and K. Inoue, Phys. Rev. B 96, 014419 (2017).
  * Hosoi  _et al._ (2018) M. Hosoi, H. Matsuura, and M. Ogata, J. Phys. Soc. Jpn. 87, 075001 (2018).
  * Yamamoto  _et al._ (2021) D. Yamamoto, T. Sakurai, R. Okuto, S. Okubo, H. Ohta, H. Tanaka, and Y. Uwatoko, Nat. Commun. 12, 4263 (2021).
  * Altarawneh  _et al._ (2009) M. M. Altarawneh, C. H. Mielke, and J. S. Brooks, Rev. Sci. Instrum. 80, 066104 (2009).
  * Ghannadzadeh  _et al._ (2011) S. Ghannadzadeh, M. Coak, I. Franke, P. A. Goddard, J. Singleton, and J. L. Manson, Rev. Sci. Instrum. 82, 113902 (2011).
  * Smith and Chu (1967) T. F. Smith and C. W. Chu, Phys. Rev. 159, 353 (1967).
  * Christy  _et al._ (1994) A. G. Christy, R. J. Angel, J. Haines, and S. M. Clark, J. Phys.: Condens. Matter 6, 3125 (1994).
  * Katsumata  _et al._ (1983) K. Katsumata, K. Sugiyama, and M. Date, J. Phys. Soc. Jpn. 52, 3312 (1983).