# FashionSD-X: Multimodal Fashion Garment Synthesis using Latent Diffusion

Abhishek Kumar Singh<EMAIL_ADDRESS>
Queen Mary University of London

###### Abstract

The fashion industry, in its constant quest for innovation, needs the
integration of generative AI to help in the design process. We propose a novel
pipeline to help fashion designers materialize their ideas, using latent
diffusion models. In this paper, we introduce a pipeline using ControlNet and
LoRA Fine-tuning that can generate high-quality images conditioned on
multimodal inputs like text and sketch. We used state-of-the-art fashion
virtual try-on datasets, Multimodal Dress Code, and VITON-HD and extended them
further to incorporate sketches. Our findings upon qualitative and
quantitative evaluation based on, FID, CLIP Score, and KID showcase our
model’s superior performance compared to the vanilla stable diffusion model,
signifying the potential for diffusion models to be involved in the fashion
design process. This study paves the way for a more interactive, personalized,
and technologically advanced future in fashion design and representation.

###### Index Terms:

Stable Diffusion, computer vision, Fashion Technology

Figure 1: Results of our proposed pipeline FashionSD-X, where the results are
guided by a given input text prompt and sketch.

##  I Introduction

The field of generative AI has experienced a considerable upswing of
innovation and success over the past decade, introducing new frontiers in
various domains, like art, design, and fashion. These tremendous developments
are possible thanks to the emergence of powerful models capable of
synthesizing high-quality images from multimodal input. Researchers have
started focusing on utilizing these models in the fashion domain, especially
working on similar garment recommendation systems and virtual try-on clothes
and accessories. But in this context, there haven’t been many significant
works to help fashion designers generate clothing garments given a text
description and a sketch and texture of the desired garment.

Diffusion models are in the spotlight due to their exceptional generative
capabilities and the ability to cater to several downstream tasks in multiple
domains. They define a Markov chain of diffusion steps that progressively add
random noise to data and then learn to reverse this process, constructing
desired data samples from the noise. Several notable diffusion-based
generative models underlie these principles, including diffusion probabilistic
models [3], noise-conditioned score network [4], and denoising diffusion
probabilistic models [5]. A new kind of diffusion model known as the Latent
Diffusion Models (LDMs) has emerged recently and has instantly become one of
the leading diffusion models because of its low computational cost. The
LDMs[6] work in the latent space of the pre-trained autoencoder, finding an
optimal trade-off between computational load and image quality. These models
introduce a latent variable into the generative process, which provides a more
expressive and flexible model of the data distribution. The latent variable
allows the model to capture complex, multi-modal distributions and can be
manipulated to control the generation process. Given their adaptability and
outstanding generation capabilities, the use of diffusion models in the
fashion domain is still only being remotely explored.

In this work, we explore and define a new task of Novel Fashion Garment
generation using Stable Diffusion guided and conditioned by multimodal inputs
like text prompts, sketches, and texture. For this, we implement a novel
pipeline combining Stable Diffusion Low-Rank Adaption (LoRA) and ControlNet.
We created two separate pipelines for this task, The first pipeline Fig.2
takes the stable diffusion model from Hugging Face and fine-tunes it on our
dataset using LoRA method. In the second pipeline, we first train a ControlNet
on sketches as an added condition and then use our first pipeline with the
ControlNet, to give us a stable diffusion model capable of generating high-
quality fashion garments using just text and a sketch. This will help in
manifesting the concept of a dress a fashion designer has in their mind in the
form of an image which they can then later use as inspiration. The text prompt
provides high-level semantic information about the desired garment, such as
its type, color, and style. The sketch provides a detailed outline of the
garment’s shape and structure. By combining these two input forms, we aim to
create a model that can generate detailed, realistic images of garments that
closely match the user’s specifications. This work builds upon previous
research in text-to-image synthesis, sketch-based image generation, and
ControlNet and extends it to the specific domain of fashion garment
generation. We believe that this approach has the potential to significantly
improve the quality and realism of generated garment images, and could have
wide-ranging applications in the fashion industry, like automated design.

For this task, we used two existing virtual Try-On Datasets, Dress Code [7]
and VITON-HD [8] and we also made use of the Multimodal Dress Code and
Multimodal VITON-HD which were extended by Morelli et al[1, 2] to add
conditioning factors like fine grain textual descriptions of the garments,
model sketch, model pose, segmentation maps, etc. We further extended these
datasets to extract and add the sketches of the garments in the dataset. We
then converted this into a hugging face dataset format. We also propose a new
evaluation metric that gives us a measure of how structurally and visually
similar the generated dresses are to the given sketches.

Contributions: To sum up, our contributions are:

  * •

We implement a novel pipeline of a Stable Diffusion LoRA with ControlNet
trained on the virtual try-on benchmark datasets like Dress Code and VITON-HD,
which to our best knowledge has never been implemented before in the fashion
domain.

  * •

We introduce a novel fashion-centric generative model to help fashion
designers based on latent diffusion models conditioned on multimodal prompts
like texts, sketches/textures. We extended the datasets further by extracting
sketches using adaptive thresholding and then creating a new hugging face
dataset.

  * •

A novel evaluation metric to indicate how much reliance the input sketches
have on the final model output and how structurally similar the output is to
the input sketch.

##  II Related Work

Text-to-Image Synthesis:.
In the context of Text-to-Image synthesis, GANs [9, 10, 11, 12] have been a
popular choice for image synthesis due to their ability to generate realistic
images. However, despite their success, GANs have limitations, including mode
collapse and training instability, which have led researchers to explore
alternative generative models. In most cases now, Diffusion models [13, 14, 6]
are considered to be the state of the art when talking of text-guided image
generation. In the fashion domain, only a handful of works have been done to
this date, earlier works used GANs. One very popular approach is FICE: Text-
Conditioned Fashion Image-Editing With Guided GAN Inversion by Pernus et al
[16] incorporates a latent code regularization technique to enhance the GAN
inversion process. This technique leverages CLIP textual embeddings [17] to
guide the image editing procedure. In another study, Zhu et al.[15] introduced
a GAN-based method that utilizes both textual descriptions and semantic
layouts to generate the final output. On the other hand, Jiang et al. [18] put
forward an architecture that synthesizes full-body images by mapping textual
descriptions of clothing items to one-hot vectors. However, this approach
restricts the expressive capacity of the conditioning signal.

Diffusion Models Diffusion models, which were initially introduced by Sohl-
Dickstein et al. (2015)[3] are considered to be fundamental in the line of
research in the field of image synthesis [21, 5, 22, 20]. These models work by
transforming a simple prior distribution into a complex data distribution
through a sequence of small diffusion steps, which are typically modeled as
Gaussian noise. The Denoising Diffusion Probabilistic Models (DDPMs) [5]
introduced a significant improvement over the original diffusion models. DDPMs
utilize a denoising process where the noise added at each step is gradually
removed, resulting in a sample from the data distribution. This approach has
been shown to generate high-quality samples and applied to various domains,
including image synthesis. Nichol et al. [20] built further on the work in [5]
by learning the variance parameter of the reverse diffusion process and
producing output with fewer forward passes while maintaining sample quality.

While most of the works until now were focused on working in the pixel space,
Rombach et al. [6] proposed a variant working in the latent space of a pre-
trained autoencoder, enabling higher computational efficiency. This allowed
for the models to become more flexible and expressive in modeling data
distribution. Song, Meng, and Ermon (2021) introduced the Denoising Diffusion
Implicit Models (DDIM)[23], a new sampling algorithm that significantly
improves the quality of generated samples. DDIM uses a denoising autoencoder
to model the data distribution and employs an implicit modeling approach to
generate new samples.

Figure 2: Overview of the Architecture of the proposed FashionSDX , a fashion-
centric fine-tuned stable diffusion model

Schölkopf et al. (2021) introduced the Stable Diffusion Models[6], which
further improve the stability and performance of diffusion models. Diffusion
models’ impact has quickly become disruptive in a variety of tasks, including
text-to-image synthesis [14], image-to-image translation, image editing, and
inpainting [1, 2]. In the domain of fashion domain, works have majorly been
done on virtual try-on tasks, Bhunia et al. [24] addressed the problem of
pose-guided human generation by creating a texture diffusion block based on
cross-attention and conditioned on multi-scale texture patterns from the
encoded source image. Two notable works, that contributed the most are
Multimodal Garment Designer and LaDI-VTON by Davide Morelli et al[1, 2]. They
developed state-of-the-art virtual try-on tasks using a Stable Diffusion
textual inversion-enhanced model. In [1, 2] Morelli et al. used a stable
diffusion inpainting pipeline with a modified architecture to factor multiple
modalities like sketch, human pose, etc.

Multimodal Image Generation with diffusion models.
In multimodal image generation, several works have leveraged diffusion models
to incorporate various conditioning signals, such as strokes and sketches, to
guide the image synthesis process. The work by Chenlin Meng et al., titled
”SDEdit: Guided image synthesis and editing with stochastic differential
equations” [25], presents an approach that adds noise to a stroke-based input
and applies the reverse stochastic differential equation to synthesize images.
This method does not require additional training, making it a flexible and
efficient solution for guided image synthesis.

Other recent works have explored the use of sketches as additional
conditioning signals. For instance, the work by Shin-I Cheng et al.[26],
proposes to concatenate the sketches with the model input. This approach
allows the model to directly utilize the sketch information during the image
synthesis process. Similarly, the work by Andrey Voynov et al., titled
”Sketch-Guided Text-to-Image Diffusion Models” [27], proposes to train an MLP-
based edge predictor to map latent features to spatial maps. This approach
allows the model to generate images that closely follow the provided sketches,
resulting in more accurate and realistic images. These works demonstrate the
versatility and potential of diffusion models in the field of multimodal image
generation. They provide various methods to incorporate different conditioning
signals, such as strokes and sketches, to guide the image synthesis process,
resulting in high-quality and realistic images. In contrast to all the works
mentioned above, we propose a fashion-centric diffusion model that generates
high-quality fashion garments by directly exploiting the semantic descriptions
of the garment and garment sketches.

##  III Methodology

In this section, we propose a novel method for the generation of fashion
garments with images conditioned on text and a sketch of the garment.
Specifically, given the garment image $I\in\mathbb{R}^{W\times H\times 3}$, a
textual description Y of a garment, and a sketch $S\in\mathbb{R}^{W\times
H\times 3}$, we want to generate a new image $I^{\prime}\in\mathbb{R}^{W\times
H\times 3}$ that generates the picture of the garment in accordance to the
given text and sketch inputs. We propose two separate pipelines for this task,
1) LoRA Fine-tune model: Fashion-SDX presented in Fig. 2 and 2) LoRA +
ControlNet Fashion- SDX presented in Fig.3. The first pipeline is a text-to-
fashion image model, which is capable of generating diverse and aesthetic
results conditioned just on text prompt. In our second pipeline, we
incorporated sketches as an added conditioning. We trained our own ControlNet
conditioned on sketches from our extended dataset, we added this trained
ControlNet to our fine-tuned Fashion-SDX. Now in the following section, we
will discuss all the preliminaries that are essential for our study.

###  III-A Preliminaries

Stable Diffusion:
Stable Diffusion is a latent diffusion model, which consists of an Autoencoder
$\mathcal{A}$ with an Encoder $\mathcal{E}$, Decoder $\mathcal{D}$. It also
has a text-time conditioning denoising U-Net $\mathcal{E_{\theta}}$, a CLIP
text-encoder $\mathcal{T}_{E}$ which takes the input text prompt and converts
and maps a sequence of input tokens to a sequence of latent text-embeddings.
The Image $I\in\mathbb{R}^{3*W*H}$ is compressed by the Encoder $\mathcal{E}$
into a latent space of lower dimension $\in\mathbb{R}^{4*w*h}$, where w = W /
8 and h = H / 8. The Decoder $\mathcal{D}$ inverts and converts the latent
variables back to pixel space. The training of the denoising network U-Net
$\mathcal{E_{\theta}}$ is performed minimizing the loss function defined as
follows:

$\mathrm{L}=\mathbb{E}_{\mathcal{E}(I),Y,\epsilon\sim\mathcal{N}(0,1),t}[\,\lvert\lvert\mathcal{E}-\mathcal{E_{\theta}}({\gamma},{\psi})\rvert\rvert_{2}^{2}\,],$
\- (1)

Where $t$ is diffusion timestep and $\gamma=z_{t}$, $z_{t}$ is the fully
noised image or encoded image $\mathcal{E}(I)$ which is produced by gradually
adding Gaussian Noise $\mathcal{N}(0,1)$ and ${\psi}=[t;\mathcal{T}_{E}(Y)]$,
where $\mathcal{T}_{E}(Y)$ is the encoded text prompt by the CLIP Text
Encoder. In the above equation, ${\gamma}$ is the spatial input of the
convolutional network (U-Net) and ${\psi}=[t;\mathcal{T}_{E}(Y)]$ is the
attention conditioning input.
We use the Standard Stable Diffusion Pipeline, for our task with the aim to
generate an Image I of the garment alone without any artifacts or human model
wearing the dress. To do this we fine-tune the Stable Diffusion model on our
dataset to learn the style of the output we expect.
Low Rank Adaptation:
Low-Rank Adaptation(LoRA) [28] is a parameter-efficient training technique
initially used for fine-tuning Large Language Models (LLMs). LoRA reduces the
number of trainable parameters by learning the pairs of rank-decomposition
matrix while freezing the original weights. These pairs of rank decomposition
matrices are called update matrices and are added to existing weights. This
technique can also be used in other Large models like Stable Diffusion.

Figure 3: Overview of our Pipeline for Integrating sketches to guide the
diffusion process of FashionSDX model Figure 4: Basic Architecture of
ControlNet Network

$~{}\mathrm{W^{\prime}}=\mathrm{W}+\Delta\mathrm{W}\hfill-(2)~{}$
In the LoRA fine-tuning process we are only training the $\Delta\mathrm{W}$ in
eq. (2). Where we can further decompose $\Delta\mathrm{W}$ into low rank
matrices $\Delta\mathrm{W}=AB^{T}$ where
$A\in\mathit{R}^{\mathtt{n}\mathtt{x}\mathtt{d}}$ and
$B\in\mathit{R}^{\mathtt{m}\mathtt{x}\mathtt{d}}$ where $d<<n$. So we can just
fine-tune A and B matrices instead of W. The resultant trained weights are
much smaller in size only $\sim(2-200Mb)$. In the case of stable diffusion
models, only Q,K,V i.e the attention layers are enough to fine-tune the entire
model. As can be seen from Figure 2.
ControlNet:
ControlNet[29] is a neural network designed to control pretrained large
diffusion models and support any additional input conditioning. The controlnet
makes a clone or copies the weights of the diffusion model into a ”Traininable
Copy” and a ”Locked Copy”, here only the trainable copy is made to learn the
task-specific conditioning by training on the task-specified dataset. The
trainable and locked copies of the network are linked by Zero-convolutions.
The training is fast and comparable to time fine-tuning time because zero
convolutions do not add any new noise to the deep features. The ControlNet
architecture is shown in Figure. 4.

CLIP:
CLIP is a transformer-based vision-language model[17] which consists of a
vision transformer for building image embeddings and a text transformer for
generating text embedding. It aligns the text and visual inputs into a common
embedding space.

The visual encoder and text encoder extract these feature representations
respectively, $\mathit{V_{E}(I)}\in\mathbb{R}^{d}$ has a dimensionality of ’d’
for an input Image $\mathit{I}$, the text encoder
$\mathit{T_{E}(E_{L}(Y))}\in\mathbb{R}^{d}$ has a dimensionality of ’d’, for a
text input of Y. In the above, $\mathit{E_{L}}$ is the embedding lookup that
maps every tokenized word in $\mathit{Y}$ to the token embedding space
$\mathcal{W}$, and ’d’ is the size of CLIP embedding space.

###  III-B Fashion-centric Image Synthesis

In our task, we aim to generate accurate results from text prompts that are
not just realistic and coherent to the text prompt but also in line with what
the designers have in mind. To tackle this task of realistic fashion garment
synthesis, we took the stable diffusion pipeline by hugging face’s diffusers
library and fine-tuned the model to learn the style of the in-shop garments.
In our pipeline, we start from a clothing garment in our dataset C, which is
extracted from the CLIP visual encoder, the text prompt is then projected to
the latent space using the CLIP text encoder. We then gradually add noise to
the image latent vectors for some given timestep and predict the added noise
to the image latent. A reconstruction loss is calculated between the predicted
noise and the original noise. Finally, the diffusion model parameters of the
U-net are optimized with respect to this loss using gradient descent. The pre-
trained text and image encoders are kept frozen. The LoRA finetuning
significantly helps in training the model fast and helps guide the generation
process to properly resemble the style of in-shop garments in our training
dataset.
Sketch Driven Generation:
It is very hard to fully describe the clothing garment you want just based on
a textual description due to many factors like ambiguity of natural language
and other complexities such as technical terminology. The text might fail to
convey certain intricate details such as the size, shape, and texture of the
desired garment. Therefore, we proposed the idea of incorporating sketches and
texture information of the dresses with our model to further facilitate the
fashion generation process. To do this, we incorporated ControlNet into our
pipeline.

###  III-C Training

We followed the script [30] given by huggingface to finetune and train a
stable diffusion model, in our case we chose to use the Low-rank adaptation.
The proposed denoising U-net is trained to predict the noise being
stochastically added to z according to Equation given below:
$\mathrm{L}=\mathbb{E}_{\mathcal{E}(I),Y,\epsilon\sim\mathcal{N}(0,1),t}[\,\lvert\lvert\mathcal{E}-\mathcal{E_{\theta}}({\gamma},{\psi})\rvert\rvert_{2}^{2}\,],$

For training on ControlNet, first, we need to decide which stable diffusion
model we need to train and which part of the Stable Diffusion we want to
train. Usually, all weights inside the ControlNet are copied from the stable
diffusion model we select and no layer is trained from scratch, just like
fine-tuning. We can also decide which part of the ControlNet and Stable
diffusion we want to train. From Fig. 4 we can see that ControlNet has a
trainable copy and a locked copy, we can decide to unlock the bottom layers of
our locked neural network, this will train a few layers of the original stable
diffusion model. We decided to use this functionality for our specific task
since our dataset is diverse enough and we think this might help in domain-
specific tasks such as Fashion Generation. We also need to decide on a
condition to train ControlNet on, We chose to train the network on two
separate conditions one sketch and sketch + texture. Our pipeline for
generating images conditioned on sketch and text is shown in Figure.3.

##  IV Experimental Setup

###  IV-A Dataset

Datasets:

TABLE I: Table 1. Comparison of the publically available datasets for fashion

Dataset No. of Images No. of Unique Texts No. of Sketches DeepFashion 44,096
10,253 - Be Your Own Prada 78,979 3,972 - VITON-HD (Ours) 13,679 5,143 13,679
DressCode (Ours) 53,792 25,596 53,596

There are currently very limited datasets available for fashion generation
tasks, most of the datasets that are available lack the conditioning
information like text descriptions or sketches. For our task which requires us
to finetune a latent diffusion model to generate fashion-centric images using
multimodal conditions like text descriptions and sketches. That’s why we opted
to go for the famous Fashion Try-On Datasets which were further extended by
Morelli et al [1], known as Dress Code Multimodal and VITON-HD Multimodal, the
authors incorporated textual sentences and morphed garment sketches for the
virtual try-on tasks. Both these datasets include images of resolution 1024 x
768, each with an image pair of a garment and the corresponding model wearing
the garment. Dress Code Multimodal contains 53,792 images of garments split
into 3 classes, namely upper body, lower body, and dresses. This dataset
contained two prompt files:-
Fine-Grained Text Prompts: This prompts file contains manually annotated
prompts for 26,400 garments out of 53,792 garments in the dataset.
Coarse-Grained Texts Prompts: For the remaining garments were automatically
annotated by a fine-tuned OpenCLIP ViT-B32 model pre-trained on LAION5B by
Morelli et al.[1].
We converted both of these text files into two JSON Lines files, and randomly
selected and merged the above-mentioned annotations in a single file with
mapping to the image paths in the train and test folders. We then split the
JSON files into test metadata files and train metadata files. We then use
huggingface’s Image Folder library to create a dataset in a column format,
where the first column has Images loaded in PIL Format and the second column
has their respective text prompt in a string format. A similar, strategy was
used to annotate all the garments of the VITON-HD dataset. For our task, in
both of the datasets, we only take the in-shop garments and their
corresponding textual descriptions from the multimodal datasets and we remove
all other multimodal information such as human pose, human key points, and
morphed garment sketches. We then extracted the sketch information of each
garment using our custom framework.
Incorporating Sketches:
Sketch augmentation to our pipeline is essential to provide the users the
added functionality to have control over the generated outputs according to
their expectations because there might be cases where discrepancies occur when
relying solely on text prompts. The resulting pipeline would allow the users
to procure outputs that are more coherent, comprehensive, and accurate
representations of their expectations of the garments, which in most cases
would lead to a better quality of the generated results. Although the
multimodal datasets we chose for our tasks have sketches they are not sketches
of the in-shop garments but sketches of the garments morphed on the human
model’s body. Instead, we chose to extract sketches of all the garments in the
two datasets using the Canny Edge detection algorithm and our own Adaptive
Thresholding. We had to create a mapping of each sketch to its respective
image. Due to memory issues, we had to extract sketches on the fly for every
single image and create the mapping at the same time instead of extracting all
sketches first. After mapping each sketch to its reference image, we created
two new datasets which have an added column with the sketch, this will be our
added conditioning. There are very limited fashion datasets that are publicly
available and the ones that are available don’t have in-shop garment images
which is crucial in our task. Out of these, there are only two such datasets
that have corresponding text captions associated with each garment. These
datasets are an extension of DeepFashion Dataset[31]. We show in Table I. the
distribution of the words and texts in these datasets compared to the ones we
used. We can see that the dataset we have selected is much more verbose than
the competitors and also has sketches associated with each garment.

###  IV-B Training and Inference

Figure 5: Comparison of Stable Diffusion v/s our trained model for the same
prompt.

All the models were trained on the same system on a single Quadro RTX 6000 (24
Gb VRAM), Intel CPU XEON Gold 5222 with RAM of 24 GB. We trained two models
via Low-rank adaptation on Dress Code and VITON-HD datasets, for 181k Steps
(15 Epochs) and 58k Steps (10 Epochs) for the Dress Code and VITON-HD datasets
respectively, at the batch size of 4 and batch size of 2 respectively. The
learning rate in the above two models was ${10}^{-4}$ with a linear learning
rate warmup for the first 500 steps. We make use of the accelerate library
[32] and script provided by huggingface to speed up the training process,
where we opted for mixed precision training ’fp16’ due to limited computation
resources and to speed up the training process. The model selection was a
crucial process, we chose runwayml’s stable-diffusion-v1-5 training on the
”laion-aesthetics v2 5+” [33] dataset. The reason why we chose this model is
that it performed really well in synthesizing realistic and aesthetic outputs
and the input resolution was $512\times 512$ with 10% text drop rate which
improved classifier-free guidance.
For training the ControlNet pipeline, we trained two models on VITON-HD, and
Dress Code on our extended datasets. For VITON-HD, we trained the model for 5
epochs (18k steps) at a batch size of 4, and for Dress Code, we trained our
model for 5 epochs (24k Steps) at a batch size of 4. The learning rate for all
ControlNet models is ${10}^{-5}$ and we used a constant learning rate
scheduler.

During Inference, we used PNDMScheduler[34] and DDIM Scheduler [23] with the
number of inference steps set to 30 and the classifier-free guidance scale
$\alpha$ set to 7.5. During, the sketch-guided pipeline inference, we used the
PNDMScheduler with a sketch conditioning scale set to 0.6. For comparison to
other baselines, we compared our results with the vanilla Stable Diffusion
model from Huggingface’s runwayml’s stable diffusion-v1-5 and compared all our
trained models to results generated from the vanilla stable diffusion.

##  V Experimental Evaluation

###  V-A Evaluation Metrics

To evaluate our results for generated fashion images on the basis of realism
and coherence to the style of our input dataset, we chose to employ the
following evaluation metrics: the Frechet Inception Distance (FID) [35], FID-
CLIP[36] which uses CLIP feature space for computing FID and the Kernel
Inception Distance (KID)[37]. We used the implementation for both these
metrics proposed by Parmar et al. in [38]. We also used Structural Similarity,
SSIM for measuring the image quality of generated outputs using sketch. We do
this by first extracting a sketch using edge detection from output results and
then comparing them using SSIM index with the input sketches.

To verify the adherence of our generated results to the text we used CLIP
Score using the implementation provided in the TorchMetrics library, using the
OpenAI clip-vit-base-patch16, and the implementation provided by HuggingFace.
Since our results could not be entirely evaluated by qualitative metrics, we
chose to do a qualitative survey using our generated results. We asked fashion
designer students studying at the London College of Fashion to provide us with
sample prompts and gave them 50 images generated by our model and Vanilla
Stable Diffusion then asked them to tell us which results they preferred when
comparing the two for the 50 sample images. Table III shows the percentage of
times the users preferred our model compared to results generated by Vanilla
Stable Diffusion.

Sketch Similarity: We also propose, a novel evaluation metric to estimate the
adherence of the generated images to the conditioning constraints in our case
the sketches. To compute the similarity we first extract the sketch from the
generated images using adaptive thresholding and then compute the Mean Squared
Error between the extracted sketch and the given sketch’s per pixel distance
between the two inputs.

$SSim(I_{S},\tilde{I_{S}})={MSE}({Sketch(I_{S}),Sketch(\tilde{I_{S}})})$

Figure 6: This Figure Displays some of our failure cases, a) First row
displays VITON-HD model’s failure to comprehend white colour in prompts. The
first image also fails to generate male clothes due to the disproportionate
distribution of male and female clothes in VITON-HD Dataset. b) The Second Row
shows how the ControlNet gave results without training, it changed the
background according to the prompt but failed to make the garments in
accordance to sketch.

###  V-B Experiments

Since we trained multiple models on our two datasets, we tried to experiment
with different hyperparameters and different training techniques. In this
section, we will discuss all the different approaches we took throughout this
project before finalizing our final models. The details of our experimentation
for the VITON-HD training are shown in our wandb profile here. We initially,
trained our first fine-tuning model without LoRA, it suffered from forgetting
and took a long time to train, due to scarce computational resources we had to
stop the training prematurely and switch to LoRA fine-tuning for our models.
On testing, we found that the model trained on VITON-HD for 151k steps overfit
easily, it had learned the style of the trainset too well. There, were
certain, patterns that kept showing up in the results, and one more prominent
and problematic issue was that most of the results even after we specifically
mentioned against in prompts were women’s wear. This could be because of two
reasons, the proportion of female garments in the VITON-HD was much much
higher than the men’s garments, and the number of samples in the VITON-HD was
not enough for us to perform fine-tuning for 151k steps, so we ended up
overtraining the model. One more interesting thing we noticed was that when we
gave the prompts for any garment with white colour in it, our VITON fine-tuned
model always gave black and white stripes, this could be the fault of the
prompt given during training. To solve, these issues we trained the models for
much fewer steps around 52k, and the results were of much better quality.

Model | Resolution |  | Dress | Code |  |  | VITON-HD |  |
---|---|---|---|---|---|---|---|---|---
|  | FID | FID-CLIP | KID | CLIP-Score | FID | FID-CLIP | KID | CLIP-Score
Stable Diffusion | 512 x 512 | 84.434 | 32.993 | 0.0522 | 32.1053 | 84.434 | 32.993 | 0.0522 | 32.1053
FashionSDX (Ours) | 512 x 512 | 23.804 | 13.344 | 0.0082 | 32.1498 | 42.657 | 15.257 | 0.0180 | 31.0188
Fashion SDX + ControlNet (Ours) | 512 x 512 | 20.512 | 16.397 | 0.0110 | 32.5812 | 34.659 | 19.453 | 0.0288 | 31.5135
TABLE II: Quantitative results of FashionSD-X trained on Dress Code and VITON-
HD and compared to results from a pre-trained Stable Diffusion Model.

When experimenting with the ControlNet pipeline, initially we attempted to add
the pre-trained model directly with our FashionSD-X model in the pipeline, The
results were quite interesting, the results given by this pipeline failed to
comprehend sketch boundaries and instead of creating a garment that adheres to
the sketch, it gave the sketch back as it is but changed the background to
match the text prompt. These findings motivated us to train our own ControlNet
model, conditioned on the sketches extracted by us in our extended dataset.
The details for training the ControlNet are shown in our wandb profile here.

TABLE III: Results of a user study conducted by us, where the users were shown
a random 50 images generated by stable diffusion and our models. The users
were asked to pick either Stable Diffusion or Our model based on the Realism
of the generated garments and coherence to the text prompts. The results here
are the percentage of times the users preferred our results compared to stable
diffusion

Models Inputs Realism Prompt Coherence Text Sketch SD SD FashionSD-X (VITON-
HD) Y N 51.45% 62.69% Y Y 64.58% 67.2% FashionSD-X(Dress Code) Y N 84.84%
82.1% Y Y 86.26% 82.66%

TABLE IV: The table compares the adherence of the outputs to the input sketch
using a new metric Sketch Similarity (the lower the value the closer the
results to the sketch) and SSIM (the closer the value to 1 the closer the
results to the input sketch).

Model Sketch Similarity SSIM Dress Code VITON-HD Dress Code VITON-HD Stable
Diffusion 4525 4421 0.70 0.67 FashionSD-X 3581 3896 0.78 0.68

Fig.6 Illustrates a few of the findings we got via experimentation, on the
VITON-HD Dataset and the ControlNet Pipeline. We learned a lot from our above-
mentioned mistakes and then fine-tuned two separate pipelines Fashion-SDX and
ControlNet + FashionSD-X trained on Dress Code and VITON-HD successfully.

##  VI Results

We test our proposed pipeline against the pre-trained stable diffusion model
using quantitative and qualitative metrics as described in the previous
section. Table II compares our models FashionSD-X and FashionSD-X + ControlNet
with the Stable Diffusion results on the basis of FID Score, FID-CLIP score,
KID, and CLIP Score. To evaluate our results, we generated outputs using our
test set’s prompts and extracted sketches using thresholding of the garments
in our test and then fed all the prompts and sketches to our two pipelines. As
can be noticed, from the results the FID Score for all the models is quite
high, this can be attributed to the background of the results being
drastically different. The results would be much closer to zero if we had
segmented the clothes and pasted them on 512 $\times$ 512 white background.
Nonetheless, Both our models outperform Stable Diffusion based on Realism -
FID, FID-CLIP, and KID Scores. The Closer the FID Score and KID scores are to
zero the less distance there is between our generated images and sample images
in our test dataset in terms of realism. The CLIP-Score of our model on many
occasions struggled to outperform Stable Diffusion, but the final results when
computed for the entire test dataset were found to be quite comparable for the
Dress Code dataset. The CLIP score is highly dependent on the model used for
the encoding process, the discrepancy found in our results even though
visually superior to Stable Diffusion can be possibly attributed to its
failure to understand the nuances of our specific fashion domain.

We notice from Table II that our models trained on VITON-HD, outperformed
Stable Diffusion results in terms of FID and KID scores but the CLIP-Score was
lower than Stable Diffusion. The results for our models trained on the Dress
Code dataset were far superior to the ones from the VITON-HD dataset. The
reason for this is possibly due to the smaller size of samples in VITON-HD
(13k images) with less diverse data than the Dress Code data (54k images). We
showed some of the failure cases where the model trained on VITON-HD failed to
generate ”White t-shirt” in fig. 6.

Figure 7: Sample Images generated on Dress Code dataset, demonstrating how our
model’s performance is superior in terms of quality of results and coherence
to prompts.

We used a new metric called Sketch Similarity to compare how close the outputs
are to input sketches, the results in Table IV show the Sketch Similarity
which is the MSE between the input sketches and the sketches extracted from
the output dresses using Canny Edge Detection. We observe that our ControlNet
pipeline with FashionSD-X performs much better than Stable Diffusion on both
datasets in both metrics Sketch Similarity and Structural Similarity Index. We
observed from the stable diffusion’s results conditioned on sketch that most
of the time the results bleed outside of the sketch boundary, and there’s a
morphed human present in the results. Fig 7 showcases the sample output for
the same sketch and prompts how Stable Diffusion’s outputs bleed outside the
sketch boundary.

For the qualitative results, highlighted by Fig 1, Fig 5, Fig 7 and Table IV
we can observe based on the generated results that the quality of fashion
garments generated by our model is far superior to Stable Diffusion in terms
of realism and quality. We also conducted a mini user-study, to validate our
results in accordance with professional and human judgment. We collected
responses from 10 Fashion Design students at the London College of Fashion and
asked them to select their preference between a pair of images one generated
by our model and the other by stable diffusion, we did this for over 50
randomly sampled images from our generated images. The results are presented
in Table II which shows the percentage of times our model was preferred by the
users in the study, We observe that our model trained on Dress Code performed
phenomenally conditioned on both sketch and text, the users preferred our
model’s output over stable diffusion in almost all cases. We also noticed that
the results generated by our model on the VITON-HD dataset were quite inferior
compared to Dress Code. The users preferred stable diffusion’s output almost
48% times over our model trained on VITON-HD.

We experimented with batch size and the number of training steps during the
training of the ControlNet part, We trained three different models a) Batch
Size = 4, No. of Training steps = 25k, b) Batch Size = 4, No. of Training
Steps = 18k and c) Batch Size = 8 , Steps = 12k. We also performed validation
on the two models to see their performance qualitatively. The model c) had the
worst performance when compared to the other two. The qualitative results for
this can be seen in Fig. 8

Figure 8: Abalation results for ControlNet Training.

The results were the best from the model that was trained for 25k steps, the
model also had a high structural similarity index when compared to the other
two controlnet models.

During inference, we experimented with a number of Noise Schedulers, When
experimenting with DDIM Noise Scheduler we noticed something rather
interesting this scheduler was always giving images that kept getting flagged
as NSFW images, and a black image was displayed, even though when we disabled
the safety checker the results the results came out as black images. This was
not experienced, with any other noise scheduler we experimented on. This is
something that requires further work.

##  VII Conclusion

This research explores the multifaceted landscape of diffusion models and
their application to fashion design. We provide an exhaustive investigation of
diffusion model’s potential, challenges, and contributions to fashion. We
propose a novel pipeline, for generating fashion garments conditioned on
sketch and text. We delve into the interplay between visual and textual cues,
different datasets, training strategies, and evaluation metrics. The
exploration unfolds a comprehensive understanding of diffusion models, not
only enhancing the theoretical understanding of generative models but also
bridging the gap between artificial intelligence and creative design in the
dynamic fashion industry. The results showcase one of the most significant
works in this field in doing a fashion designer’s job and helping them in the
designing of innovative and creative garments. Our work stands as a testament
to the flexibility and robustness of diffusion models, positioning them as
vital tools in modern design and artificial intelligence research.

## References

  * [1] Baldrati, Alberto, et al. “Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editing.” arXiv.Org, 23 Aug. 2023, arxiv.org/abs/2304.02051.
  * [2] Morelli, D. et al. (2023) Ladi-VTON: Latent diffusion textual-inversion enhanced virtual try-on, arXiv.org. Available at: https://arxiv.org/abs/2305.13501 (Accessed: 25 August 2023).
  * [3] Sohl-Dickstein, J. et al. (2015) Deep unsupervised learning using nonequilibrium thermodynamics, arXiv.org. Available at: https://arxiv.org/abs/1503.03585 (Accessed: 25 August 2023).
  * [4] Song, Y. and Ermon, S. (2020) Generative modeling by estimating gradients of the data distribution, arXiv.org. Available at: https://arxiv.org/abs/1907.05600 (Accessed: 25 August 2023).
  * [5] Ho, J., Jain, A. and Abbeel, P. (2020) Denoising Diffusion Probabilistic models, arXiv.org. Available at: https://arxiv.org/abs/2006.11239 (Accessed: 25 August 2023).
  * [6] Rombach, R. et al. (2022) High-resolution image synthesis with Latent Diffusion Models, arXiv.org. Available at: https://arxiv.org/abs/2112.10752 (Accessed: 25 August 2023).
  * [7] Morelli, D. et al. (2022) Dress code: High-resolution multi-category virtual try-on, arXiv.org. Available at: https://arxiv.org/abs/2204.08532 (Accessed: 25 August 2023).
  * [8] Choi, S. et al. (2021) Viton-HD: High-resolution virtual try-on via misalignment-aware normalization, arXiv.org. Available at: https://arxiv.org/abs/2103.16874 (Accessed: 25 August 2023).
  * [9] Tao, M. et al. (2022) DF-Gan: A simple and effective baseline for text-to-image synthesis, arXiv.org. Available at: https://arxiv.org/abs/2008.05865 (Accessed: 25 August 2023).
  * [10] Xu, T. et al. (2017) Attngan: Fine-grained text to image generation with attentional generative adversarial networks, arXiv.org. Available at: https://arxiv.org/abs/1711.10485 (Accessed: 25 August 2023).
  * [11] Zhang, H. et al. (2022) Cross-modal contrastive learning for text-to-image generation, arXiv.org. Available at: https://arxiv.org/abs/2101.04702 (Accessed: 25 August 2023).
  * [12] Zhu, M. et al. (2019) DM-Gan: Dynamic memory generative adversarial networks for text-to-image synthesis, arXiv.org. Available at: https://arxiv.org/abs/1904.01310 (Accessed: 25 August 2023).
  * [13] Nichol, A. et al. (2022) Glide: Towards photorealistic image generation and editing with text-guided Diffusion Models, arXiv.org. Available at: https://arxiv.org/abs/2112.10741 (Accessed: 25 August 2023).
  * [14] Ramesh, A. et al. (2022) Hierarchical text-conditional image generation with clip latents, arXiv.org. Available at: https://arxiv.org/abs/2204.06125 (Accessed: 25 August 2023).
  * [15] Zhu, S. et al. (2017) Be your own prada: Fashion synthesis with structural coherence, arXiv.org. Available at: https://arxiv.org/abs/1710.07346 (Accessed: 25 August 2023).
  * [16] Pernuš, M. et al. (2023) FICE: Text-conditioned fashion image editing with guided Gan Inversion, arXiv.org. Available at: https://arxiv.org/abs/2301.02110 (Accessed: 25 August 2023).
  * [17] Radford, A. et al. (2021) Learning transferable visual models from Natural Language Supervision, arXiv.org. Available at: https://arxiv.org/abs/2103.00020 (Accessed: 25 August 2023).
  * [18] Jiang, Y. et al. (2022) Text2human: Text-driven controllable human image generation, arXiv.org. Available at: https://arxiv.org/abs/2205.15996 (Accessed: 25 August 2023).
  * [19] He, K. et al. (2015) Delving deep into rectifiers: Surpassing human-level performance on ImageNet Classification, arXiv.org. Available at: https://arxiv.org/abs/1502.01852 (Accessed: 25 August 2023).
  * [20] Nichol, A.Q. and Dhariwal, P. (2021) Improved denoising diffusion probabilistic models, PMLR. Available at: https://proceedings.mlr.press/v139/nichol21a.html (Accessed: 25 August 2023).
  * [21] Dhariwal, P. and Nichol, A. (2021) Diffusion models beat gans on image synthesis, arXiv.org. Available at: https://arxiv.org/abs/2105.05233 (Accessed: 25 August 2023).
  * [22] Ho, J. and Salimans, T. (2022) Classifier-free diffusion guidance, arXiv.org. Available at: https://arxiv.org/abs/2207.12598 (Accessed: 25 August 2023).
  * [23] Song, J., Meng, C. and Ermon, S. (2022) Denoising diffusion implicit models, arXiv.org. Available at: https://arxiv.org/abs/2010.02502 (Accessed: 25 August 2023).
  * [24] Bhunia, A.K. et al. (2023) Person image synthesis via denoising diffusion model, arXiv.org. Available at: https://arxiv.org/abs/2211.12500 (Accessed: 25 August 2023).
  * [25] Meng, C. et al. (2022) SDEdit: Guided Image Synthesis and editing with stochastic differential equations, arXiv.org. Available at: https://arxiv.org/abs/2108.01073 (Accessed: 25 August 2023).
  * [26] Cheng, S.-I. et al. (2022) Adaptively-realistic image generation from stroke and sketch with Diffusion Model, arXiv.org. Available at: https://arxiv.org/abs/2208.12675 (Accessed: 25 August 2023).
  * [27] Voynov, A., Aberman, K. and Cohen-Or, D. (2022) Sketch-guided text-to-image diffusion models, arXiv.org. Available at: https://arxiv.org/abs/2211.13752 (Accessed: 25 August 2023).
  * [28] Hu, E.J. et al. (2021) Lora: Low-rank adaptation of large language models, arXiv.org. Available at: https://arxiv.org/abs/2106.09685 (Accessed: 25 August 2023).
  * [29] Zhang, L. and Agrawala, M. (2023) Adding conditional control to text-to-image diffusion models, arXiv.org. Available at: https://arxiv.org/abs/2302.05543 (Accessed: 25 August 2023).
  * [30] Stable diffusion text-to-image fine-tuning (no date) Stable Diffusion text-to-image fine-tuning. Available at: https://huggingface.co/docs/diffusers/v0.13.0/en/training/text2image (Accessed: 25 August 2023).
  * [31] Ge, Y. et al. (2019) Deepfashion2: A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images, arXiv.org. Available at: https://arxiv.org/abs/1901.07973 (Accessed: 25 August 2023).
  * [32] Accelerate (no date) Hugging Face – The AI community building the future. Available at: https://huggingface.co/docs/accelerate/ (Accessed: 25 August 2023).
  * [33] Runwayml/stable-diffusion-V1-5 · hugging face (no date) runwayml/stable-diffusion-v1-5 · Hugging Face. Available at: https://huggingface.co/runwayml/stable-diffusion-v1-5 (Accessed: 25 August 2023).
  * [34] Karras, T. et al. (2022) Elucidating the design space of diffusion-based Generative Models, arXiv.org. Available at: https://arxiv.org/abs/2206.00364 (Accessed: 25 August 2023).
  * [35] Heusel, M. et al. (2018) Gans trained by a two time-scale update rule converge to a local Nash equilibrium, arXiv.org. Available at: https://arxiv.org/abs/1706.08500 (Accessed: 25 August 2023).
  * [36] Kynkäänniemi, T. et al. (2023) The role of ImageNet classes in Fréchet Inception Distance, arXiv.org. Available at: https://arxiv.org/abs/2203.06026 (Accessed: 25 August 2023).
  * [37] Bińkowski, M. et al. (2021) Demystifying MMD gans, arXiv.org. Available at: https://arxiv.org/abs/1801.01401 (Accessed: 25 August 2023).
  * [38] Parmar, G., Zhang, R. and Zhu, J.-Y. (2022) On aliased resizing and surprising subtleties in gan evaluation, arXiv.org. Available at: https://arxiv.org/abs/2104.11222 (Accessed: 25 August 2023).
  * [39]


# The Scientific Discovery Space for the Roman Galactic Bulge Time Domain
Survey

Jennifer C. Yee  Center for Astrophysics $|$ Harvard & Smithsonian, 60 Garden
St., Cambridge, MA 02138, USA  Andrew Gould  Max-Planck-Institute for
Astronomy, Königstuhl 17, 69117 Heidelberg, Germany  Department of Astronomy,
Ohio State University, 140 W. 18th Ave., Columbus, OH 43210, USA

##  1 State of the Field

Figure 1: Roman Galactic Bulge Time Domain Survey discovery space compared to
microlensing discoveries over time (NASA Exoplanet Archive, accessed 5/19/23).
Top, left axis (magenta): number of microlensing planets published each year.
A publication rate of 40 planets/year from the ground is assumed for future
years. Top, right axis (black): cumulative number of published microlensing
planets (solid=actual, dotted=projected). The black dashed lines indicate the
approximate times when white papers were being prepared for the 2010 and 2020
decadal surveys and the start of the Roman Galactic Bulge Time Domain Survey.
Bottom (Penny et al., 2019): squares = known planet distribution at the end of
2008 (magenta), 2018 (cyan), and now (black) compared to the Solar System
planets and simulated Roman planet yield (gray points). The blue line
indicates the expected Roman , sensitivity limit.

The microlensing planet landscape has changed dramatically since a space-based
microlensing mission was recommended by the 2010 Decadal Survey. At the end of
2008, when white papers were being prepared for the decadal, only eight
microlensing planets had been detected from ground-based surveys (Figure 1).
In that context, a space-based mission that could detect hundreds to thousands
of planets was an obvious advance. Even toward the end of 2018, when almost
100 microlensing planets had been detected, there were only a handful of non-
giant planet detections. However, today, the total number of planets has
doubled and is projected111KMTNet currently detects about 30 planets per year
but is also still dealing with a publication backlog, so we have assumed a
future publication rate of 40 planets per year. to double again prior to the
start of the Roman Galactic Bulge Time Domain Survey (RGBS). In addition,
there are now detections from the ground of several dozen planets with masses
between 1 and 10 $M_{\oplus}$, a number that should also roughly double over
the next few years. Hence, designing an optimal RGBS requires carefully
considering the new scientific discovery space opened up by Roman.

Comparing the known distribution of microlensing planets (NASA Exoplanet
Archive, accessed 5/19/23) and the simulated Roman detections from Penny et
al. (2019) in Figure 1 readily reveals two of the major strengths of RGBS
compared to what can be (and has been) done from the ground. First, RGBS is
projected to find planets in wider orbits, out to and beyond the orbits of
Uranus and Neptune. Second, RGBS should find planets as small as or smaller
than Mars. In addition, RGBS will be better than ground-based surveys at
characterizing the smallest free-floating planets (Gould et al., 2023b). All
of these discovery spaces share the characteristics of having a low-cross
section for detection, short duration signals, and potentially lower amplitude
signals, and so lead to similar considerations for enhancing their detection
rates.

In this white paper, we will begin by reviewing the relative sensitivity of
RGBS relative to existing ground-based surveys. We will then discuss the
challenges of each of the three discovery spaces described above and how they
affect survey design. Because these challenges are mostly rooted in light
curve degeneracies, these sections are necessarily somewhat technical. We
conclude with a high-level summary including our recommendations.

##  2 A Change of Parameters

Figure 2: Known microlensing planets (magenta) compared to simulated RGBS
planets (gray) in the microlensing parameter space: $s$, $q$. The new
discovery space opened up by Roman will be for wide-orbit ($s\gtrsim 2.5$) and
small ($q\lesssim 3\times 10^{-5}$) planets (black dotted lines).

Figure 1 shows planet mass vs. semi-major axis. However, the actual
microlensing observables are planet mass ratio and projected separation scaled
to the Einstein radius:

$q\equiv\frac{m_{\rm p}}{M_{\rm host}};\quad
s\equiv\frac{a_{\perp}}{\theta_{\rm E}D_{\rm host}},$ (1)

respectively, where

$\theta_{\rm E}=\sqrt{\kappa M_{\rm host}{\pi_{\rm
rel}}}\quad\mathrm{and}\quad\pi_{\rm rel}=\mathrm{AU}\left(\frac{1}{D_{\rm
host}}-\frac{1}{D_{\rm source}}\right).$ (2)

Therefore, Figure 1 is smeared out over the distribution of host masses and
distances as well as the distances to the sources.

To illuminate the underlying the sensitivity space for RGBS, we re-plot the
planet discoveries in Figure 2 using the parameters $s$ and $q$. For
simplicity, for planet detections, we use the default solution from the
Exoplanet Archive, but as we will discuss below, the degeneracies can be quite
severe. The basic regions of interest remain the same, but we now give them
formal definitions. For the purposes of this white paper, we define wide-orbit
planets as those with $\log s>0.4$ and small planets as those with $\log
q<-4.5$. We will define “free-floating planets” as any short timescale
($\lesssim 1\,$ day) signal without a microlensing light curve signature from
the host.222Based on the light curve alone, these are necessarily only
candidates because a host may simply be too distant to create a significant
lensing signal, although the existence or absence of a host star can be
further constrained by Roman limits on host light.

##  3 Roman vs. KMTNet

Figure 3: Roman’s primary advantage over ground-based surveys comes at low
magnification. For $A=1$, Roman has a signal-to-noise advantage of $\sim
5$–$12$ over the KMTNet, but at $A=100$ that advantage is a relatively modest
factor of $\sim 2$. Bottom: Absolute magnitudes of RGBS sources compared to
KMTNet sources at the same cumulative fraction; e.g., half of all RGBS sources
will be $M_{I}<6$ whereas half of all KMTNet sources are $M_{I}<3.25$. Top:
relative signal-to-noise ratio between Roman and KMTNet as a function of
cumulative fraction of sources and magnification. Middle: same as top, but as
a function of Roman absolute source magnitude, for which the corresponding
KMTNet source magnitude can be found from the bottom panel; e.g., for a Roman
source with $M_{I}=6$, the corresponding KMTNet source will be $M_{I}=3.25$,
and for such sources, Roman has a SNR advantage of $\sim 11.5$ over KMTNet at
$A=1$, but only $\sim 3$ at $A=100$. Figures from Gould et al. (2023b).

The qualitative comparison between RGBS and the ground shown in Figure 1
neglects some potentially important details. In particular, the ground-based
planets are actual detections whereas the RGBS planets are simulated by
extrapolating an assumed mass function into new regimes. The mass function
used in the simulations shown (Penny et al., 2019) assumes the occurrence rate
of planets is flat (in log space) below $m_{\rm p}=5.2M_{\oplus}$, whereas
Suzuki et al. (2016) found a sharp break in the mass-ratio distribution with
few or no planets smaller than $\log q\sim-5$. If the true mass function has
fewer smaller or wide-orbit planets than assumed, the qualitative differences
between RGBS and the ground would likely be reduced. Fortunately, Gould et al.
(2023b) have shown that RGBS should, in fact, be intrinsically more sensitive
to planet detections than the ground and also that that advantage is most
pronounced at low magnifications, e.g., for wide-orbit or free-floating
planets.

In brief, Gould et al. (2023b) showed that 10 years of the Korea Microlensing
Telescope Network (KMTNet) survey will yield of order 10 times as many
microlensing events as RGBS, but that RGBS has an advantage in signal-to-noise
ratio. This advantage in signal-to-noise ratio arises from the difference in
the sources probed by each survey and RGBS’s lower background (mainly due to
lower sky brightness).

Because RGBS is deeper than KMTNet, the typical source in RGBS is
intrinsically fainter and smaller. The bottom panel of Figure 3 shows the
result of matching the cumulative fractions of KMTNet and RGBS sources to
quantify this effect; i.e., a given fraction of RGBS sources will all be
brighter than some magnitude, $M_{I}$ (Roman) and for the same fraction,
KMTNet sources will all be brighter than some (brighter) magnitude, $M_{I}$
(KMT). Then, any given RGBS source can be matched to the corresponding KMTNet
source, and the total signal-to-noise ratios for those sources can be directly
compared (middle and top panels of Figure 3). At high magnification, $A$,
events tend to be above the background in both KMTNet and RGBS, but at low
magnification, e.g. $A\sim 1$, RGBS sources remain above the background
whereas KMTNet sources are below. This gives RGBS a large advantage in SNR at
low magnifications.

Low magnifications also correspond to the regimes of interest for wide-orbit,
small, and free-floating planets. Because wide-orbit planets produce planetary
caustics far from their host stars, if the planets are detected through that
channel (rather than from central caustic perturbations), the perturbations
must occur at low magnification ($A\sim 1$). The situation is analogous for
free-floating planets. For small planets, ground-based sensitivity is limited
to detections with $\log s\sim 0$, for which the caustic structures occur at
moderate to high magnification (Abe et al., 2013; Yee et al., 2021). By
contrast, to the extent that Roman detections are coming from planetary
caustic perturbations, those will also tend to be at lower magnification.

In conclusion, the work of Gould et al. (2023b) shows that regardless of the
intrinsic mass (or mass-ratio) distribution, RGBS has the most advantage over
the ground for wide-orbit, small, or free-floating planet detections.

##  4 Detection & Characterization

###  4.1 Wide-Orbit Planets

Figure 4: The “bump”-like anomaly in OGLE-2017-BLG-1806 has multiple possible
solutions with different values of the separation $s$ and mass ratio $q$. Such
anomalies must also be checked for the 1L2S (binary source) degeneracy, which
has a second source star instead of a planet (Figure from Zang et al., 2023).

The scientific impact of RGBS depends not only on simply detecting wide-orbit
planets, but also on the ability of RGBS to characterize and interpret them.
If these planets can be successfully characterized, RGBS will open a new
region of parameter space that will enable a measurement of the planet
occurrence rate as a function of separation from $\lesssim 1\,$ au to $\gtrsim
30$ au.

The region of planet-host separation that is of greatest interest for RGBS is
planets with $\log s>0.4$ (i.e, $s\gtrsim 2.5$). In this regime, the caustics
are well-separated, so a planet may either be detected through a central
caustic perturbation or a planetary caustic perturbation. Both of these
regimes are subject to degeneracies.

Central caustic detections very often suffer from an intrinsic $s\rightarrow
s^{-1}$ degeneracy (Griest & Safizadeh, 1998) meaning that they cannot be
unambiguously identified as wide-orbit planets. While they are rare relative
to the total number of events, they may contribute a larger fraction of
detected planets due to the fact that the planet perturbations occur near the
peak of the event. This makes them relatively easier to detect because the
anomalies occur when the events are brighter and the underlying stellar events
are easier to detect. However, if they suffer from a severe degeneracy in $s$,
they are not good candidates for a study of the separation distribution of
planets. Hence, it is crucial to investigate what fraction of Roman wide-orbit
planets detections will be due to central caustics, and among these, for what
fraction (if any) the $s\rightarrow s^{-1}$ degeneracy can be resolved.

While planetary caustic perturbations do not suffer from the $s\rightarrow
s^{-1}$ degeneracy, they are subject to a number of other degeneracies that
may lead to ambiguity in interpreting an observed anomaly. Such planet
perturbations manifest as “bump-type” anomalies, i.e., the planet causes an
increase in source magnification. Bump-like anomalies may also be produced by
either a binary source (1L2S; Gaudi, 1998) or interacting with the triangular
caustic of a larger $q$ planet with $s<1$. In some cases, such anomalies may
also suffer from the von Schlieffen vs. Cannae degeneracy, which can lead to
ambiguities in both mass ratio $q$ and the source size $\rho$, affecting the
physical interpretation of the event (Hwang et al., 2018). Finally, there can
also be degeneracies as to which side of the caustic the source passes, i.e.,
the “inner-outer” degeneracy (Gaudi & Gould, 1997). Even if unresolved, the
uncertainties due to the “inner-outer” degeneracy are usually without major
physical implications, so we do not discuss them here.

For illustration, Figure 4 shows an example of some of these potential
degeneracies as seen in ground-based data. In this event, we see that there
are subtle and not so subtle differences in the light curves of the various
models. Hence, light curves with sufficient cadence and precision should be
able to resolve these various degeneracies based on the detailed morphology of
the perturbation. However, this is just one example, and in this example, the
source crosses the caustic structure leading to a distinct, localized,
perturbation. By contrast, Zhu et al. (2014) predicted that about half of all
planets would be detected from non-caustic crossing perturbations, and Jung et
al. (2023) has confirmed this prediction for KMTNet. These perturbations can
be much weaker making the potential degeneracies more severe.

A detailed investigation is needed to explore how these degeneracies vary with
various properties, such as the amount of blending, the source size, and the
location of the source trajectory relative to the caustic (which affects the
strength of the perturbation). This simulation will need to determine the
cadence of observations needed to resolve these degeneracies given the
expected photometric precision of Roman and also the fraction of cases for
which the degeneracies cannot be resolved.

An additional consideration for planetary caustic perturbations is that they
are likely to occur well away from the peak of the stellar event. For example,
with $\log s=0.4$ the perturbation will occur at $\tau\equiv(t_{0}-t)/t_{\rm
E}\gtrsim 2.1$. Hence, for a typical event with $t_{\rm E}=20\,$ days, the
interesting wide-orbit planets will all have $\Delta t>42\,$ days. So for many
of them, the peak of the event will not be captured due to the short ($\sim
72\,$ day) Roman observing season.

This poses two fundamental challenges. First, it may be more challenging to
detect the event in the first place due to the lack of signal from the stellar
event. Second, there may be fundamental challenges to characterizing the
planet. When the peak of an event is not captured, there is uncertainty in
$u_{0}$ which leads to ambiguities in $t_{\rm E}$. These uncertainties can
then propagate to uncertainties in $s$ and $q$. Furthermore, such partial
light curves may also lead to uncertainties in the measurement of the
microlens parallax, one of the observables used to constrain the masses of the
host star. The potential for uncertainty in these parameters for wide-orbit
planets should be investigated in more detail.

However, this degeneracy can also be mitigated through additional
observations. The problem of characterizing the underlying stellar event will
not be resolved by Roman observations in subsequent observing windows ($\sim
110\,$ days later), because the events will already have returned to baseline.
But it can be mitigated by either increasing the duration of the Roman
observing window or obtaining observations from another observatory (e.g., the
ground) during the RGBS gaps to capture and characterize the peak of the light
curve. The latter would be facilitated by choosing lower extinction fields
that are accessible with ground-based optical telescopes.

###  4.2 Small Planets

Figure 5: Small mass ratio planets can produce “bump”-like perturbations that
may be subject to degeneracies such as the “central-resonant” degeneracy. This
example shows two possible models for the planet in OGLE-2016-BLG-1195.
Similar to potential degeneracies for wide-orbit planets, the RGBS cadence
will need to be high enough to distinguish between various possible models.
(Figure from Gould et al., 2023a)

The issues for small ($\log q\lesssim-4.5$) are similar to the issues for
wide-orbit planets. Some fraction of those planets will be detected as short-
duration, “bump”-like anomalies. As an example, consider Figure 5, which shows
a “bump”-like perturbation due to a $\log q\sim-4.5$ planet (Gould et al.,
2023a). This perturbation can be roughly described by either a central or a
resonant caustic model. In these two different models, $q$ differs by a factor
of $\sim 2$ and there is also a 10% difference in $\rho$, which will affect
the measurement of the Einstein radius, $\theta_{\rm E}$, and so the
constraint on the host mass. There are also other cases in which this type of
degeneracy leads to larger uncertainties in $\rho$ (e.g., Ryu et al., 2022).
Binary source solutions must also be considered for the case shown in Figure
5.

So, the question is the same as for wide-orbit planets: Given the expected
photometric precision, is the cadence high enough to measure the detailed
light curve morphology with sufficient accuracy to distinguish between
potentially degenerate solutions?

In addition, the perturbation in Figure 5 also suffers from the $s\rightarrow
s^{-1}$ degeneracy. Because this degeneracy is mathematical in nature, it is
often unresolved. Although it does not affect our ability to measure the mass
ratio function, we should consider the potential impact on our ability to
measure the separation distribution of small planets. For example, if (as we
very roughly estimate) 2/3 of detectable planetary anomalies suffer from this
degeneracy, we would need three times as many planets to achieve a given
statistical precision, and so would need to monitor three times as many stars.

###  4.3 Free-Floating Planets

Figure 6: Relative expected number of finite-source point-lens events detected
by a 10-yr KMT survey compared to Roman as a function of the Einstein radius,
$\theta_{\rm E}$, which is proportional to the square-root of the lens mass.
Roman has the most scientific power for $\theta_{\rm E}<1\mu{\rm as}$, so the
characterization of those events is what will set the requirements for the
RGBS cadence. Figure from Gould et al. (2023b)

The majority of RGBS free-floating planet candidates are essentially an
extreme limit of wide-orbit planetary-caustic detections. Once they have been
detected, the major issue for characterizing them is assessing whether or not
they have a host star. One aspect of this is searching for an underlying
stellar microlensing event, which might peak outside the Roman observing
window. As with wide-orbit planets this search would benefit from observations
between Roman observing windows from another observatory. Another aspect of
host searches would be to search for a perturbation in the free-floating
planet candidate event due to the host star’s effect on the planetary caustic.
Again, as with wide-orbit planets, this requires that the cadence to be high
enough to measure the detailed light curve morphology, although tests need to
be done to establish whether the required cadence is different from wide-orbit
planets with distinct stellar events.

For the minority of RGBS events with measurable finite-source effects, Gould
et al. (2023b) have shown that Roman has a substantial advantage over KMTNet
only for the smallest Einstein radii: $\theta_{\rm E}/\mu{\rm as}\lesssim 1$.
See Figure 6. Such events are extremely valuable because finite-source effects
allow a measurement of $\theta_{\rm E}$, which is a much stronger constraint
on the lens mass than $t_{\rm E}$ alone. Furthermore, the small $\theta_{\rm
E}$ regime that RGBS will dominate also corresponds to the smallest free-
floating planet candidates ($\sim 1$ to a few $M_{\oplus}$). This is the
regime that RGBS can contribute most to measuring the mass function of free-
floating planet candidates. Hence, the cadence of Roman needs to be high
enough to measure the finite-source effect, i.e., $\rho$, for this class of
event.

##  5 A Note on Colors/Multi-band Observations

The nominal RGBS will take most of its observations in $W146$-band, with
observations in $Z087$-band every 12 hours. In principle, such observations
can be used to confirm the microlensing nature of a perturbation (which should
be achromatic). In practice, this can almost always be determined on the basis
of light curve morphology. So, the primary purpose of the $Z087$-band
observations is to measure the colors of the sources.

In principle, if a given perturbation is due to a binary source rather than a
planet, and if the two components of the binary source have significantly
different colors, it is possible to rule out a planetary solution if a change
in color is observed during the perturbation. However, if the color remains
the same, depending on the flux ratio between the sources, a binary source
solution is not necessarily ruled out because the two source stars may simply
have the same color. In addition, in practice, the short durations of the
perturbations mean that it is relatively unlikely that a $Z087$-band
observation will be taken during the perturbation, so it will be difficult to
measure such a color change.

Due to limitations in the number of filter wheel changes for Roman, it is
doubtful that the cadence of $Z087$-band observations can be increased
sufficiently to guarantee a color measurement during the short perturbations
that are typical of wide-orbit or small planets. This could be investigated in
more detail, which would require assessing both the fraction of cases for
which a significant color change would be expected as well as the number of
$Z087$-band observations required to measure it. However, because at least 2
or 3 $Z087$-band observations would likely be required during a few hour
perturbation, color measurements are not likely to be a viable channel for
distinguishing between binary source and planetary models. Hence, the cadence
of $Z087$-band observations only needs to be sufficient to measure the color
of the primary source, which may imply that the cadence can be further reduced
from the nominal $1/12\,{\rm hr}^{-1}$.

##  6 Summary

Maximizing the scientific return of Roman requires focusing on the scientific
discovery space opened up by Roman relative to the ground: i.e., planets in
wide orbits ($\log s\gtrsim 0.4$), the smallest mass-ratio planets ($\log
q\lesssim-4.5$), and free-floating planet candidates (especially those with
$\theta_{\rm E}\lesssim 1\mu{\rm as}$). However, capitalizing on that leverage
requires not just detecting such planets but characterizing them sufficiently
that they can be used in a statistical analysis. In particular, the signals
from all three categories are all prone to light curve degeneracies that may
lead to ambiguities in the planet mass-ratio $q$, separation $s$, and the size
of the source $\rho$ (used to measure $\theta_{\rm E}$ and constrain the host
mass). Bound planets may also have light curves that are degenerate with
models that include a second source rather than a planet.

The most immediate need for designing the Roman Galactic Bulge Time Domain
Survey is a detailed simulation of wide-orbit and small planetary
perturbations to investigate how well the planet perturbations will be
characterized. This simulation will need to answer the following questions:

  * •

What fraction of the planet detections will be due to central caustic vs.
planetary caustic perturbations?

  * •

What fraction of those will be due to caustic crossings rather than weak
distortions?

  * •

For what fraction of planet perturbations can potentially degenerate light
curve solutions be resolved in order to conclusively detect the planet and
measure $q$ and $s$ within some tolerance?

  * •

What cadence of observations is needed to resolve these degeneracies?

  * •

What fraction of planet detections will be unambiguously characterized?

If there is a large fraction of cases with unresolved degeneracies, the survey
area may need to be increased to account for the reduced number of “useable”
planets. This will need to be balanced with the minimum cadence for
distinguishing between solutions. In addition, it should be studied whether a
mix of cadences over a larger area (as done in ground-based surveys) would be
better at balancing these two considerations.

For wide-orbit planets, simulations also need to account for the fact that the
stellar microlensing perturbation may be very weak and/or occur outside of the
RGBS observing season. Such simulations need to address these questions:

  * •

For what fraction of wide-orbit planets will the microlensing event be
detected on the basis of the planetary, rather than the stellar, signal?

  * •

In cases for which only part of the stellar light curve is captured by RGBS,
what are the resulting uncertainties in $t_{\rm E}$ and $\pi_{\rm E}$, and how
do those uncertainties affect our ability to measure $q$ and $m$?

  * •

What cadence of observations between RGBS observing windows is needed to
reduce those uncertainties below some tolerance?

It is likely that a cadence of order 1 observation per day or every few days
between RGBS observing seasons will be sufficient for characterizing the
stellar events. However, unless the pointing tolerances of Roman are
significantly relaxed, these observations will need to be taken a different
observatory. Choosing less extincted fields would enable these observations to
be taken from the ground, e.g., with LSST.

These investigations and related trade-studies must be done in order to
maximize Roman’s ability to take advantage of new parameter space. In
particular, one of the unique features of Roman is that the majority of Roman
planets will come with measurements of the host masses and system distances.
This highlights the importance of fully understanding these issues and
optimizing the survey to overcome them, because if the degeneracies can be
resolved, this will result in direct measurements of the masses and projected
semi-major axes of these planets.

## References

  * Abe et al. (2013) Abe, F., Airey, C., Barnard, E., et al. 2013, MNRAS, 431, 2975
  * Gaudi (1998) Gaudi, B. S. 1998, ApJ, 506, 533
  * Gaudi & Gould (1997) Gaudi, B. S., & Gould, A. 1997, ApJ, 477, 152
  * Gould et al. (2023a) Gould, A., Shvartzvald, Y., Zhang, J., et al. 2023a, arXiv e-prints, arXiv:2303.08876
  * Gould et al. (2023b) Gould, A., Ryu, Y.-H., Yee, J. C., et al. 2023b, arXiv e-prints, arXiv:2306.04870
  * Griest & Safizadeh (1998) Griest, K., & Safizadeh, N. 1998, ApJ, 500, 37
  * Hwang et al. (2018) Hwang, K.-H., Udalski, A., Shvartzvald, Y., et al. 2018, AJ, 155, 20
  * Jung et al. (2023) Jung, Y. K., Zang, W., Wang, H., et al. 2023, AJ, 165, 226
  * Penny et al. (2019) Penny, M. T., Gaudi, B. S., Kerins, E., et al. 2019, ApJS, 241, 3
  * Ryu et al. (2022) Ryu, Y.-H., Kil Jung, Y., Yang, H., et al. 2022, AJ, 164, 180
  * Suzuki et al. (2016) Suzuki, D., Bennett, D. P., Sumi, T., et al. 2016, ApJ, 833, 145
  * Yee et al. (2021) Yee, J. C., Zang, W., Udalski, A., et al. 2021, arXiv e-prints, arXiv:2101.04696
  * Zang et al. (2023) Zang, W., Jung, Y. K., Yang, H., et al. 2023, AJ, 165, 103
  * Zhu et al. (2014) Zhu, W., Gould, A., Penny, M., Mao, S., & Gendron, R. 2014, ApJ, 794, 53