# Relative Position Prediction as Pre-training for Text Encoders

Rickard Brüel-Gabrielsson
MIT CSAIL
<EMAIL_ADDRESS>
&Chris Scarvelis
MIT CSAIL
<EMAIL_ADDRESS>

###### Abstract

Meaning is defined by the company it keeps. However, company is two-fold: It’s
based on the identity of tokens and also on their position (topology). We
argue that a position-centric perspective is more general and useful. The
classic MLM and CLM objectives in NLP are easily phrased as position
predictions over the whole vocabulary. Adapting the relative position encoding
paradigm in NLP to create relative labels for self-supervised learning, we
seek to show superior pretraining judged by performance on downstream tasks.

##  1 Introduction

_Word embeddings_ form a core component of modern deep learning-based natural
language processing. Intuitively, word embeddings map words to vectors in a
semantically-meaningful manner: a desired property of word embeddings is that
similarity between word representations in their embedding space (as measured
by well-understood (dis)similarity measures such as cosine similarity or
Euclidean distance) reflect some notion of semantic similarity.

These embeddings are typically constructed by leveraging Harris’
_distributional hypothesis_ Harris (1954), which posits that words which are
similar in meaning tend to often co-occur in the same contexts. This
hypothesis implies that we may use essentially statistical, easily-measured
properties of words as a proxy for semantic similarity, which is typically
more difficult to quantify. Crucially, word co-occurrence statistics may be
computed without the intervention of human annotators, enabling word
embeddings to be trained in an _unsupervised_ manner.

Classical approaches such as latent semantic analysis (LSA) Deerwester et al.
(1990) construct dense, low-dimensional word embedding vectors by performing a
low-rank factorization (typically a singular value decomposition (SVD)) of a
(possibly reweighted) term-document matrix $X\in\mathbb{R}^{|V|\times|D|}$
such that $X_{ij}$ stores the frequency at which term $i$ occurs in document
$j$.

More recently, algorithms such as word2vec Mikolov et al. (2013) have been
developed to construct word embeddings using fast and scalable algorithms such
as skip-gram with negative sampling (SGNS). These embeddings are constructed
by solving a classification task in a self-supervised manner: They model the
probability that a word $c$ is in the _context_ of word $w$ as an increasing
(in practice generally sigmoid) function of the dot product $\langle
w,c\rangle$ of their embeddings and optimize the corresponding embeddings such
that words that do often co-occur are assigned embedding vectors with large
dot product.

These word embeddings learned by these algorithms possess a number of striking
properties. Aside from encoding similarity in word meaning, they have been
shown to encode _relational_ similarity: Computing the expression
($\vec{king}$) - ($\vec{man}$) + ($\vec{woman}$) results in a vector that’s
close to the word embedding for ”queen,” which matches our intuitive notion of
the relationships between the words in this expression. Furthermore, there
exists a body of theoretical work connecting word2vec to the factorization of
a shifted pointwise mutual information matrix Levy and Goldberg (2014), and
subsequent work Hashimoto et al. (2016) has framed popular word embedding
algorithms as consistent methods for performing metric recovery in semantic
space under an elegant random-walk based model of corpus generation.

However, word embeddings computed by word2vec and GloVe possess a number of
drawbacks. Most notably, these embeddings are _static_ : A single embedding
vector is computed for each word in the vocabulary. Given that many words in
natural languages are polysemous, this is a substantial limitation, and it is
reasonable to consider word embeddings that are capable of capturing different
meanings of the same word.

This problem is partially resolved by _dynamic_ (contextual) word embeddings
such as ELMo Peters et al. (2018) and BERT Devlin et al. (2019) and GPT-2
Radford et al. (2019), which learn word representations that are dependent on
a word’s context. This allows the word embedding model to leverage the fact
that a word’s meaning may be disambiguated via its context: For example, the
appropriate meaning of ”bank” cannot be determine from the word itself, but it
_can_ be determined from examining its context in the sentences ”I deposited
the cheque at the bank” and ”I went fishing by the river bank”. These
embeddings are trained by optimizing causal or masked language modeling
(CLM/MLM) objectives: They are trained to correctly predict the next token in
a sequence (CLM) or to predict a randomly-masked token (MLM). The resulting
embeddings have achieved state of the art performance on a variety of
downstream NLP tasks.

We propose an alternative approach based on optimizing a _relative position_
objective. We train a language model to predict the (signed) relative
positions $p_{ij}$ of pairs of words $x_{i},x_{j}$ in a context
$x_{1},...,x_{k}$. We believe that this approach presents several advantages
over existing CLM and MLM objectives, including:

  1. 1.

Increased label density. In predicting relative positions, we have $n\times n$
labels instead of $n$ (CLM) or $n\times p$ (MLM), where $n$ is the length of
the sequence and $p$ is masking probability.

  2. 2.

More effective sampling. Examples in the current context are more likely to be
relevant than other words in the vocabulary. We can improve sampling even more
by (i) finding the hardest negative examples, (ii) exploiting the fact that
far away positions don’t have to be exact, and (iii) predicting relative
positions within batches.

  3. 3.

Stronger signal. Many positions are distance-like and we do not need to encode
our labels as one-hot targets; we can instead use continuous distance as
targets, which allows us to use a variety of loss functions and encode
stronger signals. (We can take inspiration from metric learning and self-
supervised learning on images and 3D data).

  4. 4.

Computational efficiency. We avoid computing an output distribution over the
complete vocabulary as in MLM and CLM.

  5. 5.

More generality. Since relative positions have successfully been adopted in
both vision and graph learning Brüel-Gabrielsson et al. (2022), our approach
directly translates to those settings; especially where the topology is more
varied than in NLP.

##  2 Related Work

Self-supervised representation learning has been widely adopted in the NLP
literature. Models which incorporate self-supervised learning include seminal
works such as Word2Vec Mikolov et al. (2013), ELMo Peters et al. (2018), GPT
Brown et al. (2020), BERT Devlin et al. (2019), BART Lewis et al. (2020),
Electra Clark et al. (2020), XLNet Yang et al. (2019), and T5 Raffel et al.
(2020). These approaches rely on an objective that encourages the model to
reconstruct some corrupted/noisy text of language; typically incomplete text
with certain tokens missing. The primary advantage of this approach is that it
enables the model to learn semantically-meaningful contextual word
representations from large amounts of unlabeled text ”in the wild.” However,
computing word probabilities in a language model typically involves computing
a distribution over the entire vocabulary; for sufficiently large vocabularies
(which we often encounter in practice) this is a costly operation. This
problem is exacerbated in the setting of self-supervised learning for computer
vision and 3D learning tasks, where the spaces of possible images or 3D
configurations are infinite.

Contrastive learning Gutmann and Hyvärinen (2010) has been used to learn a
distribution without requiring the normalizing factor over the whole
vocabulary, though it requires negative examples. Other approaches Chen and He
(2020) use Siamese Representation Learning; this is akin to a clustering
method that uses architecture asymmetries to avoid model collapse to
attributing all points to a single cluster, and thus doesn’t need negative
examples. The approach of Noroozi and Favaro (2016) is to construct ”jigsaw
puzzles” from patches of an image which deep neural networks are trained to
solve. Similarly, Doersch et al. (2015) learns visual representations by
training a convolutional neural network to predict the relative position of
randomly-chosen patches in unlabeled images. These works show that this
pretext task leads to useful representations for downstream image recognition
tasks.

Our work aims to provide a useful pre-trained deep language model via a self-
supervised task that essentially corresponds to reshuffling the words into the
correct order. However, we do this by predicting the relative positions rather
than the absolute positions; hence, achieving increased label density,
stronger signal, and more generality.

##  3 Technical Approach

Given a context $c=x_{1},\dots,x_{k}$, a word $x_{i},1\leq i\leq k$, and
(relative) positions $p=p_{i,j},1\leq i,j\leq k$, masked language models
(MLMs) and causal language models (CLMs) attempt to correctly predict
$P(x_{i}|c-x_{i},p)$ for a training dataset, i.e. the correct word given the
context and relative positions. In contrast, our position-centric approach
broadly attempts to predict $P(p_{i,j}|c,p-p_{i,:}-p_{:,j})$, i.e. the correct
_relative positions_ of pairs of words given their context and the words
themselves.

Let $F:\mathbb{R}^{k\times h}\rightarrow\mathbb{R}^{k\times h}$ be a
Transformer and let $e_{T}:\mathcal{V}\rightarrow\mathbb{R}^{d}$ be the target
embeddings. Then one typically models the probability of the $i$-th word given
the remainder of its context as follows:

$P(x_{i}|c-x_{i},p)=\frac{\exp(e_{T}(x_{i})^{T}F(x_{1},\dots,x_{k})_{i})}{\sum_{x_{j}\in\mathcal{V}}\exp(e_{T}(x_{j})^{T}F(x_{1},\dots,x_{k})_{i})}$

with appropriately masked tokens or causal masking. In our position-centric
approach we model the probability of each pair of words’ relative positions as
follows:

$P(p_{i,j}|c,p-p_{i,:}-p_{:,j})\\\
=\phi(F(x_{1},\dots,x_{k})_{i},F(x_{1},\dots,x_{k})_{j})$

where $\phi:\mathbb{R}^{2\times
h}\rightarrow\mathcal{P}(\\{p_{i,j}|i,j\in\mathbb{N}\\})$ is a function to the
set of all probability distributions over the relative positions.

It is unclear a priori how we might construct $\phi$. Typically a Transformer
model computes the following (for each $i$ and $j$) in its multi-head
attention module:

$\textrm{Concat}(\textrm{head}_{1},\dots,\textrm{head}_{n_{h}})_{i,j}\in\mathbb{R}^{n_{h}}$

where

$\textrm{head}_{l}=(F(x_{1},\dots,x_{k})_{i}W^{l}_{q})^{T}(F(x_{1},\dots,x_{k})_{j}W^{l}_{k})$

and where $W_{q},W_{k}\in\mathbb{R}^{h\times h}$ are learnable weight matrices
and $n_{h}$ is the number of heads.

A simple approach for constructing an appropriate function $\phi$ is to first
retrieve these head-scores, corresponding to the unknown relative positions in
the last attention layer of the Transformer model and append a readout layer
to predict the ground truth relative positions:

$\psi(\textrm{Concat}(\textrm{head}_{1},\dots,\textrm{head}_{n_{h}})_{i,j}))\in\mathbb{R}^{n_{p}}$

Here $\psi:\mathbb{R}^{n_{h}}\rightarrow\mathbb{R}^{n_{p}}$ can be a linear
layer or a sequence of fully connected layers, and $n_{p}=|\\{p_{i,j}\ |\
i,j\in\mathbb{N}\\}|$. By normalizing the output of $\psi$ via a softmax
function we obtain the desired probability distribution over all possible
relative positions.

###  3.1 Masking or Permuting Positions

The previous discussion adopts the perspective of masking some positions and
then predicting them. An alternative approach is to _permute_ rather than mask
positions. This raises the following questions:

  1. 1.

If we permute positions, should our model predict only the permuted positions,
or should it also predict the non-permuted positions?

  2. 2.

If we permute positions and predict the non-permuted positions as well, should
we instead consider the binary problem of predicting whether a position is
correct or not?

We call the approach of masking positions and predicting them Position-Mask-
Language-Modeling (PMLM) and the approach of permuting positions are
predicting the correct one for Position-Permutation-Language-Modeling (PPLM).
Specifically, when permuting positions we permute a certain percentage of all
tokens and we predict the correct positions of all tokens; if we predict just
the permuted positions we call it PPLM-Some, and if we predict binary correct-
or-not-correct positions for each position we call it PPLM-Binary.

We investigate the effectiveness of masking and these alternative permutation-
based approaches empirically in the results section.

##  4 Results

During pre-training, we train on the BookCorpus dataset Zhu et al. (2015) and
English-language Wikipedia Foundation (0123). We use as our transformer
architecture a Roberta-Transformer Liu et al. (2019) with 12 layers. We employ
a sequence length of 128 and batch size of 64. We train for 40K batch
iterations, which takes about 4 hours on a single Tesla V100 GPU.

We evaluate our pre-trained model’s performance on three downstream tasks
after fine-tuning: multi-genre natural language inference (MNLI) Williams et
al. (2018), linguistic acceptability on the Corpus of Linguistic Acceptability
(CoLA) warstadt-etal-2019-neural, and question-answering natural language
inference (QNLI) Rajpurkar et al. (2016). All of these tasks are components of
the General Language Understanding Evaluation (GLUE) benchmark Wang et al.
(2018). For all downstream tasks, we fine-tune our pre-trained model by
following the procedure outlined in Appendix B of Clark et al. (2020).

The MNLI corpus is a crowd-sourced collection of 433k sentence pairs annotated
with textual entailment information. Each sentence pair is assigned one of
three labels: neutral, entailment, or contradiction, and the objective of the
task is to correctly predict the label for each sentence pair.

The CoLA dataset is a compilation of 10.6k English word sequences annotated
with a judgment of the sequence’s grammatical acceptability. These sequences
are drawn from works of linguistic theory. The task is to correctly predict
each sentence’s acceptability label.

QNLI is based on the Stanford Question Answering Dataset Rajpurkar et al.
(2016). This dataset consists of question-paragraph pairs. Each paragraph
contains a sentence which answers the associated question. QNLI modifies this
dataset by converting the task to sentence-pair classification by pairing the
question sentence with each sentence in the associated paragraph; the task is
then to determine whether each sentence in the paragraph answers the question.

For further training details see Appendix A. We record our results in Table 1.

Pretraining: MLM-30% PMLM-60% PPLM-60% PPLM-Some-60% PPLM-Binary-60% MNLI
Accuracy: 0.597 0.6123 0.626 0.590 0.595 CoLA Accuracy: 0.614 0.618 0.617
0.618 0.607 QNLI Accuracy: 0.666 0.642 0.673 0.631 0.634

Table 1: Results of Masked Language Modeling (MLM) vs versions of Positional
Masked Language Modeling (PMLM) and Positional Permuted Language Modeling
(PPLM).

Our results indicate that our proposed PMLM and PPLM pre-training objectives
achieve competitive performance on the downstream MNLI task. The largest
improvement in accuracy arises from using a model that is pre-trained using
the PPLM objective with 60% of the tokens permuted. Significant (though
smaller) improvements were achieved by pre-training using the PMLM objective,
which masks positions rather than permuting them.

On the other hand, pre-training using the PPLM-Some and PPLM-Binary led to
small degradations in performance relative to the MLM baseline. This indicates
that our model’s ability to predict the actual relative position of all tokens
is crucial to its ability to learn representations that lead to good
performance on the downstream MNLI task. Merely predicting the relative
positions of pairs of permuted words is insufficient; future work may seek to
determine whether this is a byproduct of insufficient label density in the
PPLM-Some problem. The PPLM is a strictly harder task than PPLM-Some and
involves determining whether an ordered pair of tokens have been permuted or
not, in addition to determining the ordered pairs relative position; in PPLM-
Some all relative positions of the ordered pairs that the model is asked to
predict have all been permuted, while this is not true for PPLM. Results show
this to be a crucial difference.

Our pre-training objectives led to more muted improvements on the MLM baseline
for the CoLA task. Whereas the MLM baseline achieved an accuracy of 0.614 on
CoLA, our best-performing model (PMLM-60%) achieved an accuracy of 0.618 on
the same task. On the QNLI task, our best-performing model (again PPLM-60%)
achieved an accuracy of 0.673, whereas the baseline accuracy was 0.666; this
is once again a notable improvement over the baseline, especially given the
small scale of our experiments.

##  5 Discussion

The results that we have outlined above indicate that our relative position-
based objective shows significant promise as a new method for pre-training
large transformer-based language models. We conjecture that some of the
benefits of our model arise from the increased label density afforded by
relative position-based training: The number of labels available for our model
to predict in pre-training is quadratic in the length of each context rather
than linear as is the case with the standard CLM and MLM objectives.

Further benefits arise from the increased computational efficiency of our
approach. As our model does not involve the computation of an output
distribution over the vocabulary but rather a distribution over the (much
smaller) set of valid relative positions, our model trains more quickly using
our objective and hence is able to achieve improved downstream performance
given a fixed wall-clock time budget. Note that in these experiments we
trained for a fixed number of iterations, not flops, and the MLM uses more
flops than PPLM and PMLM.

Finally, our model predicts relative positions of words within the same
context; these words are more likely to be relevant to each other than words
in the remainder of the vocabulary, potentially allowing our objective to
provide a stronger signal than traditional CLM/MLM objectives that predicts
words drawn from the entire vocabulary.

##  6 Conclusion

In this work, we have investigated the effectiveness of a _relative position
prediction_ pretext task for the purpose of pre-training language models. Our
results indicate that while there are some technical challenges associated
with this approach, relative position prediction can lead to notable
improvements in performance on downstream tasks relative to pre-training
baselines based on MLM or CLM objectives.

In future work, we hope to investigate how our model responds to increased
scale. In particular, we only trained our model for 4 hours on a single Tesla
V100 GPU. Modern language models are particularly likely to benefit from
extended training on very large datasets Brown et al. (2020), and we hope that
further training will expand the performance gap between our relative
position-based approach and the CLM/MLM objectives that are currently used in
state-of-the-art language models. We especially hope that our objective’s
computational efficiency will lead to significant advantages when training
models at scale; it is plausible that the small scale of our experiments in
this work may have dampened some the potential benefits of our approach.

In addition, there are several alternative losses that may be used that are
more distance-like; we could for example use regression losses rather than
classification losses, or incorporate optimal transport distances such as
Wasserstein distances between tokens.

## References

  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Brüel-Gabrielsson et al. (2022) Rickard Brüel-Gabrielsson, Mikhail Yurochkin, and Justin Solomon. 2022.  Rewiring with positional encodings for graph neural networks.
  * Chen and He (2020) Xinlei Chen and Kaiming He. 2020.  Exploring simple siamese representation learning.  _CoRR_ , abs/2011.10566.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  _CoRR_ , abs/2003.10555.
  * Deerwester et al. (1990) Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. 1990.  Indexing by latent semantic analysis.  _Journal of the American society for information science_ , 41(6):391–407.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Doersch et al. (2015) Carl Doersch, Abhinav Gupta, and Alexei A Efros. 2015.  Unsupervised visual representation learning by context prediction.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1422–1430.
  * Foundation (0123) Wikimedia Foundation. 0123.  Wikimedia downloads.
  * Gutmann and Hyvärinen (2010) Michael Gutmann and Aapo Hyvärinen. 2010.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , volume 9 of _Proceedings of Machine Learning Research_ , pages 297–304, Chia Laguna Resort, Sardinia, Italy. PMLR.
  * Harris (1954) Zellig S. Harris. 1954.  Distributional structure.  _WORD_ , 10(2-3):146–162.
  * Hashimoto et al. (2016) Tatsunori Hashimoto, David Alvarez-Melis, and Tommi Jaakkola. 2016.  Word embeddings as metric recovery in semantic spaces.  _Transactions of the Association for Computational Linguistics_ , 4(0):273–286.
  * Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014.  Neural word embedding as implicit matrix factorization.  In _Advances in Neural Information Processing Systems_ , volume 27. Curran Associates, Inc.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  In _Advances in Neural Information Processing Systems_ , volume 26. Curran Associates, Inc.
  * Noroozi and Favaro (2016) Mehdi Noroozi and Paolo Favaro. 2016.  Unsupervised learning of visual representations by solving jigsaw puzzles.  _CoRR_ , abs/1603.09246.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _The IEEE International Conference on Computer Vision (ICCV)_.

##  Appendix A Experiment Parameters

Shared hyperparameters for all experiments:

  1. 1.

vocab-size = 50265

  2. 2.

num-hidden-layers = 12

  3. 3.

hidden-size = 256

  4. 4.

intermediate-size = 1024

  5. 5.

max-position-embeddings = 128

  6. 6.

num-attention-heads = 16

  7. 7.

attention-probs-dropout-prob = 0.1

  8. 8.

hidden-dropout-prob = 0.1

  9. 9.

learning-rate = 5e-4


Hawking temperature for constant curvature black bole and its analogue in de
Sitter Space

Rong-Gen Cai1,***Email address<EMAIL_ADDRESS>and Yun Soo Myung2,†††Email
address<EMAIL_ADDRESS>
1Key Laboratory of Frontiers in Theoretical Physics, Institute of Theoretical
Physics, Chinese Academy of Sciences, P.O. Box 2735, Beijing 100190, China
2Institute of Basic Science and School of Computer Aided Science, Inje
University, Gimhae 621-749, Korea

Abstract

The constant curvature (CC) black holes are higher dimensional generalizations
of BTZ black holes. It is known that these black holes have the unusual
topology of ${\cal M}_{D-1}\times S^{1}$, where $D$ is the spacetime dimension
and ${\cal M}_{D-1}$ stands for a conformal Minkowski spacetime in $D-1$
dimensions. The unusual topology and time-dependence for the exterior of these
black holes cause some difficulties to derive their thermodynamic quantities.
In this work, by using globally embedding approach, we obtain the Hawking
temperature of the CC black holes. We find that the Hawking temperature takes
the same form when using both the static and global coordinates. Also it is
identical to the Gibbons-Hawking temperature of the boundary de Sitter spaces
of these CC black holes. Employing the same approach, we obtain the Hawking
temperature for the counterparts of CC black holes in de Sitter spaces.

##  1 Introduction

Over the past years, the so-called BTZ (Banados-Teitelboim-Zanelli) [1] black
hole solutions have played the important role in understanding microscopic
degrees of freedom of black hole. The BTZ black hole is an exact solution of
Einstein field equations with a negative cosmological constant in three
dimensions. It is well known the BTZ black hole can be constructed by
identifying points along the orbit of a Killing vector in a three dimensional
anti-de Sitter (AdS) space.

The BTZ black hole has a topology of ${\cal M}_{2}\times S^{1}$, where $M_{2}$
denotes a conformal Minkowski space in two dimensions. Following the same way
as done in three dimensions, one can construct analogues of the BTZ solution,
the so-called constant curvature (CC) black holes in higher $(D\geq 4$)
dimensional AdS spaces [2, 3, 4]. However, such black holes have topology of
${\cal M}_{D-1}\times S^{1}$ in $D$ dimensions, which is quite different from
the known topology of ${\cal M}_{2}\times S^{D-2}$ for the usual black holes
in $D$ dimensions. In addition, the exterior region of these CC black holes is
time-dependent and thus, there is no global time-like Killing vector [2].
Because of this, it is difficult to discuss Hawking radiation and
thermodynamics associated with these black holes. For example, see [5, 6, 7]
and references therein.

On the other hand, these spacetimes are interesting examples of smooth time-
dependent solutions. Particularly, they are consistent background spacetimes
for string theory at least to leading order since they are vacuum solutions to
Einstein field equations with a negative cosmological constant too. Further we
note that these spacetimes are time-dependent, the boundary metric is also
time-dependent, and it is asymptotically AdS. Therefore, it might open a
window to investigate dual strong coupling field theory in the time-dependent
backgrounds through the AdS/CFT correspondence [8]. Especially, the
$D$-dimensional CC black holes have the boundary topology of $dS_{D-2}\times
S^{1}$, where $dS_{D-2}$ denotes a $(D-2)$-dimensional de Sitter (dS) space.
Resorting the AdS/CFT correspondence, these CC black holes are gravity duals
to strong coupling conformal field theories living on $dS_{D-2}\times S^{1}$.
Finally it is observed in [9] that these CC black holes have a close
connection to the so-called “bubbles of nothing” in AdS space [10, 11]. The
bubbles of nothing were constructed by analytically continuing (Schwarzschild,
Reissner-Nordström, and Kerr) black holes in AdS spaces. The stress-energy
tensor for dual conformal field theories to these CC black holes was
calculated in [9, 11].

It is well known that there is the Gibbons-Hawking temperature $T_{\rm GH}$
for a comoving observer in a dS space [12]. This temperature may be viewed as
the Hawking temperature $T_{\rm HK}$ associated with cosmological horizon of
dS space. A $D$-dimensional dS space can be embedded as a hypersurface into a
$(D+1)$-dimensional Minkowski space. Then, the comoving observer in dS space
is identical to an observer with a constant acceleration in Minkowski space.
According to Davies [13] and Unruh [14], an observer with a constant
acceleration in Minkowski space will see a hot bath with the Davies-Unruh
temperature $T_{\rm DU}=a/2\pi$ where $a$ is the acceleration of the observer.
It turns out that the Gibbons-Hawking temperature of dS space is equivalent to
the Davies-Unruh temperature of the corresponding observer in Minkowski space.

One decade ago, it was shown that an observer with a constant acceleration $a$
in dS space will detect a temperature given by $\sqrt{a^{2}+1/l^{2}}/2\pi$,
where $l$ is the radius of the dS space [15]. This was soon generalized to the
cases of dS/AdS space by Deser and Levin [16] with temperatures of
$\sqrt{a^{2}\pm 1/l^{2}}/2\pi$. Further, Deser and Levin have shown that the
temperature is equivalent to the Davies-Unruh temperature for the
corresponding observer in Minkowski space. Further examples for the
equivalence have been shown by globally embedding curved spaces including BTZ,
Schwarzschild, Schwarzschild-AdS (dS), and Reissner-Nordström solutions into
higher dimensional Minkowski spaces in Ref.[17]. For more examples on the
equivalence, see [18] and references therein.

In this work, the “globally embedding approach” will be employed to determine
the Hawking temperature of CC black holes and positive CC spaces. This
approach shows a clear way to compute the Hawking temperature, in comparison
to other methods with ambiguity to calculate it.

In the next section, we show that the Hawking temperature of the constant
curvature black holes is given by $T_{\rm HK}=r_{+}/(2\pi l)$ using both the
static and global coordinates in AdS spaces. Further it is shown that the
Hawking temperature is identical to the Gibbons-Hawking temperature of the
boundary dS space. In section 3 we consider the counterparts of the CC black
holes in dS spaces. These are constant curvature (CC) spaces with the
cosmological horizon. We find that the Hawking temperature for these spaces
are given by $T_{\rm HK}=r_{+}/(2\pi l)$, where $r_{+}$ and $l$ are the
cosmological horizon radius and the radius of dS spaces, respectively. We give
our conclusions and discussions in section 4\. In this paper, we confine
ourselves to the five dimensional space. The generalization to other
dimensions is straightforward.

##  2 Hawking temperature of CC black holes

A five dimensional AdS space is defined as the universal covering space of a
surface obeying

$-x_{0}^{2}+x_{1}^{2}+x_{2}^{2}+x_{3}^{2}+x_{4}^{2}-x_{5}^{2}=-l^{2},$ (2.1)

where $l$ is the AdS radius. This surface has fifteen Killing vectors of seven
rotations and eight boosts. We consider the boost
$\xi=(r_{+}/l)(x_{4}\partial_{5}+x_{5}\partial_{4})$ with its norm
$\xi^{2}=r_{+}^{2}(-x_{4}^{2}+x_{5}^{2})/l^{2}$ where $r_{+}$ is an arbitrary
real constant. The so-called CC black hole is constructed by identifying
points along the orbit of the Killing vector $\xi$. Since the starting point
is the AdS, the resulting black hole has a constant curvature as the AdS does
show. The topology of the black holes is changed to be ${\cal M}_{4}\times
S^{1}$, which is quite different from the usual topology of ${\cal
M}_{2}\times S^{3}$ for five dimensional black holes. Here ${\cal M}_{n}$
denotes a conformal Minkowski space in $n$ dimensions. For more details for
the construction of the black hole, see [3, 4].

The CC black holes can be nicely described by using Kruskal coordinates. For
this purpose, a set of coordinates on the AdS for the region of $\xi^{2}>0$
has been introduced in Ref.[3]. The six dimensionless local coordinates
$(y_{i},\varphi)$ are given by

$\displaystyle x_{i}=\frac{2ly_{i}}{1-y^{2}},\ \ \ i=0,1,2,3$ $\displaystyle
x_{4}=\frac{lr}{r_{+}}\sinh\left(\frac{r_{+}\varphi}{l}\right),$
$\displaystyle
x_{5}=\frac{lr}{r_{+}}\cosh\left(\frac{r_{+}\varphi}{l}\right),$ (2.2)

with

$r=r_{+}\frac{1+y^{2}}{1-y^{2}},\ \ \
y^{2}=-y_{0}^{2}+y_{1}^{2}+y_{2}^{2}+y_{3}^{2}.$ (2.3)

Here the allowed regions are $-\infty<y_{i}<\infty$ and
$-\infty<\varphi<\infty$ with the restriction $-1<y^{2}<1$. In these
coordinates, the boundary at $r\to\infty$ corresponds to the hyperbolic “ball”
which satisfies $y^{2}=1$. The induced line element can be written down

$ds^{2}=\frac{l^{2}(r+r_{+})^{2}}{r_{+}^{2}}(-dy_{0}^{2}+dy_{1}^{2}+dy_{2}^{2}+dy_{3}^{2})+r^{2}d\varphi^{2}.$
(2.4)

Obviously, the Killing vector is given by $\xi=\partial_{\varphi}$ with its
norm $\xi^{2}=r^{2}$. The black hole spacetime could be obtained by
identifying $\varphi\sim\varphi+2\pi n$, and the topology of the black hole
takes the form of ${\cal M}_{4}\times S^{1}$ clearly.

On the other hand, the CC black holes can also be described by introducing
Schwarzschild coordinates. The local “spherical” coordinates
($t,r,\theta,\chi$) in the hyperplane $y_{i}$ are

$\displaystyle y_{0}=f\sin\theta\sinh(r_{+}t/l),\ \ \
y_{1}=f\sin\theta\cosh(r_{+}t/l),$ $\displaystyle y_{2}=f\cos\theta\sin\chi,\
\ \ \ \ \ y_{3}=f\cos\theta\cos\chi,$ (2.5)

where $f=[(r-r_{+})/(r+r_{+})]^{1/2}$, $0\leq\theta\leq\pi/2$, $0\leq\chi\leq
2\pi$, and $r_{+}\leq r<\infty$. One finds that the solution (2.4) can be
rewritten as

$ds^{2}=l^{2}N^{2}d\Omega_{3}^{2}+N^{-2}dr^{2}+r^{2}d\varphi^{2},$ (2.6)

where

$N^{2}=\frac{r^{2}-r_{+}^{2}}{l^{2}},\ \ \ d\Omega_{3}^{2}=-\sin^{2}\theta
dt^{2}+\frac{l^{2}}{r_{+}^{2}}(d\theta^{2}+\cos^{2}\theta d\chi^{2}).$ (2.7)

This is the black hole solution expressed in terms of Schwarzschild
coordinates. Here $r=r_{+}$ is the location of black hole horizon. In these
coordinates the solution seems static. However, we observe from (2.6) that the
form (2.7) does not cover the whole exterior region of black hole since the
difference of $y^{2}_{1}-y_{0}^{2}$ is required to be positive in the region
covered by these coordinates. Indeed, it has been proved that there is no
globally timelike Killing vector in this geometry [2].

Now we consider a static observer with constant
$(r>r_{+},\theta,\chi,\varphi)$ in the black hole background (2.6). To this
observer, we find that an associated acceleration $a_{5}$ is given by

$a_{5}^{2}=\frac{1}{l^{2}(r^{2}-r_{+}^{2})}\frac{1}{\sin^{2}\theta}\left(r^{2}\sin^{2}\theta+r_{+}^{2}\cos^{2}\theta\right).$
(2.8)

On the other hand, the acceleration of $a_{6}$ for the corresponding observer
in six dimensional embedding Minkowski space is given by

$a_{6}^{-2}=x_{1}^{2}-x_{0}^{2}=\frac{l^{2}(r^{2}-r_{+}^{2})}{r_{+}^{2}}\sin^{2}\theta.$
(2.9)

It is easy to check that these two accelerations obey the relation

$a_{6}^{2}=-\frac{1}{l^{2}}+a^{2}_{5}.$ (2.10)

This shows that the Davies-Unruh temperature for the local observer in six
dimensional Minkowski space is

$T_{\rm DU}=\frac{a_{6}}{2\pi}=\frac{r_{+}}{2\pi
l\sqrt{r^{2}-r_{+}^{2}}}\frac{1}{\sin\theta}.$ (2.11)

We note that the redshift factor of $\sqrt{-g_{00}}=lN\sin\theta$ for the
black hole (2.6) is necessary to define the Hawking temperature. Hence we
conclude that the Hawking temperature of the CC black hole is

$T_{\rm HK}=\sqrt{-g_{00}}~{}T_{\rm DU}=\frac{r_{+}}{2\pi l}.$ (2.12)

We notice that the Hawking temperature $T_{\rm HK}$ is consistent with the
inverse period of the Euclidean time derived from the solution (2.7).

As the case in four dimensions [4], there is another set of coordinates
covering the whole exterior of the Minkowskian black hole geometry as [9]

$\displaystyle y_{0}=f\sinh(r_{+}t/l),\ \ \ y_{1}=f\cos\theta\cosh(r_{+}t/l),$
$\displaystyle y_{2}=f\sin\theta\cos\chi\cosh(r_{+}t/l),\ \ \
y_{3}=f\sin\theta\sin\chi\cosh(r_{+}t/l),$ (2.13)

where $f$ is given by (2) and the allowed regions are $0\leq\theta\leq\pi$,
$r_{+}\leq r<\infty$, and $0\leq\chi\leq 2\pi$. In these coordinates, the
solution can be expressed as

$ds^{2}=N^{2}l^{2}d\Omega_{3}^{2}+N^{-2}dr^{2}+r^{2}d\varphi^{2},$ (2.14)

where $N^{2}=(r^{2}-r_{+}^{2})/l^{2}$ and

$d\Omega_{3}^{2}=-dt^{2}+\frac{l^{2}}{r_{+}^{2}}\cosh^{2}(r_{+}t/l)(d\theta^{2}+\sin^{2}\theta
d\chi^{2}).$ (2.15)

The time-dependence of the solution is manifest in this coordinate system. We
introduce a static observer located at constant $r>r_{+}$, $\varphi$ and
$\chi$, but $\theta=0$ due to the spherical symmetry of the solution [16].
Here, we find that the acceleration $a_{5}$ associated with the observer is

$a_{5}^{2}=\frac{r^{2}}{l^{2}(r^{2}-r_{+}^{2})},$ (2.16)

while the acceleration $a_{6}$ of the corresponding observer in six
dimensional Minkowski space is given by

$a^{-2}_{6}=x_{1}^{2}-x_{0}^{2}=\frac{r_{+}^{2}}{l^{2}(r^{2}-r_{+}^{2})}.$
(2.17)

They satisfy the relation (2.10) too. In this case, the Davies-Unruh
temperature is given by

$T_{\rm DU}=\frac{a_{6}}{2\pi}=\frac{r_{+}}{2\pi l\sqrt{r^{2}-r_{+}^{2}}}.$
(2.18)

Considering the redshift factor of $\sqrt{-g_{00}}$, we get the Hawking
temperature of the black hole in the line element of (2.14) as

$T_{\rm HK}=\frac{r_{+}}{2\pi l}.$ (2.19)

Thus we have obtained the Hawking temperature of the CC black hole by
employing globally embedding approach combined with the Davies-Unruh
temperature in six dimensional Minkowski space. The Hawking temperatures
(2.12) and (2.19) are our main results.

Here some remarks are in order. First, in general, Hawking temperature of
black hole depends on coordinates used to calculate it. That is, the Hawking
temperature may be different when using different coordinates, even for the
same black hole. In our case, we obtained the same Hawking temperature for the
CC black hole even when used the different coordinate systems (2.6) and
(2.14). Second, we mention that the Hawking temperature (2.12) is the same as
the inverse period of the Euclidean time for the Euclidean sector of the
solution (2.6). However, when used the coordinates (2.15), the Hawking
temperature is no longer the same as the inverse period of the Euclidean time.
In order to see this, let us consider carefully the Euclidean sector of the
black hole solution which can be obtained by replacing the time $t$ by
$-i(\tau+\pi l/(2r_{+}))$ in (2.15). In this case, $d\Omega_{3}^{2}$ becomes

$d\Omega_{3}^{2}=d\tau^{2}+\frac{l^{2}}{r_{+}^{2}}\sin^{2}(r_{+}\tau/l)(d\theta^{2}+\sin^{2}\theta
d\chi^{2}).$ (2.20)

In order that $d\Omega_{3}^{2}$ be a regular three-sphere, $\tau$ must have
the period of $\tau\sim\tau+\tilde{\beta}$ with

$\tilde{\beta}=\frac{\pi l}{r_{+}}.$ (2.21)

Clearly this is not the inverse of Hawking temperature. This shows that the
Euclidean method does not always provide a correct Hawking temperature of CC
black holes. However, using both coordinates (2) and (2), the Euclidean time
$\tau=it$ obtained by Wick rotation leads to the fact that it has a
periodicity with period $2\pi l/r_{+}$, which gives a correct Hawking
temperature of the black hole. This may be related to the issue of the factor
2 in [19].

Finally, we observe from (2.14) that the black hole solution has a boundary
topology $dS_{3}\times S^{1}$ at $r=\infty$. The three dimensional de Sitter
space $dS_{3}$ has a Hubble constant $H=r_{+}/l$. It is well known that for a
de Sitter space with a Hubble constant $H$, there is the Gibbons-Hawking
temperature $T=H/2\pi$ for a comving observer. We find that the Gibbons-
Hawking temperature in our case is identical to the Hawking temperature of the
CC black hole

$T_{\rm GH}=\frac{H}{2\pi}=\frac{r_{+}}{2\pi l}=T_{\rm HK}.$ (2.22)

##  3 Hawking temperature of a positive CC space

In this section we consider the analogue of the CC black hole in dS space.
This space is constructed by identifying points along the orbit of a Killing
vector in dS space. In fact, this space is a generalization of the three-
dimensional Schwarzschild-de Sitter solution in higher dimensions. This space
has a cosmological event horizon, and its topology is ${\cal M}_{D-1}\times
S^{1}$ where ${\cal M}_{D-1}$ denotes a $(D-1)$-dimensional conformal
Minkowski spacetime. Such space was constructed in Ref.[20].

As the case with a negative cosmological constant, we consider a five
dimensional de Sitter space, which can be viewed as a hypersurface embedded
into a six dimensional Minkowski space, satisfying

$-x_{0}^{2}+x_{1}^{2}+x_{2}^{2}+x_{3}^{2}+x_{4}^{2}+x_{5}^{2}=l^{2}$ (3.1)

with $l$ the radius of the dS space. This dS space has fifteen Killing vectors
of five boosts and ten rotations. We consider a rotational Killing vector
$\xi=(r_{+}/l)(x_{4}\partial_{5}-x_{5}\partial_{4})$ with its norm
$\xi^{2}=r_{+}^{2}/l^{2}(x_{4}^{2}+x_{5}^{2})$ where $r_{+}$ is an arbitrary
real constant. Identifying points along the orbit of a Killing vector $\xi$,
another one-dimensional manifold becomes compact and it is isomorphic to
$S^{1}$. Thus, we obtain a spacetime of topology ${\cal M}^{4}\times S^{1}$
with cosmological horizon. For details of the construction of the space, see
[20].

We can describe the spacetime in the region with $0\leq\xi^{2}\leq r_{+}^{2}$
by introducing six dimensionless local coordinates $(y_{i},\phi)$,

$\displaystyle x_{i}=\frac{2ly_{i}}{1+y^{2}},\ \ \ i=0,1,2,3$ $\displaystyle
x_{4}=\frac{lr}{r_{+}}\sin\left(\frac{r_{+}\phi}{l}\right),$ $\displaystyle
x_{5}=\frac{lr}{r_{+}}\cos\left(\frac{r_{+}\phi}{l}\right),$ (3.2)

where

$r=r_{+}\frac{1-y^{2}}{1+y^{2}},\ \ \
y^{2}=-y_{0}^{2}+y_{1}^{2}+y_{2}^{2}+y_{3}^{2}.$ (3.3)

Here the allowed regions are $-\infty<y_{i}<+\infty$ and
$-\infty<\phi<+\infty$ with the restriction $-1<y^{2}<1$ to have a positive
$r$. In the coordinates (3), the induced line element is

$ds^{2}=\frac{l^{2}(r+r_{+})^{2}}{r_{+}^{2}}(-dy_{0}^{2}+dy_{1}^{2}+dy_{2}^{2}+dy_{3}^{2})+r^{2}d\phi^{2},$
(3.4)

which is the same form as the case of a negative constant curvature [3].
However, it is noted that the coordinates (3) and the definition of $r$ differ
from those in the CC black holes. In this coordinate system, it is evident
that the Killing vector is $\xi=\partial_{\phi}$ with norm $\xi^{2}=r^{2}$.
Imposing the identification $\phi\sim\phi+2\pi n$, the solution has the
topology ${\cal M}_{4}\times S^{1}$.

We now introduce the Schwarzschild coordinates to describe the solution. Using
local “spherical” coordinates $(t,r,\theta,\chi)$ defined as

$\displaystyle y_{0}=f\sin\theta\sinh(r_{+}t/l),\
~{}~{}~{}~{}~{}y_{1}=f\sin\theta\cosh(r_{+}t/l),$ (3.5) $\displaystyle
y_{2}=f\cos\theta\sin\chi,\ ~{}~{}~{}~{}~{}y_{3}=f\cos\theta\cos\chi,$

where $f=[(r_{+}-r)/(r+r_{+})]^{1/2}$, and the allowed coordinate ranges are
$0<\theta<\pi/2$, $0<\chi<2\pi$, and $0<r<r_{+}$. The line element can be
expressed as

$ds^{2}=l^{2}N^{2}d\Omega_{3}^{2}+N^{-2}dr^{2}+r^{2}d\phi^{2}.$ (3.6)

Here $N^{2}=(r_{+}^{2}-r^{2})/l^{2}$ and

$d\Omega_{3}^{2}=-\sin^{2}\theta
dt^{2}+\frac{l^{2}}{r_{+}^{2}}(d\theta^{2}+\cos^{2}\theta d\chi^{2}).$ (3.7)

Clearly the location of $r=r_{+}$ represents a cosmological horizon. This
solution is the counterpart of a five dimensional CC black hole described in
the previous section. The only difference is that
$N^{2}=(r^{2}-r_{+}^{2})/l^{2}$ is replaced by $N^{2}=(r_{+}^{2}-r^{2})/l^{2}$
here. Further, in three dimensions, the corresponding induced line element
takes the form

$ds^{2}=-(r_{+}^{2}-r^{2})dt^{2}+\frac{l^{2}}{r_{+}^{2}-r^{2}}dr^{2}+r^{2}d\phi^{2},$
(3.8)

After a suitable rescaling of coordinates, it can be transformed to three
dimensional Schwarzschild-de Sitter solution [21]. In this sense, the solution
(3.6) can be viewed as an analogue of the three dimensional Schwarzschild-de
Sitter solution in five dimensions.

The solution (3.6) seems to be static, but it does not cover the whole region
within the cosmological horizon. It can be seen from the definition of
coordinates (3.5) because they must obey the constraint:
$y_{1}^{2}-y_{0}^{2}=f^{2}\cos^{2}\theta\geq 0$. Considering a static observer
located at constant $(r<r_{+},\theta,\chi$) in the background (3.6), we find
that the static observer has a constant acceleration $a_{5}$ as

$a_{5}^{2}=\frac{1}{l^{2}(r_{+}^{2}-r^{2})\sin^{2}\theta}\left(r^{2}\sin^{2}\theta+r_{+}^{2}\cos^{2}\theta\right),$
(3.9)

while the observer in six dimensional Minkowski space has a constant
acceleration $a_{6}$ as

$a_{6}^{-2}=x_{1}^{2}-x_{0}^{2}=\frac{l^{2}(r_{+}^{2}-r^{2})}{r_{+}^{2}}\sin^{2}\theta.$
(3.10)

These two accelerations are related to each other as

$a_{6}^{2}=1/l^{2}+a_{5}^{2}.$ (3.11)

According to Davies and Unruh, the observer has a temperature as

$T_{\rm DU}=\frac{a_{6}}{2\pi}=\frac{r_{+}}{2\pi
l\sqrt{r_{+}^{2}-r^{2}}\sin\theta}.$ (3.12)

Taking into account the redshift factor $\sqrt{-g_{00}}$ of the observer, one
has the Hawking temperature as

$T_{\rm HK}=\frac{r_{+}}{2\pi l}.$ (3.13)

On the other hand, one has another set of coordinates which covers the whole
region within the cosmological horizon,

$\displaystyle y_{0}=f\sinh(r_{+}t/l),\
~{}~{}~{}~{}~{}~{}y_{1}=f\cos\theta\cosh(r_{+}t/l),$ $\displaystyle
y_{2}=f\sin\theta\cos\chi\cosh(r_{+}t/l),\
~{}~{}~{}~{}~{}~{}y_{3}=f\sin\theta\sin\chi\cosh(r_{+}t/l).$ (3.14)

In this case, the line element is described by

$ds^{2}=l^{2}N^{2}\tilde{d\Omega_{3}^{2}}+N^{-2}dr^{2}+r^{2}d\phi^{2},$ (3.15)

where $N^{2}=(r_{+}^{2}-r^{2})/l^{2}$ and

$\tilde{d\Omega_{3}^{2}}=-dt^{2}+\frac{l^{2}}{r_{+}^{2}}\cosh^{2}(r_{+}t/l)(d\theta^{2}+\sin^{2}\theta
d\chi^{2}).$ (3.16)

We consider a static observer located at constant position of
$(r<r_{+},\chi,\varphi)$ and $\theta=0$. For such an observer, we have a
constant acceleration $a_{5}$ as

$a_{5}^{2}=\frac{r^{2}}{l^{2}(r^{2}_{+}-r^{2})}.$ (3.17)

In six dimensional Minkowski space, the acceleration $a_{6}$ associated with
the corresponding observer takes the form

$a_{6}^{-2}=x_{1}^{2}-x_{0}^{2}=\frac{l^{2}(r_{+}^{2}-r^{2})}{r_{+}^{2}}.$
(3.18)

We check that they obey the relation $a_{6}^{2}=a_{5}^{2}+1/l^{2}$. We
conclude that the observer has the Davies-Unruh temperature

$T_{\rm DU}=\frac{a_{6}}{2\pi}=\frac{r_{+}}{2\pi l\sqrt{r_{+}^{2}-r^{2}}}.$
(3.19)

Considering the redshift factor for the observer in the background (3.15), we
have the Hawking temperature observed as

$T_{\rm HK}=\frac{r_{+}}{2\pi l}.$ (3.20)

Consequently, we find the same Hawking temperature as (3.13) obtained when
using the coordinates (3.6).

##  4 Conclusions and Discussions

A $D$-dimensional CC black hole has unusual topological structure ${\cal
M}_{D-1}\times S^{1}$ and there is no globally timelike Killing vector in the
geometry of the black hole. Hence it was quite difficult to discuss
thermodynamic properties and Hawking temperature associated with this black
hole.

For example, Banados has considered a five dimensional rotating CC black hole
and embedded it into a Chern-Simons supergravity theory [3]. By computing
related conserved charges, it was shown that the black hole mass is
proportional to the product of outer horizon $r_{+}$ and inner horizon
$r_{-}$, while the angular momentum is proportional to the sum of two
horizons. In this case, the entropy of black hole is found to be proportional
not to the outer horizon $r_{+}$ but the inner horizon $r_{-}$. This approach
has two drawbacks. One is that the result cannot be degenerated to the non-
rotating case. The other is that it cannot be generalized to other dimensions.

Creighton and Mann have considered the quasilocal thermodynamics of a four
dimensional CC black hole in general relativity by computing thermodynamic
quantities at a finite boundary which encloses the black hole [5]. They have
shown that the entropy is not associated with the event horizon, but the
Killing horizon of a static observer which is tangent to the event horizon of
the black hole. The quasilocal energy density [see (11) of [5]] is negative.

In this work, we have derived Hawking temperature of CC black holes by
employing the globally embedding approach since these black holes can be
embedded into higher dimensional Minkowski space. We found that the Hawking
temperature of CC black holes is given by $r_{+}/2\pi l$ when using both
static and global coordinates. Here $r_{+}$ and $l$ are black hole horizon and
the radius of AdS space. Furthermore we found that the Hawking temperature is
also identical to the Gibbons-Hawking temperature of the boundary dS space of
the CC black holes. Importantly, we mention that the Hawking temperature
obtained in this work is the same as that obtained from semi-classical
tunneling method [19]. It turns out that the globally embedding technique is
powerful to determine the Hawking temperature of CC black hole without any
ambiguity. Using the same approach, we also obtained Hawking temperature of a
positive CC space which is counterpart of CC black hole in dS space.

Finally, we comment that those solutions including CC black holes and positive
CC spaces depend on an arbitrary real constant $r_{+}$. The $r_{+}$-dependence
can be made disappear by rescaling coordinates. In this case, the Hawking
temperature is given by $1/2\pi l$.

## Acknowledgments

This work was initiated during the APCTP joint focus program: frontiers of
black hole physics, Dec. 6-17, Pohang, Korea and Inje workshop on gravity and
numerical relativity during Dec. 16-18, Busan, Korea, the warm hospitality
extended to the authors in both places are grateful. RGC thanks S.P. Kim for
useful discussions during APCTP focus program. This work was supported in part
by a grant from Chinese Academy of Sciences and in part by the National
Natural Science Foundation of China under Grant Nos. 10821504, 10975168 and
11035008, and by the Ministry of Science and Technology of China under Grant
No. 2010CB833004. YSM was supported by the National Research Foundation of
Korea (NRF) grant funded by the Korea government (MEST) (No.2009-0062869).

## References

  * [1] M. Banados, C. Teitelboim and J. Zanelli, Phys. Rev. Lett. 69, 1849 (1992) [arXiv:hep-th/9204099]; M. Banados, M. Henneaux, C. Teitelboim and J. Zanelli, Phys. Rev. D 48, 1506 (1993) [arXiv:gr-qc/9302012].
  * [2] S. Holst and P. Peldan, Class. Quant. Grav. 14, 3433 (1997) [arXiv:gr-qc/9705067].
  * [3] M. Banados, Phys. Rev. D 57, 1068 (1998) [arXiv:gr-qc/9703040].
  * [4] M. Banados, A. Gomberoff and C. Martinez, Class. Quant. Grav. 15, 3575 (1998) [arXiv:hep-th/9805087].
  * [5] J. D. Creighton and R. B. Mann, Phys. Rev. D 58, 024013 (1998) [arXiv:gr-qc/9710042].
  * [6] S. F. Ross and G. Titchener, JHEP 0502, 021 (2005) [arXiv:hep-th/0411128].
  * [7] J. A. Hutasoit, S. P. Kumar and J. Rafferty, JHEP 0904, 063 (2009) [arXiv:0902.1658 [hep-th]].
  * [8] J. M. Maldacena, Adv. Theor. Math. Phys. 2, 231 (1998) [Int. J. Theor. Phys. 38, 1113 (1999)] [arXiv:hep-th/9711200]; S. S. Gubser, I. R. Klebanov and A. M. Polyakov, Phys. Lett. B 428, 105 (1998) [arXiv:hep-th/9802109]; E. Witten, Adv. Theor. Math. Phys. 2, 253 (1998) [arXiv:hep-th/9802150].
  * [9] R. G. Cai, Phys. Lett. B 544, 176 (2002) [arXiv:hep-th/0206223].
  * [10] D. Birmingham and M. Rinaldi, Phys. Lett. B 544, 316 (2002) [arXiv:hep-th/0205246].
  * [11] V. Balasubramanian and S. F. Ross, Phys. Rev. D 66, 086002 (2002) [arXiv:hep-th/0205290].
  * [12] G. W. Gibbons and S. W. Hawking, Phys. Rev. D 15, 2738 (1977).
  * [13] P. C. W. Davies, J. Phys. A 8, 609 (1975).
  * [14] W. G. Unruh, Phys. Rev. D 14, 870 (1976).
  * [15] H. Narnhofer, I. Peter and W. E. Thirring, Int. J. Mod. Phys. B 10, 1507 (1996).
  * [16] S. Deser and O. Levin, Class. Quant. Grav. 14, L163 (1997) [arXiv:gr-qc/9706018]; S. Deser and O. Levin, Class. Quant. Grav. 15, L85 (1998) [arXiv:hep-th/9806223].
  * [17] S. Deser and O. Levin, Phys. Rev. D 59, 064004 (1999) [arXiv:hep-th/9809159].
  * [18] Y. W. Kim, J. Choi and Y. J. Park, Int. J. Mod. Phys. A 25, 3107 (2010) [arXiv:0909.3176 [gr-qc]].
  * [19] A. Yale, arXiv:1012.2114 [gr-qc].
  * [20] R. G. Cai, Phys. Lett. B 552, 66 (2003) [arXiv:hep-th/0207053].
  * [21] S. Deser and R. Jackiw, Annals Phys. 153, 405 (1984); M. I. Park, Phys. Lett. B 440, 275 (1998) [arXiv:hep-th/9806119].


# Fine-Tuning Stable Diffusion XL for Stylistic Icon Generation: A Comparison
of Caption Size

Youssef Sultan1, Jiangqin Ma1, Yu-Ying Liao1  1 These authors contributed
equally to this work.  College of Engineering,
Georgia Institute of Technology,
801 Atlantic Dr NW, Atlanta, 30332, GA, USA
{ysultan, jma416<EMAIL_ADDRESS>
###### Abstract

In this paper, we show different fine-tuning methods for Stable Diffusion XL;
this includes inference steps, and caption customization for each image to
align with generating images in the style of a commercial 2D icon training
set. We also show how important it is to properly define what ”high-quality”
really is especially for a commercial-use environment. As generative AI models
continue to gain widespread acceptance and usage, there emerge many different
ways to optimize and evaluate them for various applications [1]. Specifically
text-to-image models, such as Stable Diffusion XL [2] and DALL-E 3 [3] require
distinct evaluation practices to effectively generate high-quality icons
according to a specific style. Although some images that are generated based
on a certain style may have a lower FID score (better), we show how this is
not absolute in and of itself even for rasterized icons. While FID [5][6]
scores reflect the similarity of generated images to the overall training set,
CLIP [4] scores measure the alignment between generated images and their
textual descriptions. We show how FID scores miss significant aspects, such as
the minority of pixel differences that matter most in an icon, while CLIP
scores result in misjudging the quality of icons. The CLIP model’s
understanding of ”similarity” is shaped by its own training data; which does
not account for feature variation in our style of choice. Our findings
highlight the need for specialized evaluation metrics and fine-tuning
approaches when generating high-quality commercial icons, potentially leading
to more effective and tailored applications of text-to-image models in
professional design contexts.

###### Index Terms:

generative AI models, text-to-image models, icon generation, style-specific
image generation

##  I Introduction

Diffusion models have shifted the field of machine learning by enabling the
creation of supposed high-fidelity synthetic data. They work by iteratively
denoising data through a stochastic process, then transforming that random
noise into coherent and realistic outputs [7]. Denoising Diffusion
Probabilistic Models (DDPMs) model the data distribution with precision
through a forward and reverse process. Within the forward process, Gaussian
noise is added to the data, mapping it to a noise distribution. However, the
reverse process learns how to denoise this distribution. The results are a
reconstruction of the original data distribution, but in a different form.
This different form outlines the generative aspect of the model, allowing it
to capture intricate details and produce something different yet coherent and
in some cases realistic.

The main caveat with these models unfortunately is the efficiency, training
time and resources needed in order to train or fine-tune them tailored to a
specific use-case. This has led to additional works further expanding into the
field of efficiency and efficacy where techniques like weight quantization in
”BitsFusion” [8] have significantly reduced these requirements while
maintaining performance. Advancements of diffusion models for tasks tailored
to commercial or user-driven use-cases have been a critical area of research.
”DreamBooth” [9] and ”StyleDrop” [11] modify text-to-image diffusion models to
generate content that is aligned with a specific style or aesthetic,
highlighting an easier route for such use-cases.

Additionally, high-resolution image synthesis as discussed by Rombach et al.
[10] provides a great vantage point of latent diffusion models (LDMs) in
producing detailed and high-quality images. LDMs work in a compressed latent
space as opposed to directly in pixel space, which is better for efficiency,
model training and inference (image-generation). Despite such compression,
they discard noise and redundant information within the image, and capture
essential data characteristics through variational autoencoders [13].

The works of Meng et al. in ”SDEdit” [12] also introduce guided image
synthesis and editing using stochastic differential equations, adding a new
barometer to the playing field. By manipulating the trajectory of the image
generation process through user-defined synthesis criteria, it allows for a
balance between deterministic control and stochastic control at inference
time. This of course translates to the tailoring of desired image
characteristics whether it be specific styles, textures, color schemes or
structural elements of the image.

Stable Diffusion XL [2], also a latent diffusion model, leverages more
parameters in its architecture and offers improved fine-tuning capabilities.
The key advancement in Stable Diffusion XL is its refinement model, which is
employed at the end to enhance the visual fidelity of the generated image. Its
ability to adhere to specific visual styles make it particularly well-suited
for tasks like stylistic icon generation. This characteristic, combined with
its advanced architecture, makes Stable Diffusion XL an ideal candidate for
our study on generating high-quality commercial icons.

##  II Methods

In this section we detail the approaches and techniques used to fine-tune
Stable Diffusion XL in order to generate icons pertaining to a specific preset
style for commercial use. We explore various aspects including inference
steps, and the integration of different sized captions to align generated
images with our specific icon style. Additionally, we compare the outcomes
using the same captions with DALL-E 3 to understand the CLIP and FID score
comparison which also shows how these metrics fluctuate relatively to the
actual quality of the icons assessed by a human.

Figure 1: Distribution of file types for icon images related to kitchen
cabinets, kitchens, and screws in the provided commercial data.

We utilized a dataset from The Home Depot containing images of various icon
types categorized by their descriptions within the file name. For example one
image icon may be a ”phillips head screw” with an icon of a 2D phillips head
screw. The focus here was to distinguish screws and kitchen cabinets for our
experiments, so we cleaned the data in order to filter out any unrelated
images to those two categories. This left us with 42 icon images related to
screws and 17 icon images related to kitchen cabinets. The overall
distribution of the file types regarding these images can be seen in Figure 1.

Training these types of models requires an image paired with a caption. In the
phillips head screw example, our goal is to have a caption that best describes
what this image represents. Therefore, for every image, we have manually
defined a custom prompt for that image in a metadata.jsonl file. The purpose
of this is to fine-tune the model, where ”fine-tuning” is essentially an
adjustment of the model weights to get more of a match between the generated
image and our original image (icon in the training set). From a high-level
overview, the sequence of the fine-tuning can be interpreted in Figure 2.

Figure 2: A visual representation of how the model learns to generate images according to a style TABLE I: Training Parameters for the Models mixed_precision | fp16
---|---
resolution | 1024
train_batch_size | 1
gradient_accumulation_steps | 3
gradient_checkpointing | True
learning_rate | 1e-4
snr_gamma | 5.0
lr_scheduler | constant
lr_warmup_steps | 0
use_8bit_adam | True
max_train_steps | 500
checkpointing_steps | 717
seed | 0

For screw icons, we trained three versions of Stable Diffusion XL using the
same set of images but with different prompts to see the variation in icon
quality, FID score and CLIP score. The model variants were short prompts with
class images, short prompts and long prompts. The short prompt with class
images leveraged short keywords and 20 high-quality Home Depot head-type icon
images. The short prompt contained short keyword prompts whereas the long
prompt contained detailed descriptions of the training dataset images. A
fourth model used for comparison purposes was the DALL-E 3, which was accessed
via API to generate images using the same inference prompt. The purpose of
including DALL-E 3 was to understand performance metric variation relative to
the quality by human assessment. The parameters used for the three Stable
Diffusion XL models can be seen in Table I.

Prior-preservation loss is effective in encouraging output diversity and in
overcoming language-drift in ”DreamBooth” [9]. Therefore, we add class-
specific prior preservation loss to the Stable Diffusion XL model with short
prompt during training using class images. This approach requires additional
class images and careful balancing between instance-specific and class-generic
images. By default, the prior_loss_weight is set to 1. This parameter controls
the weight of the prior preservation loss during training, which helps to
preserve the generalization capabilities of the model by penalizing deviations
from the class-specific distribution. Class images are a set of reference
images used to guide the training process of a machine learning model. These
images are used to help the model understand the style and characteristics of
the icons we want to generate.

Our results show that using the default prior_loss_weight did not increase
performance significantly, since training with class images has better FID
score, but worse CLIP score. Consequently, we needed to experiment with this
value to find the optimal setting. This approach requires significant
computational resources and time to find the optimal prior_loss_weight value
and class images. We tried several different class images but were unable to
improve performance. Due to limited computational resources and time
constraints, we did not continue pursuing this approach.

To reproduce our results, we also conduct this on a public icon dataset
including 42 images. The public dataset includes icons downloaded from online
resources, primarily using Google search, ChatGPT [14], and Gemini [15]. The
prompts used for training are also the same as those used for The Home Depot
dataset, as we use the same head type and drive style.

For each model we generate 10 images and use the associated prompts to
calculate the average CLIP score. To obtain the FID score differences between
each model the same was done for 7 icons with their corresponding generations.

For kitchen cabinet icons, we also trained three versions of Stable Diffusion
XL using the same set of images but with different prompts to see the
variation in icon quality, FID score and CLIP score. In terms of training
data, since there are a limited number of kitchen cabinet icons provided by
The Home Depot, we gather multiple free kitchen icons from the internet in a
similar manner to our screws methodology which brings our final dataset to be
42 icons. In terms of class images, we leverage 100 icons of appliances and
furniture from Home Depot, all with the style we’re hoping to generate.

In terms of model variants, as mentioned before there are long prompts with
class images, short prompts, and long prompts. Long prompts follow the format
of ”a photo of TOK kitchen cabinet icon, $<$style$>$, Icon of a $<$cabinet
type$>$ kitchen cabinet in $<$color$>$ with $<$detail description of the
cabinet type or corner cabinet type$>$”, while short prompts follow the same
order of long prompts but with keywords, for example ”$<$style$>$, $<$cabinet
type$>$, $<$color$>$, $<$description $>$”. Finally, as a comparison we also
leveraged DALL-E 3 to generate images using the same inference prompt for both
short and long prompts. Throughout our experiments, we used ’TOK’ as a unique
identifier for cabinets in our prompts. This identifier acts as a consistent
reference point for the model. Additionally, we used the syntax $<$variable$>$
to denote elements in the prompt that we wanted the model to learn and vary.
For example, a prompt might include $<$color$>$ or $<$cabinet type$>$,
allowing the model to generate diverse cabinet icons by learning to substitute
different colors or cabinet types. This approach, inspired by [9], enables the
creation of novel, photo-realistic images of the subject, contextualized in
diverse scenes.

For each model after fine-tuning we generated 40 images using 10 different
prompts (inference of 4 for each prompt). The associated prompt used for icon
generation was saved in separate txt files. In addition, we selected the best
image(s) per prompt and calculated the average CLIP score for each model using
the txt files with prompt and corresponding image. This leaves us with 3 icons
from our training set to compare with 3 generated images to then calculate the
average FID score. Since there were not a lot of kitchen cabinet icons from
The Home Depot dataset, we could only calculate the FID on a basis of 3
images.

##  III Results

###  III-A Models trained on commercial data for screws

Figure 3: FID and CLIP scores of screw icons generated using commercial icons
for training (non-public data)

Using the model trained on commercial data with longer captions, shorter
captions, and class images for screw icon generation, the FID and CLIP scores
can be viewed in Figure 3. Here it can be seen that the short prompt with
class images performs the best in terms of CLIP and FID.

(a) Hex head screw icon

(b) Round head screw icon

(c) Phillips drive oval head screw icon

(d) Spanner drive pan head screw icon

Figure 4: Example generated icons based on commercial training + long prompts
(each caption above is what was used at inference time)

A sample of 4 generated images along with the inference prompts passed on to
the fine-tuned model can be seen in Figure 4. As mentioned previously, the FID
score is based on an average of 7 images, while the CLIP score is based on an
average of 10 images.

(a) Hex head screw icon

(b) Round head screw icon

(c) Phillips drive oval head screw icon

(d) Spanner drive pan head screw icon

Figure 5: Example generated icons based on commercial training + short prompts
(each caption above is what was used at inference time)

For the model trained on short prompts, it can be seen that better performance
metrics accurately correlate with the human assessment of better screws. That
is, the hex screw icon with a shorter prompt in Figure 5 generated looks more
coherent as opposed to the generated image in the long prompt.

(a) Hex head screw icon

(b) Round head screw icon

(c) Phillips drive oval head screw icon

(d) Spanner drive pan head screw icon

Figure 6: Example generated icons based on commercial training + short prompt
class images (each caption above is what was used at inference time)

For the model trained with short prompt class images, this is the model with
the best FID score objectively as identified previously in Figure 3. Although
this model does take the lead for the best FID score, in the context of icon
generation, we can see that these icons are not the best from the human
assessment perspective. Specifically, the oval head screw icon (c) and the
spanner drive pan head screw icon (d) do not look coherent at all. This
substantiates the fact that although the FID score may be better, it does not
necessarily mean the icon generated at inference time is representative of the
expected outcome. This is due to the fact that not all of the pixels can be
captured in reference to the original image which are of importance when it
comes to quality. In this case we can say that the short prompt icon
generations in Figure 5 are more representative of the outcome we are looking
for, despite having worse FID.

Figure 7: An example of an image with a high clip score (0.30) but incorrect
icon generation. (inference: ”cross drive round head screw icon”)

Another interesting point is that out of the short prompts which curated the
highest clip scores out of all (0.30) it is evident that just because clip
scores are high, does not mean the expected outcome is correct. In Figure 7 we
can see that this image has a high clip score in comparison to the other
images which are $\leq 0.299$. However, its generation is incorrect. This is
because despite the inference prompt to generate the image ”cross drive round
head screw icon”, it generates a screw that is not a cross drive round head,
but rather a different head. This substantiates the fact CLIP scores are
relative to the training data the CLIP model is based off of, and it is not
always able to generalize to nuanced use-cases, such as stylistic icon
generation. This further substantiates why human-assessment of use-cases such
as these is paramount.

As a cross comparison, the same prompts are used in DALL-E 3 to understand the
FID score and CLIP score as well as look at the results without any training.
The scores and generated images can be viewed in the Appendix in Figure 12.
DALL-E 3 shows the highest average CLIP score and the lowest FID score
calculated based on the commercial training set for screws.

###  III-B Models trained on commercial data for Kitchen Cabinets

Figure 8: FID and CLIP scores of kitchen cabinet icons generated using
commercial icons for training (non-public data)

Using the model trained on commercial data with longer captions and shorter
captions for kitchen cabinet icon generation, the FID and CLIP scores can be
viewed in Figure 8. Here it can be seen that the short prompt images perform
the best in terms of CLIP and FID. The short prompt with class images was not
conducted for kitchen cabinets because it was highly computational, and the
performance with the long prompt was worse than not using the class.
Therefore, it was not included in this experiment.

(a) Short Prompt: $<$style: 2D icon$>$, $<$cabinet type: wall$>$, $<$color:
gray$>$

(b) Long Prompt: a photo of TOK kitchen cabinet icon, $<$style: 2D icon$>$, a
wall kitchen cabinet in grey with two doors

(c) DALL-E 3 short prompt: $<$style: 2D icon$>$, $<$cabinet type: wall$>$,
$<$color: gray$>$

Figure 9: Wall Kitchen Cabinet in Grey

Similar to the findings from screw icons, the short prompt trained model has a
higher CLIP score. However, when looking at the FID score, the short and long
prompt have very similar FID scores as seen in Figure 8. DALL-E 3 has a better
CLIP score (see Figure 13), but this does not necessarily mean that the icon
is what we are looking for, which makes sense. Although the short prompt
trained model has better performance than the long prompt model, when looking
at CLIP and FID scores, there are cases where long prompt generated icons are
visually better and more customized based on the description in the prompt.
This can be seen in Figure 10(b) and Figure 16(b). This further substantiates
the fact that a high CLIP score is not a direct measure of whether a generated
icon is in the correct style.

(a) Short Prompt: $\langle style:\ 3D\ icon\rangle$, $\langle cabinet\ type:\
corner\rangle$, $\langle color:\ red\rangle$

(b) Long Prompt: a photo of TOK kitchen cabinet icon, $\langle style:\ 3D\
icon\rangle$, a corner kitchen cabinet in red with doors at the bottom

(c) DALL-E 3 short prompt:$\langle style:\ 3D\ icon\rangle$, $\langle cabinet\
type:\ corner\rangle$, $\langle color:\ red\rangle$

Figure 10: Corner Kitchen Cabinet in Red

Additionally, when comparing results from Stable Diffusion XL with DALL-E 3,
we observe a noticeable difference in style. This highlights the importance of
training data in generating customized and consistent styles. To explore this
further, we trained several models with a combination of icons and realistic
product photos as the training data and included style variable in the caption
jsonl file. The results show that incorporating even a few realistic product
photos can significantly skew the generated icon style to match that of the
realistic photos. This can be seen in Figures 11(c), 11(b), 11(a).

(a) Long Prompt: A photo of TOK kitchen cabinet icon, photo of a standard wall
kitchen cabinet in white with two doors

(b) Long Prompt: A photo of TOK kitchen cabinet icon, photo of a standard
kitchen cabinet in blue with pull out drawers and sink on top

(c) Long prompt: A photo of TOK kitchen cabinet icon, photo of a trash can
kitchen cabinet in unfinished color with a pull-out trash bin

Figure 11: Kitchen Cabinet with realistic product photos as Training Data

##  IV Discussion

###  IV-A Key Findings

In this study, we fine-tuned Stable Diffusion XL with different training data
and caption sizes to generate stylistic icons for a commercial environment.
Our focus was on how different training prompts and evaluation metrics impact
the quality and relevance of generated icons. We found that short prompts with
class images yielded the best FID scores but often failed to meet qualitative
expectations when assessed by humans. Long prompts, though slightly higher in
FID scores, produced icons that better aligned with the intended style
surprisingly in some cases. This highlights the importance of human assessment
in evaluating the true quality of generated icons. While FID scores measure
overall similarity between generated images and the training set, they miss
significant aspects such as detailed stylistic features. High CLIP scores were
found to not always correlate with the expected visual outcome based on a
generated image of the wrong screw that had the highest CLIP score out of all.
Also, we observed that DALL-E 3 generally had higher average CLIP scores and
FID scores. However, DALL-E 3’s generated icons were often less customized to
the specific styles we aimed for. Fine-tuning Stable Diffusion XL with
detailed prompts showed better consistency and accuracy in style sometimes for
short and sometimes for long captions with varying degree. Our experiments
also revealed that incorporating realistic product photos in the training set
skewed the generated icons towards a more realistic style, which was not
desirable for our use case of 2D stylistic icons.

###  IV-B Limitations

Our study utilized relatively small datasets - 42 screw icons and 42 kitchen
cabinet icons. This limited sample size may not fully represent the diversity
of icons in these categories, potentially affecting the generalizability of
our findings. We focused on only two types of icons (screws and kitchen
cabinets). While these provided interesting contrasts, they represent a narrow
slice of the vast array of icon types used in commercial settings. Our
findings may not apply equally to all icon categories. Due to limited
computational resources, we were unable to fully explore certain aspects of
the model, such as finding the optimal prior_loss_weight for class images.
Additionally, while we used FID and CLIP scores as evaluation metrics, these
have limitations in capturing the nuanced stylistic elements crucial for icon
quality.

###  IV-C Implications and Future Directions

Our findings have significant implications for the commercial application of
AI-generated imagery, particularly in contexts where style consistency is
crucial. The discrepancy between quantitative metrics and human assessment
underscores the need for more sophisticated evaluation methods in AI-generated
art.

We conclude that CLIP score and FID score are insufficient alone for icon
generation. Tailored evaluation criteria based on specific business use-cases,
supplemented by human assessments, are crucial for ensuring the generated
icons meet the desired quality and style. Future work should focus on
developing refined evaluation metrics that better capture stylistic nuances
and quality requirements from an automation perspective. For example,
exploring larger and more diverse datasets to improve the generalizability of
the fine-tuning process, studying the balance between stylized and realistic
training data and determining optimal training set compositions for different
icon styles and use cases.

##  V Appendix

###  V-A Additional DALL-E 3 Screw Prompt Comparisons

(a) Hex head screw icon

(b) Round head screw icon

(c) Phillips drive oval head screw icon

(d) Spanner drive pan head screw icon

Figure 12: Screw: Inference prompt generations using DALL-E 3, FID Score: 340,
CLIP Score: 0.32

###  V-B Additional Kitchen Cabinet Prompt Comparisons

Figure 13: FID and CLIP scores of kitchen icons generated using DALL-E 3

(a) Short Prompt: $<$style: 2D icon$>$, $<$cabinet type: single oven$>$,
$<$color: green$>$

(b) Long Prompt: a photo of TOK kitchen cabinet icon, $<$style: 2D icon$>$, an
oven kitchen cabinet in green with one drawer

(c) DALL-E 3 short prompt: $<$style: 2D icon$>$, $<$cabinet type: single
oven$>$, $<$color: green$>$

Figure 14: Single Oven Kitchen Cabinet in Green

(a) Short Prompt: $\langle style:\ 3D\ icon\rangle$, $\langle cabinet\ type:\
sink\ base\rangle$, $\langle color:\ beige\rangle$

(b) Long Prompt: a photo of TOK kitchen cabinet icon, $\langle style:\ 3D\
icon\rangle$, a sink base kitchen cabinet in beige with a sink on top and two
doors at the bottom

(c) DALL-E 3 short prompt:$\langle style:\ 3D\ icon\rangle$, $\langle cabinet\
type:\ sink\ base\rangle$, $\langle color:\ beige\rangle$

Figure 15: Sink Base Kitchen Cabinet in Beige

(a) Short Prompt: $\langle style:\ 3D\ icon\rangle$, $\langle cabinet\ type:\
drawer\ base\rangle$, $\langle color:\ black\rangle$

(b) Long Prompt: a photo of TOK kitchen cabinet icon, $\langle style:\ 3D\
icon\rangle$, a drawer base kitchen cabinet in black with three drawers

(c) DALL-E 3 short prompt:$\langle style:\ 3D\ icon\rangle$, $\langle cabinet\
type:\ drawer\ base\rangle$, $\langle color:\ black\rangle$

Figure 16: Drawer Base Kitchen Cabinet in Black

(a) Short Prompt: $\langle style:\ 3D\ icon\rangle$, $\langle cabinet\ type:\
base\rangle$, $\langle color:\ brown\rangle$, $\langle description:\ pull-out\
trash\ can\rangle$

(b) Long Prompt: a photo of TOK kitchen cabinet icon, $\langle style:\ 3D\
icon\rangle$, a kitchen cabinet in brown with pull-out trash can

(c) DALL-E 3 short prompt:$\langle style:\ 3D\ icon\rangle$, $\langle cabinet\
type:\ base\rangle$, $\langle color:\ brown\rangle$, $\langle description:\
pull-out\ trash\ can\rangle$

Figure 17: Trash Can Kitchen Cabinet in Brown

###  V-C Models trained on non-commercial data for Screws

(a) Hex head screw icon

(b) Round head screw icon

(c) Phillips drive oval head screw icon

(d) Spanner drive pan head screw icon

Figure 18: Example generated icons based on public training data using long
prompts

(a) Hex head screw icon

(b) Round head screw icon

(c) Phillips drive oval head screw icon

(d) Spanner drive pan head screw icon

Figure 19: Example generated icons based on public training data using short
prompts

Based on the public dataset, the FID and CLIP scores were analyzed for both
short and long prompts. The FID results show that the short prompt achieved a
lower score of 227.69 compared to the long prompt’s score of 263.48,
indicating that the short prompt is better in terms of image quality and
diversity. On the other hand, the CLIP scores were very close, with the short
prompt scoring an average of 0.3214 and the long prompt scoring slightly
higher at 0.3280. This suggests that while the short prompt performs better
for FID, the long prompt has a marginally better alignment with the textual
descriptions according to the CLIP score.

###  V-D An analysis of inference steps and file-name keyword only training

When testing the inference query on several different types of steps, it can
be evident than an optimal point is required to find the best parameter value.
As we increased the inference steps on a basic trained version of the model
(sub-optimal) using keywords only for training, we found the image icons to
get better as the inference steps increased. However there is a point where
the inference steps being too high could cause a bad quality icon. The
reasoning for the imperfections here ais due to smaller trained captions,
smaller than our short captions with only file-name parsed keywords (see
Figure 21).

When training the Stable Diffusion XL model on only keywords parsed from the
file-names themselves, such as ”Phillips head” or ”hex screw”, the results are
not as good, however are still somewhat coherent. This tells us that without a
unique identifier such as our use of ”TOK”, it is still possible to get the
shape correctly for some cases, however some cases provide non-coherent icons
(see Figure 20).

Figure 20: Various screw images trained on only relevant keywords from file
names

Figure 21: Testing different inference steps on keyword caption trained images

## References

  * [1] L. Theis, A. van den Oord, and M. Bethge, “A note on the evaluation of generative models,” arXiv preprint arXiv:1511.01844, 2016. [Online]. Available: https://arxiv.org/abs/1511.01844.
  * [2] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, “SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis,” arXiv preprint arXiv:2307.01952, 2023. [Online]. Available: https://arxiv.org/abs/2307.01952.
  * [3] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, W. Manassra, P. Dhariwal, C. Chu, Y. Jiao, and A. Ramesh, “Improving Image Generation with Better Captions,” OpenAI, 2023. [Online]. Available: https://cdn.openai.com/papers/dall-e-3.pdf.
  * [4] S. Jayasumana, S. Ramalingam, A. Veit, D. Glasner, A. Chakrabarti, and S. Kumar, “Rethinking FID: Towards a Better Evaluation Metric for Image Generation,” arXiv preprint arXiv:2401.09603, 2024. [Online]. Available: https://arxiv.org/abs/2401.09603.
  * [5] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer, and S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium,” CoRR, vol. abs/1706.08500, 2017. [Online]. Available: http://arxiv.org/abs/1706.08500.
  * [6] M. Seitzer, “pytorch-fid: FID Score for PyTorch,” GitHub, version 0.3.0, August 2020. [Online]. Available: https://github.com/mseitzer/pytorch-fid.
  * [7] J. Ho, A. Jain, and P. Abbeel, “Denoising Diffusion Probabilistic Models,” in Advances in Neural Information Processing Systems, vol. 33, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, Eds. Curran Associates, Inc., 2020, pp. 6840-6851. [Online]. Available: shortened link.
  * [8] Y. Sui, Y. Li, A. Kag, Y. Idelbayev, J. Cao, J. Hu, D. Sagar, B. Yuan, S. Tulyakov, and J. Ren, “BitsFusion: 1.99 bits Weight Quantization of Diffusion Model,” arXiv preprint arXiv:2406.04333, 2024. [Online]. Available: https://arxiv.org/abs/2406.04333.
  * [9] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, “DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation,” arXiv preprint arXiv:2208.12242, 2023. [Online]. Available: https://arxiv.org/abs/2208.12242.
  * [10] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” arXiv preprint arXiv:2112.10752, 2022. [Online]. Available: https://arxiv.org/abs/2112.10752.
  * [11] K. Sohn, N. Ruiz, K. Lee, D. C. Chin, I. Blok, H. Chang, J. Barber, L. Jiang, G. Entis, Y. Li, Y. Hao, I. Essa, M. Rubinstein, and D. Krishnan, “StyleDrop: Text-to-Image Generation in Any Style,” arXiv preprint arXiv:2306.00983, 2023. [Online]. Available: https://arxiv.org/abs/2306.00983.
  * [12] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon, “SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations,” arXiv preprint arXiv:2108.01073, 2022. [Online]. Available: https://arxiv.org/abs/2108.01073.
  * [13] D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” arXiv preprint arXiv:1312.6114, 2022. [Online]. Available: https://arxiv.org/abs/1312.6114.
  * [14] OpenAI, “ChatGPT: OpenAI’s Language Model,” 2023. [Online]. Available: https://chat.openai.com.
  * [15] Google, “Gemini: Google’s Image Generation Model,” 2023. [Online]. Available: https://gemini.google.com.