# Content-Based Search for Deep Generative Models

Daohan Lu1 Sheng-Yu Wang111footnotemark: 1 Nupur Kumari111footnotemark: 1
Rohan Agarwal111footnotemark: 1 David Bau2 Jun-Yan Zhu1
1Carnegie Mellon University 2Northeastern University empty

###### Abstract

The growing proliferation of pretrained generative models has made it
infeasible for a user to be fully cognizant of every model in existence. To
address this need, we introduce the task of _content-based model search_ :
given a query and a large set of generative models, find the models that best
match the query. Because each generative model produces a distribution of
images, we formulate the search problem as an optimization to maximize the
probability of generating a query match given a model. We develop
approximations to make this problem tractable when the query is an image, a
sketch, a text description, another generative model, or a combination of the
above. We benchmark our method in both accuracy and speed over a set of
generative models. We demonstrate that our model search retrieves suitable
models for image editing and reconstruction, few-shot transfer learning, and
latent space interpolation. Finally, we deploy our search algorithm to our
online generative model-sharing platform at https://modelverse.cs.cmu.edu/.

Figure 1: We develop a content-based search engine for Modelverse, a model-
sharing platform that contains a diverse set of deep generative models, such
as animals, landscapes, portraits, and art pieces. From left to right, our
search algorithm enables queries (1st row) with four different modalities –
text, images, sketches, and existing models. The 2nd and 3rd rows show the two
top-ranked models. The color of each model icon implies the model type. Our
method finds relevant models with similar semantic concepts in all modalities.

##  1 Introduction

We introduce the task of content-based model search, which aims to find the
most relevant deep image generative models that satisfy a user’s input query.
For example, as shown in Figure 1, we enable a user to retrieve a model either
based on its ability to synthesize images that match an image query (e.g., a
landscape photo), text query (e.g., African animals), sketch query (e.g., a
sketch of a standing cat), or its similarity to a given query model.††*
indicates equal contribution

But why is content-based model search useful? We believe that model search is
desperately needed to handle a burgeoning proliferation of generative models:
no longer being merely outputs of scientific study, deep generative models are
being created as backbones for content creation applications and software [71,
119, 11], as pre-trained models for computer vision and robotics research [39,
19], and as works of art that explore a wide range of themes [29, 40]. Each
model captures a small universe of curated subjects, which can range from
realistic rendering of faces and landscapes [56] to photos of historic pottery
[5] to cartoon caricatures [50] to single-artist stylistic elements [108].
More recently, various methods enable creative modifications and
personalization of existing models, via human-in-the-loop interfaces [8, 34,
122, 123] or fine-tuning of GANs [125, 83, 135] and text-to-image models [101,
33]. Each generative model can represent a substantial investment in a
specific idea of the model creator.

As the number of generative models grows, it is becoming increasingly
infeasible for a user to know about every interesting model, and yet it can be
crucial to choose the right model for their specific use. Each generative
model allows a user to easily synthesize an unbounded set of images,
interpolations, or latent variable manipulations, but we have found that
choosing the right generative model out of a large collection can yield
results that are far better than picking a mismatched model (Section 5). Just
as information and image retrieval allow users to find the right information
within vast collections of traditional content, model search enables users to
find a model that best fits their particular needs.

Content-based model search is a challenging task: even the simplified question
of whether a specific image can be produced by a single model can be
computationally difficult. Unfortunately, many deep generative models do not
offer an efficient or exact way to estimate density, nor do they natively
support assessing cross-modal similarity (e.g., text and image). A naive Monte
Carlo approach can compare the input query to thousands or even millions of
samples from each generative model, and identify the model whose samples most
often match the input query. Such a sampling-based approach would make model
search extremely slow.

To address the above challenges, we first present a general probabilistic
formulation of the model search problem and present a Monte Carlo baseline. To
reduce the search time and storage, we “compress” the model’s distribution
into pre-computed 1st and 2nd order moments of the deep feature embeddings of
the original samples. We then derive closed-form solutions for model retrieval
given an input image, text, sketch, or model query. Our final formula can be
computed in real-time.

We evaluate our algorithms and perform ablation studies on 133 deep generative
models such as GANs (e.g., StyleGAN-family models [58]), diffusion models
(e.g., DDPM [44]), and auto-regressive models (e.g., VQGAN [30]). Compared to
the Monte-Carlo baseline, our method enables much more efficient search
(within 0.08 milliseconds, a 5x speedup), while maintaining high accuracy.
Finally, we demonstrate applications of model search, including few-shot model
fine-tuning [125] and GAN inversion [138].

To our knowledge, our method presents the first content-based search algorithm
for machine learning models. We deploy the search algorithm to Modelverse, our
online platform for researchers, students, and artists to easily use and share
generative models at https://modelverse.cs.cmu.edu/. Please visit our project
page and GitHub for more details.

##  2 Related Works

Deep generative models. Generative models are open-sourced at an unprecedented
rate of hundreds per month. They use different learning objectives [60, 36,
114, 85, 25, 48, 44, 117], training techniques [63, 54, 55, 79, 106, 100], and
network architectures [97, 13, 57, 30]. They are also trained on different
datasets [79, 108, 129, 20] for different applications [39, 90, 133, 139, 65,
2, 103, 96]. This trend leads to the following question. Among all the models,
which one shall we use for new tasks and domains? Our goal is _not_ to
introduce a new model. Instead, we want to help researchers, students, and
artists find existing models more quickly.

Image editing with generative models. Generative models enable various image
editing capacities, thanks to the learned disentangled representations. For
example, GAN-based editing methods invert an image to the latent space [1,
121, 99, 138, 14, 9], and edit the inverted image by modifying the latent code
[109, 49, 89, 68, 137, 72]. Diffusion models [44, 115] transform a user edit
(e.g., brushstrokes) to be realistic-looking by adding noise and de-noising
[76, 96]. Generative transformers [30] can be applied to edit images with
text-guidance [21] or user-defined masks [18]. Besides image editing, several
works enable users to edit and customize a generative model. By modifying the
network weights directly, we can easily update a pre-trained GAN with simple
user interfaces, including sketching [122], warping [123], blending [8],
adding/removing certain objects [10], and providing text prompts [34, 59].
With the increasing number of generative models, it becomes essential to find
the best model to apply image editing. Moreover, the need for a model search
system has increased, as a potentially large number of new models can be
created by model editing algorithms.

Content-based retrieval. Building upon classical information retrieval [7,
75], content-based retrieval deals with queries over image, video, or other
media [38, 23, 45]. Content-based image retrieval methods use robust visual
descriptors [84, 73, 22] to match objects within video or images [111, 4].
Methods have been developed to compress visual features to scale retrieval to
very large collections [120, 126, 51, 35, 104], and deep learning has enabled
compact vector representations for retrieval [120, 62, 6, 136]. When an image
is not available, a sketch can serve as a query; sketch-based retrieval has
been studied using traditional feature descriptors [28, 15, 110, 67] as well
as deep learning methods [130, 105, 70, 94, 98]. There has also been interest
in joint visual-language embeddings [32, 113, 53, 31, 52, 95] that enable text
queries for image content. We also adopt deep image representations for our
setting, but unlike single-image retrieval, we index distributions of images
that cannot be fully materialized.

Transfer learning for generative models. Transfer learning aims to adapt
models to unseen domains and tasks [47, 61, 87, 102, 128, 131]. For generative
models like GANs and diffusion models, several works have proposed finetuning
a pre-trained network to enable image generation in an unseen limited-data
domain [125, 77, 81, 134, 66, 82, 124, 101, 33]. Various works have explored
model selection to choose pre-trained models for discriminative tasks [93, 80,
27, 12] or selecting pre-trained discriminators for training GANs [63].
Similarly, the source of pre-trained generators plays a critical role in GAN
finetuning according to recent studies [37, 82]. We show that content-based
model search can be used to automatically select pre-trained generators for a
new domain, and improve the efficiency of model finetuning.

Figure 2: Method Overview. Our search system consists of a pre-caching stage
(a, b) and an inference stage (c). Given a collection of models
$\theta\sim\text{unif}\\{\theta_{1},\theta_{2},\dots,\theta_{N}\\}$, (a) we
first generate 50K samples for each model $\theta$. (b) We then encode the
images into image features and compute the 1st and 2nd order feature
statistics for each model. The statistics are cached in our system for
efficiency. (c) At inference time, we support queries of different modalities
(text, image, or sketch). We encode the query into a feature vector and assess
the similarity between the query feature and each model’s statistics. The
models with the best similarity measures are retrieved. More details of our
algorithm are in Section 3.

##  3 Methods

We aim to build a search/retrieval system for deep generative models. When a
user specifies an image, sketch, or text query, we would like to retrieve a
model that best matches the query. In this paper we shall focus our attention
on unconditional generative models trained on image collections: we are
interested in this starting point because the user community has created a
growing proliferation of this class of models [92]. We denote the model
collection by
$\theta\sim\text{unif}\\{\theta_{1},\theta_{2},\dots,\theta_{N}\\}$ and the
user query by $q$. Every model $\theta$ captures a distribution of images
$p(x|\theta)$. Here $x$ are images output from a generative model. Since prior
retrieval methods [112, 75] search for single instances, our key challenge is
to establish the notion of retrieving probabilistic distributions.

To achieve this, we introduce a probabilistic formulation for generative model
retrieval. Our formulation is general to different query modalities and
various types of generative models, and can be extended to different
algorithms. In Section 3.1, we derive our model retrieval formulation based on
a Maximum Likelihood Estimation (MLE) objective, and we present our model
retrieval algorithms for an image, a text, and a sketch query, respectively.
In Section 3.2, we demonstrate several extensions and applications of our
search system, including finding similar models, editing real images, and
fine-tuning GANs with a few images. In Section 3.3, we present a user
interface built upon our algorithms.

###  3.1 Probabilistic Retrieval for Generative Models

Our goal is to quantify the likelihood of each model $\theta$ given the user
query $q$, by evaluating the conditional probability $p(\theta|q)$. The model
with the highest conditional probability is retrieved:

$\max_{\theta\in\\{\theta_{1},\dots,\theta_{N}\\}}p(\theta|q)=\frac{p(q|\theta)p(\theta)}{p(q)}\propto
p(q|\theta).$ (1)

In Equation 1, since we assume the model collection is uniformly distributed,
by Bayes’ rule it suffices to find the likelihood of the query $q$ given each
model $\theta$ instead.

There are two scenarios to infer the conditional probability $p(q|\theta)$.
(1) When $q$ shares the same modality with $\theta$ (e.g., searching image
generative models with image queries), we directly reduce the problem to
estimating generative model’s density. (2) When $q$ has a different modality
to $\theta$ (e.g., searching image generative models with text queries), we
take account of cross-modal similarity to estimate $p(q|\theta)$. We discuss
the two cases in the following.

Image-based model retrieval. Given an image query $q$, we directly estimate
the likelihood of the query $p(q|\theta)$ from each model. In other words, the
best-matched model is the one most likely to generate the query image. Since
density is intractable or inaccurate for many generative models (e.g., GANs
[36], VAEs [60]), we approximate each model by a Gaussian distribution of
image features [41]. We sample images from each model, denoted by $x$. We
obtain the image features $z\coloneqq\psi_{\text{im}}(x)$, where
$\psi_{\text{im}}$ is the feature extractor. Now we express $p(q|\theta)$ in
terms of image features $z$.

$\displaystyle\max_{n\in\\{1,\dots,N\\}}(2\pi|\Sigma_{n}|)^{-\frac{1}{2}}\exp\left(-\frac{1}{2}(z_{q}-\mu_{n})^{T}\Sigma_{n}^{-1}(z_{q}-\mu_{n})\right)$
(2)

where the query image feature is denoted by
$z_{q}\coloneqq\psi_{\text{im}}(q)$, and each model $\theta$ is approximated
by $p(z|\theta)\sim\mathcal{N}(\mu_{n},\Sigma_{n})$. We refer to this method
as Gaussian Density. In Section 4, we show that our method can retrieve models
that share similar visual concepts with the query. Moreover, we can apply the
same method to sketches.

Alternatively, we can train a classifier to predict the model given a query
(generated image). Unfortunately, this close-world assumption does not work
well for a collection with a rapidly growing number of models, as the
classifier needs to be re-trained every time a new model is added.

Sketch-based model retrieval. We can use the same method for sketch-based
model retrieval if the embedding network $\psi_{\text{im}}$ also works for
human sketches. In our experiment, we find that CLIP [95] can produce similar
feature embeddings for similar images and sketches. CLIP outperforms other
pre-trained networks (e.g., DINO [16]) by a large margin.

Text-based model retrieval. Given a text query $q$ and a generative model
$p(x|\theta)$ capturing a distribution of images $x$, we want to estimate the
conditional probability $p(q|\theta)$.

$\displaystyle p(q|\theta)$ $\displaystyle=\int p(q,x|\theta)dx=\int
p(q|x)p(x|\theta)dx$ (3)
$\displaystyle=\int\frac{p(x|q)p(q)}{p(x)}p(x|\theta)dx\propto\int\frac{p(x|q)}{p(x)}p(x|\theta)dx$

Here we assume conditional independence between query $q$ and model $\theta$
given image $x$, so $p(q|x,\theta)=p(q|x)$. We apply Bayes’ rule in the second
line of Equation 3 to get the final expression. In Equation 3, a text query
$q$ may correspond to multiple possible image matches $p(x|q)$, and we
estimate the term $\frac{p(x|q)}{p(x)}$ using cross-modal similarities. In
fact, this expression is proportional to the score function $f$ in contrastive
learning (e.g., InfoNCE [86]), where $f(x,q)\propto\frac{p(x|q)}{p(x)}$. Since
CLIP [95] is trained on a text-image retrieval task with the InfoNCE loss, we
can directly apply the pre-trained CLIP model to simplify Equation 3.

$\displaystyle\max_{\theta\in\\{\theta_{1},\dots,\theta_{N}\\}}\int
p(x|\theta)f(x,q)dx=\mathbb{E}_{x\sim p(x|\theta)}\left[f(x,q)\right].$ (4)

We recall that CLIP consists of an image encoder $\phi_{\text{im}}$ and a text
encoder $\phi_{\text{txt}}$, and it is trained with a score function based on
cosine similarity:

$f(x,q)=\exp\left(\frac{{h_{x}}^{T}h_{q}}{\tau(||{h_{x}}||\cdot||h_{q}||)}\right)=\exp\left(\frac{\tilde{h_{x}}^{T}\tilde{h_{q}}}{\tau}\right),$
(5)

where $h_{x}\coloneqq\phi_{\text{im}}(x)$ and
$h_{q}\coloneqq\phi_{\text{txt}}(q)$ are the image and text feature from CLIP,
respectively. $\tilde{h_{x}}=\frac{h_{x}}{||h_{x}||}$ and
$\tilde{h_{q}}=\frac{h_{q}}{||h_{q}||}$ are the normalized features. Hence,
Equation 5 can be written precisely as:

$\displaystyle\max_{\theta\in\\{\theta_{1},\dots,\theta_{N}\\}}\mathbb{E}_{h_{x}\sim
p(h_{x}|\theta)}\left[\exp\left(\frac{\tilde{h_{x}}^{T}\tilde{h_{q}}}{\tau}\right)\right]$
(6)

Now we have a tractable Monte Carlo estimate of the integral. We sample images
from each model and average the score function of each image sample $x$ and
the text query $q$. We refer to this method as Monte-Carlo. However, directly
applying Monte Carlo estimation is inefficient in practice, since we need lots
of samples to yield a robust estimate. To speed up computation, we provide two
ways to approximate Equation 6. First, we find that a point estimate at the
first moment of $p(h_{x}|\theta)$ works well. We directly estimate the cosine
distance between the mean image features and the query feature. Since the
exponential and temperature mapping is monotonically increasing, the matching
function becomes:

$\displaystyle\max_{\theta\in\\{\theta_{1},\dots,\theta_{N}\\}}\tilde{\mu_{n}}^{T}\tilde{h_{q}},$
(7) $\displaystyle\text{where}\;\;\mu_{n}=$
$\displaystyle\left(\mathbb{E}_{h_{x}\sim
p(h_{x}|\theta)}\left[h\right]\right);\;\tilde{\mu_{n}}=\frac{\mu_{n}}{||\mu_{n}||}$

We refer to this method as 1st Moment. We can also approximate
$p(h_{x}|\theta)$ using both the first and the second moment to get the
following expression.

$\displaystyle\max_{\theta\in\\{\theta_{1},\dots,\theta_{N}\\}}\frac{1}{2\tau}\tilde{h_{q}}^{T}\Sigma_{n}\tilde{h_{q}}+u_{n}^{T}\tilde{h_{q}}$
(8) where $\displaystyle
p(h_{x}|\theta)\sim\mathcal{N}(\mu_{n},\Sigma_{n})\;\;\text{for}\;n\in\\{1,\dots,N\\},$

We refer to this method as 1st \+ 2nd Moment. We provide the details of the
derivation in the Appendix. Empirically, the performance is similar between
approximation to the first or second moment. We provide more analysis in
Section 4.

###  3.2 Extensions and Applications

Here we show additional extensions of our system. Our model search system can
also facilitate several computer graphics and vision applications.

Multimodal query. We can further extend our model search to handle multiple
queries from different modalities. To achieve this, we use a Product-of-
Experts formulation [43, 46] wherein the final likelihood of a model, given a
multimodal query (e.g., text-image pair), is modeled as a product of
likelihoods given individual queries followed by a renormalization.

Finding similar models. Once a model is found, we enable navigation to similar
models. To compute the similarity between models, we use the Fréchet Distance
[26] between the models’ feature distributions. Following prior work [41, 64],
we approximate a model’s distribution by fitting a multivariate Gaussian in an
image feature space. Then the Fréchet Distance can be computed directly from
the Gaussian parameters. For each model, we pre-compute a list of similar
models based on the smallest pairwise distances.

Real image editing. There are many GAN-based image editing methods [121, 89,
68, 138, 14] that create realistic changes in an image by manipulating the
latent variables of a pre-trained generator. However, these methods all assume
that we begin with a generative model that matches the image’s domain. In the
wild, a user may have an image without a corresponding generative model. Given
a collection of generative modes, our system automatically finds a suitable
model to perform image edits. Starting with a real input image, we apply our
image-based model retrieval method (Section 3.1) to find the best matched
model. We find that the best matched model is suitable for inversion and image
editing (Table 5, Figure 8, Figure 9).

Few-shot fine-tuning. Fine-tuning from a pre-trained model is one of the
standard methods to train generative models on a limited amount of data [125,
135, 55]. It helps in mitigating overfitting and requires less compute
resources as well. The abundance of pre-trained generative models—which
inevitably will further increase—presents a unique problem of finding the best
base model to fine-tune on a small number of images from a new domain. In
Section 5.4, we show empirically that transfer learning from similar
generative models as selected by our retrieval methodology leads to on average
faster convergence and better performance in a new domain with limited data.

Figure 3: User interface of model search. The user can enter a text query
and/or upload an image in the search bar to retrieve generative models that
best match the query. Here we show top retrievals for the text query “animated
faces,” which shows StyleGAN-NADA models [34] trained on animated characters.
The user can further explore randomly generated images from the models or
search for more similar models to a particular model. Please view the
supplemental video for more details.

###  3.3 User Interface

We create a web-based UI for our search algorithm. The UI supports searching
and sampling from deep generative models in real-time. The user can enter a
text prompt, upload an image/sketch, or provide both text and an image/sketch.
The interface displays the models that match most closely with the query.
Clicking a model takes the user to a new page where they can sample new images
from the model. The website employs a backend GPU server to enable real-time
model search and image synthesis capabilities even on a mobile device. Figure
3 shows a screenshot of our UI. For more details, please watch the
accompanying video demo.

##  4 Experiments

Here we first evaluate our model retrieval methodology over text, image, and
sketch modalities and discuss several algorithmic design choices. We then show
qualitative and quantitative results for the extensions and applications
enabled by our model search.

Generative model zoo. We evaluate on a collection of 133 generative models
trained using different techniques including GANs [58, 56, 54, 79, 63, 34,
122, 107, 92, 74, 92], diffusion models [44, 116, 24], MLP-based generative
model CIPS [3], and the autoregressive model VQGAN [30]. For evauation we also
manually assign ground truth label to each model based on the type of
generated images, with 23 labels in total. Example labels include “face”,
“animals”, “indoor”, where all the face models will have the “face” label.
Similarly, models trained on datasets like bedroom and conference categories
of LSUN [129] are labeled as “indoor”.

Implementation details. For image-based model retrieval and similar model
search, we test on three different image features – Inception [118], CLIP
[95], and DINO [16]. For text-based model retrieval, we use CLIP features, as
discussed in Section 3.1. We note that CLIP learns a magnitude-invariant
feature space, since it is trained by maximizing cosine similarities between
text and image features. Hence, we use $\ell_{2}$-normalized CLIP features.

To efficiently evaluate our method and compare it across different baselines,
we pre-compute and save 50K generated image features for each model in the
CLIP, DINO, and Inception feature space. Similarly, for the Gaussian Density
based method, we pre-calculate and save the mean and covariance, following the
pre-processing steps described in clean-fid [88].

###  4.1 Model Retrieval

Evaluation metrics. Following the image retrieval literature [91, 127], we
evaluate model retrieval using two different metrics: (1) Top-k accuracy,
i.e., predicting the ground truth generative model of each query in top k, and
(2) Mean Average Precision@k (mAP@k) [127]. All the models with same label as
the query model are included as relevant models regarding the mAP calculation.
The mAP@k metric considers only the top-k predictions, as top-ranked models
matter more for retrieval purposes. The mAP@k metric is computed as below:

$\displaystyle mAP@k=\frac{1}{N}\sum_{i=1}^{N}AP@k(q_{i}),$ (9) where
$\displaystyle
AP@k(q)=\frac{1}{\min(GT_{q},k)}\sum_{j=1}^{\min(GT_{q},k)}P_{q}(j)Rel_{q}(j),$

where $P_{q}(j)$ is the precision of top-j predictions given query $q$ and
$Rel_{q}(j)$ is a binary indicator for $j^{th}$ prediction being relevant.
$GT_{q}$ is the number of relevant models corresponding to the query. The
above metric weighs all the models similar to the query equally during
evaluation. For example, given a text query face, all face generative models
are treated as relevant and should be retrieved at top.

Text-based model retrieval. To evaluate text retrieval, we manually assign a
ground-truth text description to each model in the collection and evaluate the
retrieval performance with that as query. Since there can be multiple models
with similar text descriptions, we only evaluate text retrieval using the mAP
metric. We use CLIP feature space [95], since CLIP has both text and image
encoders. In Table 1, we show retrieval performance of the different methods
as discussed in Section 3. We achieve the mAP@10 score of $0.71$ applying our
1st Moment based method. Compared to the Monte-Carlo approach with 50K
samples, it is more than $5$ times faster while performing similarly (see
runtimes in Table 3). To ensure that the retrieval is robust to variation in
text queries, we also evaluate the method with augmented queries that prepend
phrases like “an image of” and “a photo of” to each text query. As shown in
Table 1, the retrieval performance decreases only marginally. Figure 4 shows
qualitative examples of the top three and lowest-ranked retrieval given a text
query. Both quantitative numbers and visual inspection of results show that
our method retrieves relevant generative models. We also analyze the retrieval
score of all models corresponding to each query. For object categories, such
as “dogs” or “buses”, we observe a clear drop in retrieval score for
irrelevant models. For broader queries, such as “indoors”, “modern art” and
“painting”, the dropoff is gradual.

Model retrieval via image and sketch queries. To evaluate image-based model
retrieval, we automatically create image queries using images generated by
each model. We generate $50$ image queries for each model and use the
corresponding model as the ground truth for the queries. In total, we have
$133\times 50$ image queries. To obtain sketch queries, we use the method of
Chan et al. [17] to convert images into sketches.

We also apply 1st Moment method (Equation 7) to image-based model retrieval.
Specifically, given an image query, we extract the feature using CLIP’s image
encoder $\phi_{\text{im}}$. We then compute the cosine distance between the
query feature and the first moment $\mu_{n}$.

In Table 2, we show retrieval results of different formulations with CLIP,
DINO, and Inception network features. We observe that the retrieval
performance is best with CLIP features across different query types,
especially for sketches. For image-based retrieval, Gaussian Density
outperforms Monte-Carlo and 1st Moment on the mAP metric. In terms of speed,
1st Moment method performs the best ($\sim$ 5 times faster) at the cost of
worse performance, compared to Gaussian Density. For sketch-based retrieval,
CLIP features significantly outperform DINO and Inception features. Example
retrieval results are shown in Figure 4 for both image and sketch queries.
Figure 5 further shows qualitative ablation of the three pretrained networks’
feature space. For a dog sketch query, both DINO and Inception features fail
to retrieve the relevant model and return art-based models instead.

|  | mAP@k
---|---|---
|  | mAP@5 | mAP@10 | mAP
Original | Monte-Carlo (50K) | 0.76 | 0.71 | 0.71
Monte-Carlo (1K) | 0.75 | 0.71 | 0.71
1st Moment (ours) | 0.78 | 0.71 | 0.68
1st \+ 2nd Moment (ours) | 0.74 | 0.70 | 0.70
Augmented | Monte-Carlo (50K) | 0.79 | 0.73 | 0.73
Monte-Carlo (1K) | 0.78 | 0.73 | 0.72
1st Moment (ours) | 0.79 | 0.72 | 0.69
1st \+ 2nd Moment (ours) | 0.77 | 0.72 | 0.72

Table 1: Text-based model retrieval. Our 1st Moment and 1st \+ 2nd Moment
methods perform on-average similarly to the Monte-Carlo based approach while
being computationally more efficient. We evaluate the retrieval using mAP@k
evaluation metrics. For the Augmented version, we create multiple queries from
each text query by pre-pending phrases like “an image of” in front of the
query.

|  | Top-k Accuracy
---|---|---
|  | Top-1 | Top-5 | Top-10
Image (Gen.) | CLIP+Monte-Carlo (50K) | 0.82 | 0.98 | 1.00
CLIP+1st Moment (ours) | 0.75 | 0.95 | 0.99
CLIP+Gaussian Density (ours) | 0.77 | 0.95 | 1.00
DINO+Gaussian Density | 0.83 | 0.96 | 0.98
Inception+Gaussian Density | 0.70 | 0.92 | 0.98
Sketch | CLIP+Monte-Carlo (50K) | 0.36 | 0.69 | 0.86
CLIP+1st Moment (ours) | 0.35 | 0.70 | 0.84
CLIP+Gaussian Density (ours) | 0.33 | 0.67 | 0.86
DINO+Gaussian Density | 0.08 | 0.24 | 0.32
Inception+Gaussian Density | 0.08 | 0.22 | 0.29
|  | mAP@k
|  | mAP@5 | mAP@10 | mAP
Image (Gen.) | CLIP+Monte-Carlo (50K) | 0.81 | 0.74 | 0.75
CLIP+1st Moment (ours) | 0.79 | 0.74 | 0.74
CLIP+Gaussian Density (ours) | 0.81 | 0.75 | 0.76
DINO+Gaussian Density | 0.80 | 0.73 | 0.69
Inception+Gaussian Density | 0.79 | 0.72 | 0.67
Sketch | CLIP+Monte-Carlo (50K) | 0.67 | 0.63 | 0.65
CLIP+1st Moment (ours) | 0.70 | 0.64 | 0.64
CLIP+Gaussian Density (ours) | 0.68 | 0.64 | 0.66
DINO+Gaussian Density | 0.32 | 0.26 | 0.33
Inception+Gaussian Density | 0.30 | 0.25 | 0.34

Table 2: Image- and sketch-based model retrieval. We evaluate retrieval in the
feature space of CLIP, DINO, and Inception networks and observe best
performance using CLIP. We use the generated images as query for image-based
model retrieval. For sketch-based evaluation we use the method of Chan et al.
[17] to convert generated images to sketch.

|  |  Feature Extraction | Model Scoring
---|---|---|---
|  | 133 | 10K | 1M
Text | Monte-Carlo (50K) | 5.02ms | 0.46ms | OOM | OOM
1st Moment | 0.08ms | 0.20ms | 0.29ms
Image | Monte-Carlo (50K) | 6.75ms | 0.46ms | OOM | OOM
Gaussian Density | 0.44ms | 0.46ms | OOM
1st Moment | 0.08ms | 0.20ms | 0.29ms

Table 3: Model retrieval running time. Both the 1st Moment and Gaussian
Density-based scoring methods run quickly on our test machine. After we get
the score for each model, we additionally run torch.argsort to get the best
matches, which takes 0.04, 0.11, and 0.07ms for 133, 10K, and one million
models, respectively (not shown in the table). OOM stands for “out of memory”
and that the retrieval cannot be done in a single pass. Figure 4: Qualitative
results of model retrieval. Top row (image query): The still-life painting
retrieves models related to art and ranks the AFHQ Wild [20] model at the
bottom. Middle row (sketch query): Both horse and church sketches retrieve
relevant models at the top ranking. Bottom row (text query): The query “human
wearing a pair of glasses” successfully retrieves a GANSketch [122] model
finetuned for human faces with glasses. Similarly, for query “a bird that
talks”, we find a Self-Distilled GAN [79] trained on Internet parrots images.
Figure 5: Qualitative comparison of image- and sketch-based model retrieval in
different networks’ feature spaces. For image queries, all three
networks—CLIP, DINO, and Inception have similar performance. For sketch
queries, we observe that CLIP works significantly better, as shown in the
example above and in Table 2. Both inception and DINO score artistic models
higher, which might not be best suited for the given query.

Running time and memory. The computational and memory efficiency of the
retrieval algorithm is crucial to supporting many concurrent users searching
over large-scale model collections. Therefore. we profile our method on 133
generative models as well as a much greater number of simulated models. To
create simulated models, we sample the 1st Moment model statistics and Monte-
Carlo samples from a 512-dimensional normal distribution that correspond to
points in the CLIP feature space, and the second moment statistics are
generated as a unit covariance matrix with a small, uniform, symmetric noise
added. We store the complete data in the GPU’s VRAM during computation. We run
the following tests on a machine equipped with an AMD Threadripper 3960X and
NVIDIA RTX A5000 running Pytorch 1.11.0, and report them in Table 3.

During model retrieval, extracting query features is constant irrespective of
the number of models to retrieve. In case of text extracting CLIP features
takes 4.95 ms and 0.36 GB of VRAM. For retrieval out of 133 models, the full
50K-sample Monte-Carlo method takes 0.46 ms but uses 12.87 GB of VRAM, making
memory a constraint. On the other hand, the 1st Moment based method is much
more computationally efficient and on-par (see Table 2). For one million
models, the 1st Moment based scoring method takes 0.28 ms and 1.93 GB of VRAM.
When handling image-to-model or sketch-to-model retrieval, extracting query
features with CLIP takes 7.40 ms and 0.36 GB of VRAM. Gaussian Density method
requires extra memory to store the models’ covariance matrices. With $10,000$
models, it takes 0.44ms and 9.85 GB of VRAM. Switching to the 1st Moment only
method improves the computational efficiency, enabling sketch- and image-based
retrieval from one million models in a single pass, albeit at a slight
reduction in precision (see Table 2).

After getting the retrieval scores, the time required for sorting and
selecting the most relevant models is negligible on a GPU. For one million
models, sorting takes 0.12ms and 0.062 GB of VRAM. With the 1st Moment method,
a user can retrieve models from a 1-million-model collection using text,
sketch, or image query in real-time.

|  | Top-k Accuracy
---|---|---
|  | Top-1 | Top-5 | Top-10
Image (Gen.) | Description (text) | 0.14 | 0.33 | 0.46
Description (CLIP) | 0.44 | 0.78 | 0.93
CLIP+Gaussian Density (ours) | 0.77 | 0.95 | 1.00
Sketch | Description (text) | 0.12 | 0.28 | 0.39
Description (CLIP) | 0.24 | 0.53 | 0.70
CLIP+Gaussian Density (ours) | 0.33 | 0.67 | 0.86
|  | mAP@k
|  | mAP@5 | mAP@10 | mAP
Image (Gen.) | Description (text) | 0.52 | 0.46 | 0.50
Description (CLIP) | 0.71 | 0.67 | 0.62
CLIP+Gaussian Density (ours) | 0.81 | 0.75 | 0.76
Sketch | Description (text) | 0.46 | 0.41 | 0.46
Description (CLIP) | 0.66 | 0.63 | 0.61
CLIP+Gaussian Density (ours) | 0.68 | 0.64 | 0.66

Table 4: Comparison with model-description-based search. We compare our
Gaussian Density retrieval method with two model-description-based baseline
methods, Description (text) and Description (CLIP). In Description (text), we
caption image and sketch queries using ClipCap [78] and select models whose
descriptions contains any nouns/verbs/adjectives from the caption. In
Description (CLIP), we embed model descriptions into the CLIP feature space,
and directly compare the similarity between the image/sketch query feature and
the tag feature. Similar to Table 2, We use the generated images as query for
image-based model retrieval. For sketch-based evaluation we use the method of
Chan et al. [17] to convert generated images to sketch.  Figure 6: Multi-modal
user query. We show a qualitative example of how multi-modal queries can help
refine the model search. With only the image of “Nicolas Cage” we retrieve
only face models. But with the multi-model query of image and text as “dog”,
we can retrieve the StyleGAN-NADA model of “Nicolas Cage dogs”.  Figure 7:
Finding similar models. In the top row, when the query is a face model, we
retrieve more face generative models. In the middle and bottom rows, in which
we our collection does not contain models in the identical category, our
method returns models of broadly similar categories.

Baseline: Index models using model descriptions.

An alternative approach to content-based model search is to index each model
using user-defined descriptions. For each user query, we will find the model
with a description that best matches the query. To test this approach, we use
our ground-truth text description (e.g., “portraits with botero’s style” for
one of the StyleGAN-NADA models).

We compare our method with two model-description-based search methods in Table
4. (1) Description (text): we test image- and sketch-based retrieval by
generating captions from image and sketch queries using ClipCap [78]; then we
select models whose descriptions contains any nouns/verbs/adjectives from the
caption. (2) Description (CLIP): we embed model descriptions into the CLIP
feature space, and directly compare the similarity between the image/sketch
query feature and the description feature.

For both image and sketch queries, our method outperforms the two mentioned
model-description-based baselines, and it does not require manually collecting
model descriptions. Typical failures of the baselines occur when captions are
incorrect or when the object-centric tags are too limited to characterize the
full distribution. Creating a comprehensive descriptions might reduce the gap,
but it is infeasible to describe every visual aspect and anticipate users’
queries in advance. Our evaluation does not include text-based queries, as we
use the model descriptions as the ground truth. However, we expect model-
description-based retrieval of text queries to have the same limitation, where
a text query requests a visual concept that the tags fail to enumerate.

Metadata such as network architectures, dataset names, authors, and training
objectives are not reflected in the model’s content, so searching based on
author-provided metadata is a valuable complement to our content-based search.
However, our experiment shows that indexing with model descriptions will not
capture all visual details of the content of a model, since it is subject to
incomplete/incorrect manual annotations. Therefore, in our Modelverse
platform, we support the content-based search method in combination with a
metadata search.

##  5 Extensions and Applications

Our work enables users to explore available generative models and find the
best models for different use cases. Here we show several use cases, including
multi-modal queries, finding similar models, image editing, and few-shot
transfer learning. For all applications we used CLIP feature space in our
retrieval method as it performs the best across modalities.

Dataset | Image Inversion
---|---
LPIPS-alex $(\downarrow)$ | LPIPS-vgg $(\downarrow)$ | PSNR$(\uparrow)$ | $||\text{w}^{+}_{\text{opt}}-\text{w}_{\text{avg}}||\;(\downarrow)$
Rank-1 | Rank-10 | Rank-15 | Rank-1 | Rank-10 | Rank-15 | Rank-1 | Rank-10 | Rank-15 | Rank-1 | Rank-10 | Rank-15
CelebA-HQ | 0.13 | 0.31 | 0.28 | 0.21 | 0.37 | 0.31 | 23.94 | 20.68 | 22.20 | 803.18 | 1164.52 | 2291.42
LSUN Church | 0.32 | 0.38 | 0.39 | 0.36 | 0.43 | 0.44 | 17.91 | 17.78 | 18.39 | 570.90 | 1194.89 | 1759.43

Table 5: Inverting real images using different ranked models. We use $100$
images each of CelebaHQ and LSUN Church dataset as queries and invert each
image using models at 1, 10 and 15 rank in the retrieval score. The
reconstruction quality is measured using LPIPS [132] and PSNR. Top-ranked
models are significantly better at image inversion compared to lower-rank
models in both case. We also measure the mean distance between the optimized
latent $\text{w}^{+}_{\text{opt}}$ and mean latent $\text{w}_{\text{avg}}$ of
the model. For lower-rank models, the optimized latent
$\text{w}^{+}_{\text{opt}}$ deviates considerably from the mean latent which
hints at overfitting over reconstruction loss and low image editability [121,
140].  Figure 8: Projecting real images to retrieved StyleGAN2 models. Example
image inversion for CelebA-HQ and LSUN Church images using different ranked
models. Given the query image (1st column), using the top-ranked model results
in more accurate image reconstruction compared to lower-ranked models. Models
at rank-1, 10, and 15 that are retrieved for the above queries are as follows.
Row 1 and 2: StyleGAN2-FFHQ, StyleGAN2-LSUN Church, Vision-aided
StyleGAN2-100-shot-Bridge. Row 3: Vision-aided StyleGAN2-LSUN Church,
StyleGAN2-Cakes, Vision-aided StyleGAN2-Animalface Cat. Row 4: Vision-aided
StyleGAN2-LSUN Church, StyleGAN2-100-shot-Bridge, Vision-aided
StyleGAN2-Animalface Cat. Figure 9: Editing real images. We use the model
ranked first by our image-based model retrieval algorithm for inverting the
real image, and then we perform editing using GANspace [49]. Since using a
random model often leads to sub-optimal inversion, selecting relevant models
is critical for image editing applications.

###  5.1 Multimodal User Query

We show qualitatively that our search method can be extended to multimodal
queries, based on the Product-of-Experts formulation described in Section 3.2.
We demonstrate how leveraging multiple input modalities from the user can
retrieve models which are better tailored to user queries. Specifically, we
test this application for text-image pairs, as shown in Figure 6.

###  5.2 Finding Similar Models

As explained in Section 3.2, we use the FID between the feature distribution
of each generative model as the scoring method for retrieving similar models.
We use CLIP, DINO, and Inception networks’ feature space and evaluate Average
Precision using ground truth similar models (models with same label). We get
an AP of $0.68$, $0.68$, and $0.66$ respectively. Figure 7 shows qualitative
examples of similar model retrieval using FID metric in CLIP feature space.

Figure 10: Few-shot transfer learning with model search. Using generative
models similar to the source dataset as ranked by our model retrieval
algorithm leads to faster convergence and better or on-par performance in
terms of FID metric. Left: 100-shot Obama dataset finetuned using rank-1
StyleGAN2-FFHQ model and rank-20 and rank-25 models trained on LSUN Horse and
Church respectively. Middle: 100-shot Grumpy cat dataset finetuned using
rank-1 Vision-aided-Animalface Cat model and rank-20, 25 models trained on
cakes and LSUN Horse respectively. Right: Moongate dataset finetuned using
rank-1 StyleGAN2 LSUN Church model and rank-20 and rank-25 models trained on
LSUN Cat and Horse respectively.  Figure 11: Latent space interpolation using
rank-1 models vs random model. We find the most relevant model for inverting
images and then interpolate in that generative model’s latent space. Selecting
a relevant model leads to meaningful interpolation between the two images.

###  5.3 Image Reconstruction and Editing

Image inversion. For image editing, we use 15 StyleGAN2-based models at
256x256 resolution, for a fair comparison regarding image resolution and
network architectures. We evaluate real image inversion on validation set
images from LSUN Church [129] and CelebA-HQ datasets [54]. We use the
optimization-based inversion technique [58] in $W^{+}$ latent space with LPIPS
[132] and pixel level mean square loss between the target and generated image.
Given an image query, we run inversion on models ranked at 1, 10, and 15 by
our Gaussian Density retrieval method. We evaluate the reconstruction quality
between $100$ inverted and target images of both category using standard
metrics like LPIPS and PSNR. We also calculate the mean $\ell_{2}$ distance
between optimized $\text{w}^{+}_{\text{opt}}$ and mean latent
$\text{w}_{\text{avg}}$ of the model which shows the extent of overfitting to
the reconstruction loss by the model. The results are as shown in Table 5. The
top retrieved models that are similar to the image query result in better
image inversion on-average across all metrics. Moreover, for lower-ranked
models, the distance $||\text{w}^{+}_{\text{opt}}-\text{w}_{\text{avg}}||$ is
significantly higher compared to the top-rank models, which has been shown to
correlate negatively with image editability [121, 140]. Figure 8 shows some
qualitative samples of image inversion using the different ranked models.
Lower-ranked models yield inversions with substantially poorer quality.

Image editing and interpolation. We now show that images inverted with top-
ranked models can be further edited using existing GAN-based image editing
techniques such as GANSpace [49]. Figure 9 shows examples of editing on
Ukiyo-e images to change the frowning face to a smiling face. For the flower
category, we show example edits that change the petal colors. We can also
perform latent space interpolation between inverted images of the same
category and create visually compelling samples as shown in Figure 11. A
rank-1 model results in smoother interpolation of one image into another in
contrast to inversion and interpolation using a random model.

###  5.4 Few-Shot Transfer Learning

For few-shot transfer learning, we restrict our experiments to 256x256
resolution StyleGAN2 models due to limited computing resources (27 models). We
begin with a small dataset of 100-136 images. Then, we rank the 27 models by
the average retrieval score over all images. We select models at rank 1, 20
and 25 as source models for transfer learning. For finetuning the generator on
the new dataset, we use vision-aided GANs [63], one of the leading methods in
few-shot GAN training.

Datasets and evaluation metric. We use three standard few-shot datasets: Obama
[135] (100 images), Grumpy Cat [135] (100 images), and Moongate [69] (136
images). We use the Fréchet Inception Distance (FID) [42] metric for
evaluation.

Results. Figure 10 shows the results of transfer learning using different
source models with varying retrieval rankings. Finetuning from rank-1 models
results in faster convergence on average compared to lower-ranked models. This
shows empirically that training from similar models results in faster
convergence and thus requires less compute.

Figure 12: Failure cases. (Top) Sometimes, a sketch query (e.g., the bird
sketch) will match models with abstract styles. It is ambiguous whether the
CLIP feature should match the shape of the sketch, or the styles and textures.
(Bottom) For conflicting multi-modal queries (elephant text query + a dog
image), our system has difficulty retrieving models with both concepts. There
are no elephant models in the top-ranked models.

##  6 Discussion and Limitations

We have introduced the problem of content-based retrieval for deep generative
image models, whose goal is to help users find, explore, and share new
generative models more easily. Interestingly, we have found that scoring based
on a probabilistic model works well, and further that applying a Gaussian
density or first-moment approximation to the distribution of generated image
features produce accurate search results with a minimal memory and time
footprint. We have demonstrated a model search prototype using our method. Our
experiments have shown that searches over an indexed collection are useful for
finding a good model for image editing and transfer learning.

In Figure 12, we show several limitations of our current method: queries for
specific sketches (e.g., the bird sketch in the figure) will sometimes match
models that generate a wide range of abstract shapes rather than the specific
intended model. Conversely, queries intended to capture diversity will
sometimes match an overly specific model. Developing new ways to allow a user
to describe the desired diversity in a model is a promising area for future
work. Further, our method is susceptible to failure when the query itself is
inherently ambiguous. For instance, given a sketch input of a cat, our method
cannot recognize if the user intends to retrieve a model that generates cat
sketches or a model that generates real looking cat images conditioned on the
sketch. Finally, it is difficult for our method to handle conflicting
multimodal queries (e.g., elephant text query + a dog image).

Our study is a small first step. We have demonstrated a way to search over
unconditional generative models trained on image datasets. Still, we have not
yet examined conditional models nor models that synthesize text, audio, or
other media. Nevertheless, as collections of many kinds of pretrained models
continue to balloon, we have shown that model search is a feasible approach
for working with model collections, and we anticipate that, coupled with
effective search methods, collections of pretrained models will be an
increasingly valuable resource for practitioners and researchers.

Acknowledgment. We thank George Cazenavette, Gaurav Parmar, and Chonghyuk
(Andrew) Song for proofreading the draft. We are also grateful to Aaron
Hertzmann, Maneesh Agrawala, Stefano Ermon, Chenlin Meng, William Peebles,
Richard Zhang, Phillip Isola, Taesung Park, Muyang Li, Kangle Deng, and Ji Lin
for helpful comments and discussion. The work is partly supported by Adobe
Inc., Sony Corporation, and Naver Corporation.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?  In CVPR, 2020.
  * [2] Badour Albahar, Jingwan Lu, Jimei Yang, Zhixin Shu, Eli Shechtman, and Jia-Bin Huang.  Pose with Style: Detail-preserving pose-guided image synthesis with conditional stylegan.  ACM TOG, 2021.
  * [3] Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov.  Image generators with conditionally-independent pixel synthesis.  In CVPR, 2021.
  * [4] Relja Arandjelović and Andrew Zisserman.  Three things everyone should know to improve object retrieval.  In CVPR, 2012.
  * [5] Derek Philip Au.  This vessel does not exist.  https://thisvesseldoesnotexist.com/, 2019.
  * [6] Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky.  Neural codes for image retrieval.  In ECCV, 2014.
  * [7] Ricardo Baeza-Yates, Berthier Ribeiro-Neto, et al.  Modern information retrieval, volume 463.  ACM press New York, 1999.
  * [8] David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, and Antonio Torralba.  Rewriting a deep generative model.  In ECCV, 2020.
  * [9] David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba.  Semantic photo manipulation with a generative image prior.  ACM TOG, 2019.
  * [10] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Zhou Bolei, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba.  Gan dissection: Visualizing and understanding generative adversarial networks.  In ICLR, 2019.
  * [11] Amit H Bermano, Rinon Gal, Yuval Alaluf, Ron Mokady, Yotam Nitzan, Omer Tov, Or Patashnik, and Daniel Cohen-Or.  State-of-the-art in the architecture, methods and applications of stylegan.  arXiv preprint arXiv:2202.14020, 2022.
  * [12] Daniel Bolya, Rohit Mittapalli, and Judy Hoffman.  Scalable diverse model selection for accessible transfer learning.  NeurIPS, 2021.
  * [13] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  In ICLR, 2019.
  * [14] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston.  Neural photo editing with introspective adversarial networks.  In ICLR, 2017.
  * [15] Yang Cao, Changhu Wang, Liqing Zhang, and Lei Zhang.  Edgel index for large-scale sketch-based image search.  In CVPR, 2011.
  * [16] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [17] Caroline Chan, Fredo Durand, and Phillip Isola.  Learning to generate line drawings that convey geometry and semantics.  In CVPR, 2022.
  * [18] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman.  Maskgit: Masked generative image transformer.  In CVPR, 2022.
  * [19] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In ICML, 2020.
  * [20] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In CVPR, 2020.
  * [21] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff.  Vqgan-clip: Open domain image generation and editing with natural language guidance.  In ECCV, 2022.
  * [22] Navneet Dalal and Bill Triggs.  Histograms of oriented gradients for human detection.  In CVPR, 2005.
  * [23] Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z Wang.  Image retrieval: Ideas, influences, and trends of the new age.  ACM Computing Surveys (Csur), 2008.
  * [24] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  In NeurIPS, 2021.
  * [25] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp.  In ICLR, 2017.
  * [26] DC Dowson and BV666017 Landau.  The fréchet distance between multivariate normal distributions.  Journal of multivariate analysis, 1982.
  * [27] Kshitij Dwivedi and Gemma Roig.  Representation similarity analysis for efficient task taxonomy & transfer learning.  In CVPR, 2019.
  * [28] Mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, and Marc Alexa.  Sketch-based image retrieval: Benchmark and bag-of-features descriptors.  IEEE transactions on visualization and computer graphics, 17(11):1624–1636, 2010.
  * [29] Ahmed Elgammal.  Ai is blurring the definition of artist: Advanced algorithms are using machine learning to create art autonomously.  American Scientist, 107(1):18–22, 2019.
  * [30] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In CVPR, 2021.
  * [31] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler.  Vse++: Improving visual-semantic embeddings with hard negatives.  In BMVC, 2017.
  * [32] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov.  Devise: A deep visual-semantic embedding model.  In NeurIPS, 2013.
  * [33] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [34] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  ACM TOG, 2022.
  * [35] Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin.  Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval.  IEEE transactions on pattern analysis and machine intelligence, 35(12):2916–2929, 2012.
  * [36] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NeurIPS, 2014.
  * [37] Timofey Grigoryev, Andrey Voynov, and Artem Babenko.  When, why, and which pretrained gans are useful?  In ICLR, 2022.
  * [38] Venkat N Gudivada and Vijay V Raghavan.  Content based image retrieval systems.  Computer, 28(9):18–22, 1995.
  * [39] David Ha and Jürgen Schmidhuber.  World models.  arXiv preprint arXiv:1803.10122, 2018.
  * [40] Aaron Hertzmann.  Computers do not make art, people do.  Communications of the ACM, 63(5):45–48, 2020.
  * [41] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs trained by a two time-scale update rule converge to a local Nash equilibrium.  In NeurIPS, 2017.
  * [42] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NeurIPS, 2017.
  * [43] Geoffrey E Hinton.  Training products of experts by minimizing contrastive divergence.  Neural computation, 14(8):1771–1800, 2002.
  * [44] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In NeurIPS, 2020.
  * [45] Weiming Hu, Nianhua Xie, Li Li, Xianglin Zeng, and Stephen Maybank.  A survey on visual content-based video indexing and retrieval.  IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 41(6):797–819, 2011.
  * [46] Xun Huang, Arun Mallya, Ting-Chun Wang, and Ming-Yu Liu.  Multimodal conditional image synthesis with product-of-experts gans.  In ECCV, 2022.
  * [47] Minyoung Huh, Pulkit Agrawal, and Alexei A Efros.  What makes imagenet good for transfer learning?  arXiv preprint arXiv:1608.08614, 2016.
  * [48] Aapo Hyvärinen and Peter Dayan.  Estimation of non-normalized statistical models by score matching.  Journal of Machine Learning Research, 6(4), 2005.
  * [49] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  In NeurIPS, 2020.
  * [50] Wonjong Jang, Gwangjin Ju, Yucheol Jung, Jiaolong Yang, Xin Tong, and Seungyong Lee.  Stylecarigan: Caricature generation via stylegan feature map modulation.  ACM TOG, 2021.
  * [51] Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez.  Aggregating local descriptors into a compact image representation.  In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3304–3311. IEEE, 2010.
  * [52] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In ICML, 2021.
  * [53] Andrej Karpathy, Armand Joulin, and Li F Fei-Fei.  Deep fragment embeddings for bidirectional image sentence mapping.  In NeurIPS, 2014.
  * [54] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  In ICLR, 2018.
  * [55] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  In NeurIPS, 2020.
  * [56] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In NeurIPS, 2021.
  * [57] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [58] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In CVPR, 2020.
  * [59] Gwanghyun Kim and Jong Chul Ye.  Diffusionclip: Text-guided image manipulation using diffusion models.  In CVPR, 2022.
  * [60] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In ICLR, 2014.
  * [61] Simon Kornblith, Jonathon Shlens, and Quoc V Le.  Do better imagenet models transfer better?  In CVPR, 2019.
  * [62] Alex Krizhevsky and Geoffrey E Hinton.  Using very deep autoencoders for content-based image retrieval.  In ESANN, volume 1, page 2. Citeseer, 2011.
  * [63] Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Ensembling off-the-shelf models for gan training.  In CVPR, 2022.
  * [64] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen.  The role of imagenet classes in fr$\backslash$’echet inception distance.  arXiv preprint arXiv:2203.06026, 2022.
  * [65] Kathleen M Lewis, Srivatsan Varadharajan, and Ira Kemelmacher-Shlizerman.  Tryongan: Body-aware try-on via layered interpolation.  ACM TOG, 2021.
  * [66] Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman.  Few-shot image generation with elastic weight consolidation.  In NeurIPS, 2020.
  * [67] Yen-Liang Lin, Cheng-Yu Huang, Hao-Jeng Wang, and Winston Hsu.  3d sub-query expansion for improving sketch-based multi-view image retrieval.  In ICCV, 2013.
  * [68] Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler.  Editgan: High-precision semantic image editing.  In NeurIPS, 2021.
  * [69] Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal.  Towards faster and stabilized gan training for high-fidelity few-shot image synthesis.  In ICLR, 2021.
  * [70] Li Liu, Fumin Shen, Yuming Shen, Xianglong Liu, and Ling Shao.  Deep sketch hashing: Fast free-hand sketch-based image retrieval.  In CVPR, 2017.
  * [71] Ming-Yu Liu, Xun Huang, Jiahui Yu, Ting-Chun Wang, and Arun Mallya.  Generative adversarial networks for image and video synthesis: Algorithms and applications.  arXiv preprint arXiv:2008.02793, 2020.
  * [72] Yunzhe Liu, Rinon Gal, Amit H Bermano, Baoquan Chen, and Daniel Cohen-Or.  Self-conditioned generative adversarial networks for image editing.  ACM TOG, 2022.
  * [73] David G Lowe.  Distinctive image features from scale-invariant keypoints.  IJCV, 60(2):91–110, 2004.
  * [74] lucid layers.  Datasets and pretrained models for stylegan3.  https://github.com/edstoica/lucid_stylegan3_datasets_models/, 2022\.
  * [75] Christopher Manning, Prabhakar Raghavan, and Hinrich Schütze.  Introduction to information retrieval.  Natural Language Engineering, 16(1):100–103, 2010.
  * [76] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  SDEdit: Guided image synthesis and editing with stochastic differential equations.  In ICLR, 2022.
  * [77] Sangwoo Mo, Minsu Cho, and Jinwoo Shin.  Freeze the discriminator: a simple baseline for fine-tuning gans.  In CVPR Workshop, 2020.
  * [78] Ron Mokady, Amir Hertz, and Amit H Bermano.  Clipcap: Clip prefix for image captioning.  arXiv preprint arXiv:2111.09734, 2021.
  * [79] Ron Mokady, Michal Yarom, Omer Tov, Oran Lang, Daniel Cohen-Or, Tali Dekel, Michal Irani, and Inbar Mosseri.  Self-distilled stylegan: Towards generation from internet photos.  In ACM SIGGRAPH, 2022.
  * [80] Basil Mustafa, Carlos Riquelme, Joan Puigcerver, André Susano Pinto, Daniel Keysers, and Neil Houlsby.  Deep ensembles for low-data transfer learning.  arXiv preprint arXiv:2010.06866, 2020.
  * [81] Atsuhiro Noguchi and Tatsuya Harada.  Image generation from small datasets via batch statistics adaptation.  In ICCV, 2019.
  * [82] Utkarsh Ojha, Yijun Li, Cynthia Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang.  Few-shot image generation via cross-domain correspondence.  In CVPR, 2021.
  * [83] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang.  Few-shot image generation via cross-domain correspondence.  In CVPR, 2021.
  * [84] Aude Oliva and Antonio Torralba.  Modeling the shape of the scene: A holistic representation of the spatial envelope.  IJCV, 42(3):145–175, 2001.
  * [85] Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu.  Conditional image generation with pixelcnn decoders.  In NeurIPS, 2016.
  * [86] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [87] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic.  Learning and transferring mid-level image representations using convolutional neural networks.  In CVPR, 2014.
  * [88] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu.  On buggy resizing libraries and surprising subtleties in fid calculation.  In CVPR, 2022.
  * [89] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In ICCV, 2021.
  * [90] William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei Efros, and Eli Shechtman.  Gan-supervised dense visual alignment.  In CVPR, 2022.
  * [91] James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman.  Object retrieval with large vocabularies and fast spatial matching.  In CVPR, 2007.
  * [92] Justin Pinkney.  Awesome pretrained stylegan.  https://www.justinpinkney.com/pretrained-stylegan/, 2020.
  * [93] Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby.  Scalable transfer learning with expert models.  In ICLR, 2021.
  * [94] Filip Radenovic, Giorgos Tolias, and Ondrej Chum.  Deep shape matching.  In ECCV, 2018.
  * [95] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [96] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [97] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  In NeurIPS, 2019.
  * [98] Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, and Moacir Ponti.  Sketchformer: Transformer-based representation for sketched structure.  In CVPR, 2020.
  * [99] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  ACM TOG, 2021.
  * [100] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [101] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In arXiv preprint arxiv:2208.12242, 2022.
  * [102] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.  Adapting visual category models to new domains.  In ECCV, 2010.
  * [103] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Ho Jonathan, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [104] Ruslan Salakhutdinov and Geoffrey Hinton.  Semantic hashing.  International Journal of Approximate Reasoning, 50(7):969–978, 2009\.
  * [105] Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays.  The sketchy database: learning to retrieve badly drawn bunnies.  ACM TOG, 35(4):1–12, 2016.
  * [106] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger.  Projected gans converge faster.  In NeurIPS, 2021.
  * [107] Axel Sauer, Katja Schwarz, and Andreas Geiger.  Stylegan-xl: Scaling stylegan to large diverse datasets.  In ACM SIGGRAPH, 2022.
  * [108] Derrick Schultz.  Freagan, undertrained gan trained on Frea Buckler’s artwork.  https://twitter.com/dvsch/status/1255885874560225284, 2020.
  * [109] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.  Interfacegan: Interpreting the disentangled face representation learned by gans.  IEEE TPAMI, 2020.
  * [110] Abhinav Shrivastava, Tomasz Malisiewicz, Abhinav Gupta, and Alexei A Efros.  Data-driven visual similarity for cross-domain image matching.  In ACM SIGGRAPH Asia, 2011.
  * [111] Josef Sivic and Andrew Zisserman.  Video google: A text retrieval approach to object matching in videos.  In ICCV, 2003.
  * [112] Arnold WM Smeulders, Marcel Worring, Simone Santini, Amarnath Gupta, and Ramesh Jain.  Content-based image retrieval at the end of the early years.  IEEE TPAMI, 2000.
  * [113] Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng.  Grounded compositional semantics for finding and describing images with sentences.  Transactions of the Association for Computational Linguistics, 2:207–218, 2014.
  * [114] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In ICML, 2015.
  * [115] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In ICLR, 2021.
  * [116] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In ICLR, 2021.
  * [117] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In ICLR, 2021.
  * [118] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In CVPR, 2016.
  * [119] A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-Brualla, T. Simon, J. Saragih, M. Nießner, R. Pandey, S. Fanello, G. Wetzstein, J.-Y. Zhu, C. Theobalt, M. Agrawala, E. Shechtman, D. B Goldman, and M. Zollhöfer.  State of the Art on Neural Rendering.  Computer Graphics Forum (EG STAR 2020), 2020.
  * [120] Antonio Torralba, Rob Fergus, and Yair Weiss.  Small codes and large image databases for recognition.  In CVPR, 2008.
  * [121] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  ACM TOG, 2021.
  * [122] Sheng-Yu Wang, David Bau, and Jun-Yan Zhu.  Sketch your own gan.  In ICCV, 2021.
  * [123] Sheng-Yu Wang, David Bau, and Jun-Yan Zhu.  Rewriting geometric rules of a gan.  ACM TOG, 2022.
  * [124] Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer.  Minegan: effective knowledge transfer from gans to target domains with few images.  In CVPR, 2020.
  * [125] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu.  Transferring gans: generating images from limited data.  In ECCV, 2018.
  * [126] Yair Weiss, Antonio Torralba, and Rob Fergus.  Spectral hashing.  In NeurIPS, 2008.
  * [127] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim.  Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval.  In CVPR, 2020.
  * [128] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  In NeurIPS, 2014.
  * [129] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  arXiv preprint arXiv:1506.03365, 2015.
  * [130] Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, and Chen-Change Loy.  Sketch me that shoe.  In CVPR, 2016.
  * [131] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese.  Taskonomy: Disentangling task transfer learning.  In CVPR, 2018.
  * [132] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [133] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler.  Datasetgan: Efficient labeled data factory with minimal human effort.  In CVPR, 2021.
  * [134] Miaoyun Zhao, Yulai Cong, and Lawrence Carin.  On leveraging pretrained gans for generation with limited data.  In ICML, 2020.
  * [135] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han.  Differentiable augmentation for data-efficient gan training.  In NeurIPS, 2020.
  * [136] Liang Zheng, Yi Yang, and Qi Tian.  Sift meets cnn: A decade survey of instance retrieval.  IEEE TPAMI, 2017.
  * [137] Jiapeng Zhu, Ruili Feng, Yujun Shen, Deli Zhao, Zhengjun Zha, Jingren Zhou, and Qifeng Chen.  Low-rank subspaces in GANs.  In NeurIPS, 2021.
  * [138] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros.  Generative visual manipulation on the natural image manifold.  In ECCV, 2016.
  * [139] Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka.  Barbershop: Gan-based image compositing using segmentation masks.  ACM TOG, 2021.
  * [140] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka.  Improved stylegan embedding: Where are the good latents?  arXiv preprint arXiv:2012.09036, 2020.

Appendix

##  A Implementation details

Derivation for the 1st Moment method. In the main text, we derived a Monte-
Carlo estimation as a score function for text-based model retrieval as
follows:

$\displaystyle\max_{\theta\in\\{\theta_{1},\dots,\theta_{N}\\}}\mathbb{E}_{h_{x}\sim
p(h_{x}|\theta)}\left[\exp\left(\frac{\tilde{h_{x}}^{T}\tilde{h_{q}}}{\tau}\right)\right],$
(10)

where $h_{x}\coloneqq\phi_{\text{im}}(x)$ and
$h_{q}\coloneqq\phi_{\text{txt}}(q)$ are the image and text feature from CLIP,
respectively. $\tilde{h_{x}}=\frac{h_{x}}{||h_{x}||}$ and
$\tilde{h_{q}}=\frac{h_{q}}{||h_{q}||}$ are the normalized features.

To make computation more efficient, we provide ways to approximate Equation
10. First, we derive a point estimation at the first moment of the model
distribution. Specifically, we estimate the mean of the normalized CLIP image
features. (As discussed in the main text, since CLIP features are learned to
be magnitude-invariant, we normalize CLIP features in all of our methods.)
Here, we can approximate the feature distribution using a Dirac delta
function, where $p(h_{x}|\theta)\sim\delta(h_{x}-\mu_{n})$. $\mu_{n}$ is the
first moment of the distribution, where $\mu_{n}=\mathbb{E}_{h_{x}\sim
p(h_{x}|\theta)}\left[h\right]$. With this approximation, we can rewrite
Equation 10 as:

$\displaystyle\max_{\theta\in\\{\theta_{1},\dots,\theta_{N}\\}}\exp\left(\frac{\tilde{\mu_{n}}^{T}\tilde{h_{q}}}{\tau}\right)$
(11) where $\displaystyle\mu_{n}=\left(\mathbb{E}_{h_{x}\sim
p(h_{x}|\theta)}\left[h\right]\right);\;\tilde{\mu_{n}}=\frac{\mu_{n}}{||\mu_{n}||}$

Since the exponential mapping and temperature mapping is a monotonically
increasing function, we can further simplify the expression to be our 1st
Moment method in the main text:

$\displaystyle\max_{\theta\in\\{\theta_{1},\dots,\theta_{N}\\}}\tilde{\mu_{n}}^{T}\tilde{h_{q}}$
(12) $\displaystyle\text{where}\;\;\mu_{n}=$
$\displaystyle\left(\mathbb{E}_{h_{x}\sim
p(h_{x}|\theta)}\left[h\right]\right);\;\tilde{\mu_{n}}=\frac{\mu_{n}}{||\mu_{n}||}$

Derivation for the 1st \+ 2nd Moment method. Now we consider both the first
and second moment of the distribution of the normalized CLIP features. Since
all CLIP features $h_{x}$ are viewed as normalized, it is true that
$h_{x}=\tilde{h_{x}}$ and
$p(\tilde{h_{x}}|\theta)=p(h_{x}|\theta)\sim\mathcal{N}(\mu_{n},\Sigma_{n})$.
Hence, we can write Equation 10 as follows:

$\begin{split}&\mathbb{E}_{\tilde{h_{x}}\sim
p(\tilde{h_{x}}|\theta)}\left[\exp\left(\frac{\tilde{h_{x}}^{T}\tilde{h_{q}}}{\tau}\right)\right]\\\
&=(2\pi|\Sigma_{n}|)^{-\frac{1}{2}}\int\exp{\left(\frac{\tilde{h_{x}}^{T}\tilde{h_{q}}}{\tau}-\frac{1}{2}(\tilde{h_{x}}-\mu_{n})^{T}\Sigma_{n}^{-1}(\tilde{h_{x}}-\mu_{n})\right)}d\tilde{h_{x}}\\\
&=(2\pi|\Sigma_{n}|)^{-\frac{1}{2}}\cdot\\\
&\;\int\exp{\left(-\frac{1}{2}\tilde{h_{x}}^{T}\Sigma_{n}^{-1}\tilde{h_{x}}+\tilde{h_{x}}^{T}(\Sigma_{n}^{-1}\mu_{n}+\frac{\tilde{h_{q}}}{\tau})-\frac{1}{2}\mu_{n}^{T}\Sigma_{n}^{-1}\mu_{n}\right)}d\tilde{h_{x}}\\\
&=\exp\left(\frac{1}{2\tau^{2}}\tilde{h_{q}}^{T}\Sigma_{n}\tilde{h_{q}}+\frac{1}{\tau}\mu_{n}^{T}\tilde{h_{q}}\right).\end{split}$
(13)

The last step is done by completing the square term, and using the fact that
the integral of a Guassian density function equals to 1.

Again, since the exponential function is an increasing function, we can
rewrite the objective as:

$\displaystyle\max_{\theta\in\\{\theta_{1},\dots,\theta_{N}\\}}\frac{1}{2\tau^{2}}\tilde{h_{q}}^{T}\Sigma_{n}\tilde{h_{q}}+\frac{1}{\tau}u_{n}^{T}\tilde{h_{q}}$
(14) where $\displaystyle
p(h_{x}|\theta)\sim\mathcal{N}(\mu_{n},\Sigma_{n})\;\;\text{for}\;n\in\\{1,\dots,N\\}.$

We note that we treat $h_{x}=\tilde{h_{x}}$, so
$p(h_{x}|\theta)=p(\tilde{h_{x}}|\theta)$. Since the temperature $\tau$ is
positive, we can further simplify the experssion to be:

$\displaystyle\max_{\theta\in\\{\theta_{1},\dots,\theta_{N}\\}}\frac{1}{2\tau}\tilde{h_{q}}^{T}\Sigma_{n}\tilde{h_{q}}+u_{n}^{T}\tilde{h_{q}}$
(15) where $\displaystyle
p(h_{x}|\theta)\sim\mathcal{N}(\mu_{n},\Sigma_{n})\;\;\text{for}\;n\in\\{1,\dots,N\\}.$