# TAP: Targeted Prompting for task adaptive generation of textual training
instances for visual classification

###### Abstract

Vision and Language Models (VLMs), such as CLIP, have enabled visual
recognition of a potentially unlimited set of categories described by text
prompts. However, for the best visual recognition performance, these models
still require tuning to better fit the data distributions of the downstream
tasks, in order to overcome the domain shift from the web-based pre-training
data. Recently, it has been shown that it is possible to effectively tune VLMs
without any paired data, and in particular to effectively improve VLMs visual
recognition performance using text-only training data generated by Large
Language Models (LLMs). In this paper, we dive deeper into this exciting text-
only VLM training approach and explore ways it can be significantly further
improved taking the specifics of the downstream task into account when
sampling text data from LLMs. In particular, compared to the SOTA text-only
VLM training approach, we demonstrate up to $8.4\%$ performance improvement in
(cross) domain-specific adaptation, up to $8.7\%$ improvement in fine-grained
recognition, and $3.1\%$ overall average improvement in zero-shot
classification compared to strong baselines.

Index Terms—  Vision and Language Models, VLM, text-only training, zero-shot
visual classification

00footnotetext: †Correspondence<EMAIL_ADDRESS>
##  1 Introduction

Vision and Language Models (VLMs) pre-trained on large-scale paired image-text
datasets collected from the web (e.g. CLIP [1]) have made unprecedented
progress in recent years, enabling open-vocabulary zero-shot recognition,
closely rivaling closed-set (fixed classes set) pre-trained models. The
typical ‘generalist’ dual-encoder VLM zero-shot approach for (downstream)
image classification uses a collection of human-engineered prompts that are
encoded by the VLM text-encoder and compared to the image in question encoded
by the VLM image-encoder, determining the image’s class by maximal average
similarity. However, as noted by several works [2, 3], for better performance,
this VLM-based classifier needs to be tuned beyond the engineered prompts.
Since dataset-specific tuning data (paired image-text) is expensive to
collect, self-supervised [4, 5] and text-supervised [5] VLM fine-tuning
approaches were proposed. These approaches do not use any paired data and
instead successfully leverage unpaired uni-modal data, such as unlabeled image
(or video) collections [5, 4] and text-only samples generated from a strong
Large Language Model (LLM) [4], effectively tapping on its visual world
knowledge. In particular, LaFTer [5] proposes to substitute visual instances
by prompting the LLM to generate textual descriptions of categories in the
downstream dataset and uses the generated samples as training data for a VLM-
based visual classifier.

Fig. 1: Incorporating the visual characteristics of the downstream task with
TAP. Task-specific LLM prompts provide multiple samples of dataset-specific
class descriptions. These are used to train a text-based classifier, tasked
with predicting the corresponding class names. Finally, this pre-trained text-
only classifier is directly applied to classify visual data, leveraging the
shared image-text embedding space learned by the VLMs. At runtime, TAP applies
all possible targeted prompt templates to all target task classes equally.
This example illustrates TAP for the ImageNet-Rendition [6] dataset.

The past approaches [4, 5], still employ a somewhat generalist strategy for
leveraging the uni-modal data, employing the same technique regardless of the
nature of the downstream (target) task, and in particular, regardless of the
visual properties of the target objects of interest. Consequently, while
demonstrating strong improvements, these methods are not able to take
advantage of the full potential of the visual knowledge possessed by modern-
day strong LLMs and image-text-aligned representation spaces of VLMs. While
the content of the unlabeled image collections (as used in [4, 5]) is hard to
control, in contrast the generation of the text-only samples (textual
descriptions of objects in the downstream task) for VLM-based visual
classifier training can be effectively manipulated using the knowledge about
the visual properties of the downstream (target) task. In this paper, we
demonstrate for the first time a method and empirical evidence that specific
downstream targeting is possible while generating these text-only training
samples. Specifically, we show significant performance improvements when
incorporating the textual descriptions of the visual characteristics of the
downstream tasks into the prompts for the LLM that are used to generate the
text-only samples. As illustrated in Figure 1, these generated samples are in
turn used to learn a text classifier (trained to classify textual descriptions
to class names), which can be readily applied to classify the visual data,
foregoing the constraint to obtain paired image-text data.

In this work, we identify several types of domain shifts (from the large-scale
VLM, e.g., CLIP pre-training) arising while transferring the VLM-based zero-
shot visual classifier to downstream applications, and propose a simple
principled approach for generating the text-only samples for each type. Our
key idea is Targeted-Prompting (TAP): prompting the LLM to explicitly specify
the desired visual characteristics of objects to highlight while generating
the text-only training samples. TAP leverages the _visual world knowledge_ and
_domain expertise_ acquired by LLMs during their large-scale pre-training on
trillions-of-tokens-scale internet data. The samples generated via our TAP
exhibit greater visual detail in aspects relevant to the expected domain shift
corresponding to the target downstream task and consequently lead to greater
improvement when used for finetuning. Our simple targeted prompting strategy
results in significant performance gains of the visual VLM-based classifier
compared to a collection of strong baselines including the base VLM, its
adaptation and SOTA prior work ([5]) for text-only VLM finetuning.

##  2 Related work

VLMs pre-trained on large-scale web data (e.g., CLIP [1]) show strong zero-
shot performance for recognition tasks. Recently, many interesting extensions
to improve the image-text representation alignment have been proposed [7, 8,
9, 10, 11, 12]. Parallel works have explored VLM improvements with: extra
supervision [13, 14], fine-grained interactions [15], modern Hopfield networks
[10], optimal transport distillation [16], cycle consistency [17], and
hierarchical feature alignment [18]. Recent works, such as [2, 3, 19]
demonstrate that despite the great progress in zero-shot recognition tasks,
VLM performance on downstream tasks can be significantly improved with
supervised fine-tuning on annotated data. On the other hand, UPL [20], CLIP-PR
[21], MAXI [4], and LaFTer [5] show that improvements are also possible with
finetuning using unlabeled data. UPL finetunes learnable text prompts (similar
to [2]) by relying on confidence sampled pseudo-labeling, whereas CLIP-PR
relies both on offline pseudo-labeling and label distribution prior from the
training data. MAXI focuses only on zero-shot action recognition tuning on
unlabeled video data and rich text dictionaries for matching and pseudo-
labeling. LaFTer combines tuning on text samples generated from LLMs using
generic (non-task-specific) prompts and pseudo-labeled (online FixMatch [22]
style) image collection. In our work, we show that not only the text-only LLM-
generated samples can be used to tune VLM-based visual classifiers (in
contrast to [20, 21, 4]), but also that (in contrast to LaFTer [5]) those
text-samples can be manipulated in a principled manner to become downstream
task-adaptive, leading to significant performance improvements (e.g., over
[5]). Extensive empirical evaluations show the benefits of our approach w.r.t.
all the leading label-free finetuning baselines.

##  3 TAP: Targeted Prompting

CLIP [1] is trained on a large corpus of paired text and image data collected
from various sources on the internet. It consists of visual and text encoders
that project textual and visual data into a high-dimensional space. The
encoders are trained using a contrastive loss, which encourages embeddings of
same text-image pairs to be closer in the embedding space while pushing apart
embeddings of texts and images belonging to different pairs. The shared text-
image embedding space learned by CLIP (due to the contrastive loss) enables
effective zero-shot image classification. To achieve this, the test image $x$
is encoded by the vision encoder $v$ to a latent space. Similarly, the text
encoder $u$ is used to generate the embedding $u_{c}$ for a text prompt for
each class $c\in C$, where $C$ represent the set of class names for the
respective downstream dataset. The prompts can be of the form ‘ _A photo of
a{c}_’. The zero-shot classification is performed by calculating the cosine
similarity of the visual embedding with the text embeddings of each class.
More formally, the probability for a predicted class $\hat{c}$ is computed
with the temperature-scaled softmax,

$l_{\hat{c}}(x)=\frac{e^{\cos(u_{\hat{c}},v(x))/{\tau}}}{\sum_{c\in
C}e^{\cos(u_{c},v(x))/{\tau}}}.$ (1)

In [1] it was highlighted that obtaining the text embeddings by using
different prompting strategies can have an effect on the eventual zero-shot
classification performance. Recent works, such as CuPL [23] and LaFTer [5]
show that by generating the textual embeddings from generic descriptions of
the categories in the dataset (produced by an LLM in response to a generic
prompt, such as Describe an image of a _{ category}_) can improve the zero-
shot classification results. In contrast, in this work we propose to replace
visual instances by generating text-only class descriptions by prompting an
LLM in a _targeted_ manner that allows taking the visual characteristics of
the downstream task into account, leading to significantly better
classification performance than any of the previously proposed (human
engineered or LLM produced) generic class prompts. In Section 3.1, we describe
our LLM TArgeted Prompting (TAP) strategy, and in Section 3.2 we provide
details about how the generated descriptions can be used to replace visual
instances by training a text classifier which can effectively classify visual
data, leveraging cross-modal transfer capabilities of CLIP.

###  3.1 Targeted Prompts to LLM

Previous works [23, 5] have highlighted the significance of generating
descriptions of categories in a dataset (through an LLM) and using those
descriptions (instead of simple templates provided by CLIP [1]) to obtain the
text embeddings for improved zero-shot classification. Our TAP also relies on
extracting the visual knowledge from the LLM for improved zero-shot
classification. However, contrary to the previous approaches we propose to
prompt the LLM in a _targeted_ manner. While [23, 5] use the same generic
prompts for generating (visual scene) descriptions for improving zero-shot
performance for all the downstream datasets, in TAP we propose and show for
the first time the surprising effectiveness of adapting the prompts (for
generating descriptions) to the downstream dataset for which we need to
perform the zero-shot classification. This allows TAP to exploit the
additional LLM knowledge about visual properties (characteristics) of class
images of the downstream task that do not surface in the output when using
generic prompts. For example, the specific characteristics of renditions,
other imaging modalities (such as satellite imagery), or fine-grained
distinctions between very similar classes.

We have identified two principled targeted prompting strategies that make TAP
effective. These two strategies are focused on addressing the two prevalent
domain shifts appearing in many downstream tasks, namely: (i) cross-visual-
domain shifts between natural imagery (of VLM web pre-training) to various
kinds of renditions or other imaging modalities (e.g., microscopes or
satellites), and (ii) fine-grained shifts, when the classifier sensitivity to
fine-grained distinctions between similar downstream task classes needs to be
enhanced. Surprisingly, we found that simple targeted prompting strategies
work best for both of those important cases.

| Generic Prompting [23, 5] | Targeted Prompting
---|---|---
DTD |  P: Describe what a banded looks like. |  P: Describe what a banded texture looks like.
D: Banded snakes have yellow or white stripes |  D: A banded texture has distinct lines or stripes of
wrapping around their body. | contrasting colors within same color family.
P: How can you identify a braided. |  P: Describe what a braided texture looks like.
D: A braided hairstyle can be identified by its |  D: Braided texture can be identified by its
interwoven sections of hair. | woven-like appearance and feel.
Flowers-102 |  P: Describe what an artichoke looks like. |  P: Describe what an artichoke flower looks like.
D: An artichoke is a large, round, and thistle-like |  D: An artichoke is a large flower with a spiny
vegetable. | shape and thick, tough leaves.
P: How can you identify a fritillary? |  P: How can you identify a fritillary flower?
D: Fritillary butterfly is identified by its checked |  D: Fritillary has distinctive bell or pendulous
pattern on the upper side of their wings. | shape, usually with six petals.
EuroSat |  P: Describe what a forest looks like. |  P: Describe what a forest looks like from a satellite.
D: A forest is an area of land covered with trees, |  D: From a satellite, a forest is likely to show a
bushes, and other vegetation. | patchwork of varying shades of green.
P: How can you identify a river? |  P: How can you identify a river from a satellite?
D: Rivers can be identified by looking for long, |  D: To identify a river from a satellite, look for
interwoven sections of hair. | features like large meanders and wide curves.
Table 1: Comparing generic and TAP prompts. Showing two example LLM prompts
and resulting descriptions (from GPT-3.5, DaVinci-003) for two classes of
Describable Textures (DTD) [24], Flowers-102 [25] and EuroSat [26] datasets.

For fine-grained shifts, we discovered that providing super-class context is
of great importance for the LLM to produce better fitting textual description
instances. Table 1 compares the generic prompting adopted by [23, 5] with our
Targeted Prompting strategy. Using generic prompts, it is difficult for the
LLM to relate fine-grained categories to their super-class. For example, for
_texture_ classification on DTD [24], the LLM confuses banded as a type of
_snake_. Similarly, without the super-class context, the LLM wrongly describes
the braided texture as a _hairstyle_. Due to the same word used in the English
language for categories belonging to totally un-related super-classes, the LLM
provides erroneous descriptions for categories in Flowers-102 [25] dataset as
well. For example, the generic prompting results in the description of the
fritillary  _butterfly_ instead of the  _flower_. To obtain descriptions
correlated with the downstream task, our targeted prompts provide the LLM with
the additional context that it needs to describe braided as a form of
_texture_ and fritillary as a form of  _flower_.

Our strategy for cross-visual-domain shifts is illustrated in Figure 1 for
ImageNet-R [6] and in Table 1 for EuroSat [26]. We have found that LLMs
generate effective target domain descriptions if explicitly prompted via short
descriptions of the visual characteristics of the domain of interest. For
example, for rendition domains, we can specify types of renditions of
interest, requesting an LLM to produce description instances for a Cartesian
product between the set of classes and the set of rendition types. For
satellite or microscopy imagery we specifically request descriptions in those
imaging domains. For action videos (UCF-101 [27]), we ask the LLM to highlight
the temporal and motion aspects prompting for action descriptions. For,
scenery (SUN-397 [28]) we ask the LLM to focus the background features of the
visual scene asking for scene descriptions. All targeted prompts and
descriptions for the different types of domain shifts are included in our
complete codebase available at https://github.com/jmiemirza/TAP.

###  3.2 Cross-Modal Transfer

The Vision Language models have an aligned text and image embedding space due
to the contrastive learning objective between the images and the alt-text
pairs, used for pre-training. This allows these models to be used as an
effective zero-shot classifier. Due to the shared embedding space learned by
these multi-modal models, recently, DrML [29] shows that the behavior of
visual classifiers can be rectified by using textual data only. Furthermore,
LaFTer [5] highlights another advantage of the shared embedding space, where
it firsts train a classifier to classify (multiple samples of) LLM-generated
text descriptions of classes to the correct class names in the dataset and
then applies the trained classifier to classify visual embeddings,
highlighting effective cross-modal transfer.

| Describable Textures | Flowers-102 | ImageNet-Rendition | FGVC-Aircraft | UCF-101 | Food-101 | EuroSat | SUN-397 | Mean
---|---|---|---|---|---|---|---|---|---
CLIP (single) | 40.3 | 64.0 | 65.8 | 18.1 | 61.0 | 77.1 | 35.9 | 60.8 | 52.8
TOT (cls-only) | 37.4 | 60.4 | 63.1 | 15.4 | 56.4 | 74.3 | 39.1 | 56.3 | 50.3
CLIP (DST) | 42.4 | 66.6 | 68.6 | 19.5 | 62.4 | 79.3 | 45.8 | 61.7 | 55.8
TOT (DST) | 41.6 | 64.5 | 67.5 | 17.2 | 62.0 | 77.2 | 43.6 | 61.0 | 54.3
LaFTer (text-cls) | 44.3 | 64.2 | 63.6 | 18.3 | 63.7 | 78.1 | 44.2 | 60.7 | 54.6
TAP | 51.1 | 66.9 | 68.9 | 21.6 | 66.3 | 78.5 | 54.2 | 63.4 | 58.9
Table 2:  Top-1 Classification Accuracy (%) for a ViT-B/32 CLIP pre-trained network. CLIP denotes the zero-shot evaluation performed by using either single prompt template or a set of dataset-specific prompt templates (DST). Text-Only Training (TOT) results are obtained by training a text classifier to classify either the class names only or the DST texts. For a fair comparison, we also compare with results from text-only pre-trained visual classifier in LaFTer [5]. The highest accuracy is shown in bold, while the second best is underlined.  | Flowers-102 | UCF-101 | DTD | IN-R | Mean
---|---|---|---|---|---
CLIP | 66.6 | 62.4 | 42.4 | 68.6 | 60.0
CLIP-PR | 60.1 | 59.7 | 45.1 | 56.9 | 55.5
UPL | 71.5 | 63.9 | 48.0 | 65.6 | 62.3
LaFTer | 71.0 | 68.2 | 46.1 | 72.6 | 64.5
TAP∗ | 72.4 | 71.4 | 51.6 | 73.9 | 67.3
Table 3: Top-1 Classification Accuracy (%). TAP∗ result is obtained by further
finetuning the vision encoder of CLIP, with pseudo-labels generated for an
unlabeled image collection by the TAP text-classifier, following the complete
LaFTer [5] pipeline.

Cross-Modality transfer introduces an efficient way to train neural networks
on relatively cheap text data and avoid the manual labor of acquiring and
annotating images. A large quantity of text data can be conveniently harnessed
by prompting the LLM for descriptions of classes, which can effectively
replace visual instances. This is also a relatively easier alternative to
mining text data, since LLMs already represent a large corpus of text on which
they were trained. In this work, we take inspiration from [5] and also train a
classifier on a textual dataset constructed by generating descriptions of
classes. However, contrary to LaFTer [5], we generate descriptions for the
downstream dataset by explicitly prompting the LLM in a targeted manner as
described in Section 3.1. Moreover, unlike LaFTer, our TAP does not rely on
generating the text data by mixing hand-crafted templates from [1] with the
text descriptions, instead, we use the raw descriptions generated by the LLM
without requiring any post-processing.

Fig. 2: The targeted descriptions generated by the LLM are automatically
matched with the class names to generate the text dataset. This dataset is
used to train a text classifier in a _supervised_ setting (top). The trained
text classifier is used to classify the visual data (bottom).

The class descriptions from the LLM are automatically matched with the class
name and training the text classifier can be rendered as a supervised learning
problem, as depicted in Figure 2. Applying the trained text classifier to
classify visual embeddings also allows us to forego constraints that are being
imposed by the classical (VLM) zero-shot classification setup, according to
Eq. (1). In contrast, for TAP, to obtain the class likelihoods for the visual
classification, we can directly employ the trained text classifier $f$ instead
of computing the cosine similarity between embeddings from different
modalities. More formally, given a text dataset $T$, consisting of class
descriptions $t$ and their corresponding class labels $c$, the text classifier
is trained with the following _supervised_ training objective:

$\min_{\theta}\sum_{\begin{subarray}{c}(t,c)\in T\\\
n\sim\mathcal{N}(0,I)\end{subarray}}\mathcal{L}_{\textsc{SCE}}\big{(}f_{\theta}(\frac{u(t)}{\lVert
u(t)\rVert}+n),c\big{)},$ (2)

where $\theta$ is the parameter of the classifier,
$\mathcal{L}_{\textsc{SCE}}$ represents the Smoothened Cross Entropy loss [30]
and Gaussian Noise $n\sim\mathcal{N}(0,I)$ is added to the text embeddings to
regularize training.

##  4 Experimental Evaluation

###  4.1 Evaluation Details

#### Datasets

We extensively evaluate our TAP on $8$ different datasets which include
several fine-grained classification datasets, such as Flowers-102 [25], FGVC
Aircraft [31], Food-101 [32] and Describable Textures Dataset (DTD) [24], and
also domain-specific classification datasets such as UCF-101 [27], EuroSat
[26], SUN-397 [28] and ImageNet-Rendition (IN-R) [6]. These groups exhibit the
two prominent domain shifts from VLM (e.g., CLIP [1]) web-data pre-training to
downstream zero-shot application, as discussed in Section 3.1.

#### Baselines

We compare our method with several strong baselines:

  * •

CLIP [1]:  represents the zero-shot evaluations performed by using the base
CLIP network. We evaluate CLIP by using a single prompt template ‘ _a photo of
a{c}_’ and also the full set of dataset-specific templates (denoted DST)
provided by [1] for a fair evaluation.

  * •

Text-Only Training (TOT):  represents the results obtained by training a
classifier (similar to our TAP), but with two different training objectives –
classifying the class name; TOT (cls-only), and classifying DST to the class
names; TOT (DST).

  * •

LaFTer [5]:  trains a text classifier on LLM descriptions and dataset-specific
templates from [1] and later employs unsupervised fine-tuning of the visual
encoder via pseudo-labeled visual data.

#### Implementation Details

For all experiments, we use the OpenAI pre-trained ViT-B/32 CLIP [1] model. To
ensure a fair comparison with [5], the text classifier (Section 3.2) is
implemented as a single fully connected layer, where the number of the output
units of the layer equals the number of classes in the downstream dataset.
Similarly, following [5] we use GPT-3.5 DaVinci-003 to generate the
descriptions. The text dataset is loaded as a single batch and the classifier
is optimized using AdamW, with a learning rate of $0.001$. All details can be
found in our codebase: https://github.com/jmiemirza/TAP

###  4.2 Results

Table 2 shows the comparison against (unadapted) CLIP and baselines which do
not require visual instances to finetune the VLM. Our TAP consistently
improves over the zero-shot CLIP baseline (with a single ‘a photo of a …’
template) on all the $8$ datasets, showing up to $18.3\%$ improvement on
EuroSat ($6.1\%$ on average). On IN-R our TAP improves the base CLIP by
$3.3\%$, which is an out-of-distribution variant of the original ImageNet [33]
dataset. This improvement highlights that generating text descriptions can be
helpful in mitigating the distribution shifts from the text side, which are
automatically transferred to the vision domain due to the shared embedding
space.

We also compare our TAP with a stronger CLIP baseline, where text embeddings
are generated by using a collection of dataset-specific templates, also
referred to as _prompt ensembling_ in [1]. Our TAP still shows $3.1\%$ average
gains over the $8$ datasets. Individually, our TAP improves over $7$ out of
$8$ datasets, while remaining competitive on Food-101, where only $0.3\%$
degradation is observed. These results highlight the benefits of cross-modal
transfer through TAP, where training a text-classifier (on texts automatically
generated by an LLM, without requiring any labeled image data) to classify
visual data can be a better alternative for zero-shot classification than
obtaining the class likelihoods through cosine similarity.

Our TAP also performs favorably when compared with methods that train a text-
classifier on text datasets generated through different methods. For example,
while training the classifier to only classify the class names or dataset-
specific templates to the class names, TOT (cls-only) and TOT (DST) in Table
2, our TAP shows an average gain of $8.6\%$ and $4.6\%$. Moreover, while
comparing with text-only pre-training of a classifier through LaFTer, where
the text dataset is constructed by combining dataset-specific templates and
(generic) LLM descriptions of classes, we show an average improvement of
$4.3\%$. The gains over LaFTer further highlights the importance of generating
targeted descriptions with targeted prompting (TAP) as compared to generating
descriptions with a generic prompting technique (as used in LaFTer).

###  4.3 Ablation Study

In their second stage, LaFTer [5] also leverages the unlabeled visual
instances and finetunes the visual encoder from CLIP by using the pseudo-
labels generated by the text-only pre-trained visual classifier (inspired from
FixMatch [22]). The quality of the generated pseudo-labels depends on the pre-
trained text classifier. Thanks to the targeted descriptions from the LLM,
since our TAP’s text-only classifier provides better results than the LaFTer’s
text-only pre-training (Table 2), TAP can be employed seamlessly in the second
stage of LaFTer (hereby denoted as TAP∗) to provide a further improvement over
the full LaFTer pipeline. In Table 3 we compare TAP∗ with LaFTer [5], CLIP-PR
[21], and UPL [20] \- all of which use unlabeled image data for unsupervised
finetuning. Compared to the leading baselines (LaFTer / UPL) TAP∗ obtains up
to $3.6\%$ gain on the DTD dataset, while on average being $7.3\%$ and $2.8\%$
better than CLIP and the best baseline (LaFTer) on the $4$ datasets.

##  5 Conclusion and Discussion

We have introduced TAP - a simple and effective approach for leveraging LLM’s
detailed visual world knowledge to improve zero-shot classification in VLMs.
Our approach only requires knowledge of the class names and the general visual
characteristics of the target downstream task to make the VLM-based visual
classifier more effective in zero-shot recognition on the target task when
compared to a variety of baselines. Our work may lead to a number of exciting
follow-ups; currently, TAP requires some basic human knowledge of visual
characteristics in the downstream task to prompt the LLM to generate targeted
textual scene instances, later used for text-only pre-training. As future
work, it might also be possible to obtain these visual characteristics by
automatically prompting the LLM itself, effectively creating a self-contained
and almost no-human in-the-loop two-stage approach. TAP can also be naturally
extended to mitigate other domain shifts, such as: adversarial and outlier
robustness, long tail category distributions, unbalanced class hierarchies,
and more.

## References

  * [1] Alec Radford et al.,  “Learning Transferable Visual Models from Natural Language Supervision,”  in Proc. ICML, 2021.
  * [2] Kaiyang Zhou et al.,  “Learning to Prompt for Vision-Language Models,”  IJCV, 2022.
  * [3] Kaiyang Zhou et al.,  “Conditional Prompt Learning for Vision-Language Models,”  in Proc. CVPR, 2022.
  * [4] Wei Lin et al.,  “MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge,”  arXiv:2303.08914, 2023.
  * [5] M Jehanzeb Mirza et al.,  “LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections,”  arXiv:2305.18287, 2023.
  * [6] Dan Hendrycks et al.,  “The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization,”  in Proc. ICCV, 2021.
  * [7] Junnan Li et al.,  “Align before Fuse: Vision and Language Representation Learning with Momentum Distillation,”  arXiv:2107.07651, 2021.
  * [8] Jinyu Yang et al.,  “Vision-Language Pre-Training with Triple Contrastive Learning,”  in Proc. CVPR, 2022.
  * [9] Junnan Li et al.,  “BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation,”  arXiv:2201.12086, 2022.
  * [10] Andreas Fürst et al.,  “Cloob: Modern Hopfield Networks with InfoLOOB Outperform Clip,”  arXiv:2110.11316, 2021.
  * [11] Yangguang Li et al.,  “Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm,”  arXiv:2110.05208, 2021.
  * [12] Peng Gao et al.,  “Clip-adapter: Better Vision-language Models with Feature Adapters,”  arXiv:2110.04544, 2021.
  * [13] Yangguang Li et al.,  “Supervision Exists Everywhere: A data Efficient Contrastive Language-Image Pre-training Paradigm,”  arXiv:2110.05208, 2021.
  * [14] Norman Mu et al.,  “Slip: Self-supervision Meets Language-image Pre-training,”  arXiv:2112.12750, 2021.
  * [15] Lewei Yao et al.,  “Filip: Fine-grained Interactive Language-image Pre-training,”  arXiv:2111.07783, 2021.
  * [16] Bichen Wu et al.,  “Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation,”  arXiv:2112.09445, 2021.
  * [17] Shashank Goel et al.,  “CyCLIP: Cyclic Contrastive Language-Image Pretraining,”  arXiv:2205.14459, 2022.
  * [18] Yuting Gao et al.,  “PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining,”  arXiv:2204.14095, 2022.
  * [19] Muhammad Uzair Khattak et al.,  “Maple: Multi-modal Prompt Learning,”  in Proc. CVPR, 2023.
  * [20] Tony Huang et al.,  “Unsupervised Prompt Learning for Vision-Language Models,”  arXiv:2204.03649, 2022.
  * [21] Jonathan Kahana et al.,  “Improving Zero-Shot Models with Label Distribution Priors,”  arXiv:2212.00784, 2022.
  * [22] Kihyuk Sohn et al.,  “Fixmatch: Simplifying Semi-supervised Learning with Consistency and Confidence,”  in NeurIPS, 2020.
  * [23] Sarah Pratt et al.,  “What does a Platypus Look Like? Generating Customized Prompts for Zero-shot Image Classification,”  in Proc. ICCV, 2023.
  * [24] Mircea Cimpoi et al.,  “Describing Textures in the Wild,”  in Proc. CVPR, 2014.
  * [25] Maria-Elena Nilsback and Andrew Zisserman,  “Automated Flower Classification Over a Large Number of Classes,”  in Proc. ICVGIP, 2008.
  * [26] Patrick Helber et al.,  “Introducing EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification,”  in Proc. IGARSS, 2018.
  * [27] Khurram Soomro et al.,  “UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild,”  arXiv:1212.0402, 2012.
  * [28] Jianxiong Xiao et al.,  “SUN Database: Large-scale Scene Recognition from Abbey to Zoo,”  in Proc. CVPR, 2010.
  * [29] Yuhui Zhang et al.,  “DrML: Diagnosing and Rectifying Vision Models using Language,”  in Proc. ICLR, 2023.
  * [30] Christian Szegedy et al.,  “Rethinking the Inception Architecture for Computer Vision,”  in Proc. CVPR, 2016.
  * [31] Subhransu Maji et al.,  “Fine-grained Visual Classification of Aircraft,”  arXiv:1306.5151, 2013.
  * [32] Lukas Bossard et al.,  “Food-101–Mining Discriminative Components with Random Forests,”  in Proc. ECCV, 2014.
  * [33] Jia Deng et al.,  “Imagenet: A Large-scale Hierarchical Image Database,”  in Proc. CVPR, 2009.