Tree of Attributes Prompt Learning for Vision-Language Models
Under review as a conference paper at ICLR 2025

TREE OF ATTRIBUTES PROMPT LEARNING FOR VISION-
LANGUAGE MODELS

Anonymous authors
Paper under double-blind review

ABSTRACT

Prompt learning has proven effective in adapting vision language models for
downstream tasks. However, existing methods usually append learnable prompt
tokens solely with the category names to obtain textual features, which fails to
fully leverage the rich context indicated in the category name. To address this issue,
we propose the Tree of Attributes Prompt learning (TAP), which first instructs
LLMs to generate a tree of attributes with a “concept - attribute - description”
structure for each category, and then learn the hierarchy with vision and text prompt
tokens. Unlike existing methods that merely augment category names with a set of
unstructured descriptions, our approach essentially distills structured knowledge
graphs associated with class names from LLMs. Furthermore, our approach
introduces text and vision prompts designed to explicitly learn the corresponding
visual attributes, effectively serving as domain experts. Additionally, the general
and diverse descriptions generated based on the class names may be wrong or
absent in the specific given images. To address this misalignment, we further
introduce a vision-conditional pooling module to extract instance-specific text
features. Extensive experimental results demonstrate that our approach outperforms
state-of-the-art methods on the zero-shot base-to-novel generalization, cross-dataset
transfer, as well as few-shot classification across 11 diverse datasets.

1

INTRODUCTION

Recent advancements in vision-language models (VLMs) like CLIP (Radford et al., 2021) and
ALIGN (Jia et al., 2021) merge the capabilities of visual perception with linguistic understanding,
which have revolutionized the landscape with their zero-shot learning abilities. They proficiently
handle tasks on unseen data, bypassing the conventional requirement for task-specific training. This
feature has enabled a plethora of applications, ranging from content-based image retrieval to complex
visual question answering, setting new benchmarks in the domain. A crucial development in this
domain is the concept of prompt learning, which has significantly influenced both natural language
processing (NLP) (Lester et al., 2021; Li & Liang, 2021; Liu et al., 2021) and vision-only models (Jia
et al., 2022; Wang et al., 2022a;b; Zhang et al., 2022). This approach leverages learnable prompts to
guide model understanding, tailoring responses to specific tasks or datasets.

Prompt learning, particularly in vision-language models, has garnered considerable interest due to its
parameter efficiency and rapid convergence (Zhou et al., 2022b;a; Zhu et al., 2023; Derakhshani et al.,
2023; Lu et al., 2022). Techniques like CoOp (Zhou et al., 2022b) optimize learnable continuous
prompts for few-shot image recognition, enhancing model performance significantly. Recent efforts
have expanded to multimodal prompt learning, optimizing prompts in both visual and language
domains (Khattak et al., 2023a;b; Shi & Yang, 2023; Lee et al., 2023). Despite their success, these
models rely on simplistic text prompts, typically formatted as “a photo of a {class}”, illustrated in
Fig. 1 (a). While functional, this approach lacks depth, failing to encapsulate the intricacies and finer
details inherent in visual data. Such limitations hinder the model’s ability to fully leverage the rich,
descriptive potential offered by more detailed and contextually relevant textual information.

In parallel, another stream of research has been exploring the utilization of large language models
(LLMs) to generate more elaborate and descriptive text prompts for enhancing zero-shot learning
capabilities (Menon & Vondrick, 2023; Pratt et al., 2023; Roth et al., 2023; Kim et al., 2023; Parkhi
et al., 2012; Yan et al., 2023; Yang et al., 2023; Roy & Etemad, 2024; Zheng et al., 2023; Tian

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: Illustration of the methods for CLIP text prompts formation. (a) Manually created prompt
with the single “a photo of a {class}” template; (b) A unstructured set of detailed descriptions
generated by LLMs; (c) The proposed Tree of Attribute distills a knowledge graph from LLMs,
organizing the knowledge in “concept - attribute - descriptions” structure; (d) An example Tree of
Attribute for class “dumplings”, where each color represents a visual attribute.

et al., 2023). These LLM-generated descriptions offer a wealth of detail and context, potentially
enriching the model’s interpretative capabilities. However, current methodologies in integrating these
descriptions often do not exploit the full potential of this richness. As shown in Fig. 1 (b), most of
these approaches lack a structured framework to organize and utilize these descriptions effectively,
leading to a scattergun approach where not all generated descriptions are contextually relevant or
optimally aligned with the visual content. In addition, as noted in (Roth et al., 2023), descriptions
generated by such paradigms are usually diverse, which covers most possibilities of the class, but
include descriptions that are either likely not co-occurring, e.g. “steamed” and “fried”, or absent in
the input image, e.g. “long tail” for a cat shot from the front, necessitating the need for a selective
pooling mechanism for clearer image-text alignments.

In response to these challenges, our work introduces “Tree of Attribute Prompt learning (TAP),”
a method that redefines the integration and utilization of detailed descriptions within VLMs. As
indicated in Fig. 1 (c), unlike existing methods that merely augment category names with a set of
unstructured descriptions, our approach essentially distills structured knowledge graphs associated
with class names from LLMs. Specifically, we adopt a hierarchical, tree-like structure to systemati-
cally generate and integrate descriptions, ensuring a layered and comprehensive understanding of
visual content. Each branch of this tree represents a specific attribute, with finer details fleshed out in
the subsequent leaves, ensuring that every aspect of the visual content is captured and represented.
Furthermore, we reimagine the learnable prompt tokens as “domain experts”, each specializing in
different aspects of the image, supplemented by the CLS token’s global perspective. In addition, we
introduce vision-conditional layers for each expert-attribute pair, which pool the most applicable
descriptions from each of the attribute sets with condition on the input image content, ensuring
optimal image-text alignment. This setup not only provides a detailed, attribute-focused analysis but
also harmonizes these insights with the overall context.

Extensive experiments in base-to-novel generalization, cross-dataset transfer, and few-shot classi-
fication across 11 diverse datasets demonstrate the effectiveness of our method. On base-to-novel
generalization, TAP achieves average performance gains of 1.07% in harmonic mean over the state-
of-the-art methods, and 9.34% over the vanilla CLIP. On cross-dataset transfer, TAP outperforms
existing methods on both source and target datasets by 1.03% and 0.75% in average. Competitive
results are also observed in few-shot classification.

2 RELATED WORK

Prompt Learning for Vision-Language Models. Prompt learning bridges linguistic understanding
and visual perception by guiding VLMs with text prompts, a concept originated in NLP (Lester et al.,
2021; Li & Liang, 2021; Liu et al., 2021) and adapted to vision-only (Jia et al., 2022; Wang et al.,
2022a;b; Zhang et al., 2022) and multimodal contexts(Zhou et al., 2022b;a; Khattak et al., 2023a;b; Shi
& Yang, 2023; Lee et al., 2023; Tian et al., 2023; Rasheed et al., 2023; Roy & Etemad, 2024; Zheng
et al., 2023; Zhu et al., 2023; Bulat & Tzimiropoulos, 2023; Lu et al., 2022). In the textual domain,
CoOp (Zhou et al., 2022b) optimizes learnable continuous prompts in CLIP’s language branch

2

dumplingsExample Tree of A.ributeColor•a pale beige color from the dough exterior.•a golden-brown hue from pan-frying or deep-fryingShape•round with a pleated edge•crescent-shaped, with a fold in the doughTexture•so: and chewy texture from the dough•a crispy texture on the bo;om from pan-frying···Presenta9on•served steamed in a bamboo basket•served with a dipping saucedumplingswrapped in a thin doughboiled, steamed, or friedserved with a dipping sauce···A photo of a dumplingsdumplingsColorShapeTexturePresentaDon···Dis$lled Knowledge GraphUnstructured SetSingle Template🧑🔬A photo of a {class}(a)(b)(c)············(d)Under review as a conference paper at ICLR 2025

for few-shot image recognition, while CoCoOp (Zhou et al., 2022a) addresses CoOp’s overfitting
issues by conditioning prompts on visual features. In the visual domain, Visual Prompt Tuning
(VPT) (Bahng et al., 2022) and Dual-modality Prompt Tuning (DPT) (Xing et al., 2023) enhance
CLIP’s vision encoder by learning visual prompts in pixel space and dynamically generating prompts
through cross-attention, respectively. TransHP (Wang et al., 2023b) leverages category hierarchy for
prompt learning to improve classification performance. LoGoPrompt (Shi & Yang, 2023) enhances
classification by incorporating synthetic images with class name text as auxiliary visual prompts.
MaPLe (Khattak et al., 2023a) explores multimodal prompt learning, jointly optimizing prompts in
both vision and language branches. Other recent works have focused on regularizing prompt learning
to leverage the knowledge from base VLMs effectively, demonstrating enhanced generalization in
varied downstream visual tasks (Khattak et al., 2023b; Bulat & Tzimiropoulos, 2023; Roy & Etemad,
2024). PromptSRC, for instance, introduced a self-regulating method that restricts both the vision and
text prompt, demonstrating improved generalization. Distinct from these approaches, PLOT (Chen
et al., 2023) and ALIGN (Wang et al., 2023a) leverage Optimal Transport to align multiple prompts
with local visual features, either from the multi-head self-attention layer or at a token level. Our work
diverges from these methods by introducing a hierarchical "Tree of Attribute" framework derived
from LLMs to structure textual descriptions and guide the learning of specialized "domain expert"
tokens for attribute-level understanding.

Enhancing model’s understanding using visual attributes. There’s a growing emphasis on the
use of detailed visual descriptions for various visual understanding tasks, including more fine-
grained captioning (Hsieh et al., 2024), identifying subordinate-level categories (Liu et al., 2024a),
and language-guided visual classification (Menon & Vondrick, 2023). While manual creation
is impractical given the large number of image classes, existing research relies either on data
augmentation (Kim et al., 2024) or generation by LLMs such as GPT-3 (Brown et al., 2020), which
offers an efficient generation of a broad spectrum of class-specific descriptions. These descriptions,
like “fur pattern” or “tail shape” of a cat, provide fine-grained and distinctive characteristics. In an
essence, such approaches can be viewed as knowledge distillation from LLMs trained on trained
on vast and diverse textual corpora. However, existing studies often lack a structured methodology
for distillation (Kim et al., 2023; Menon & Vondrick, 2023; Parkhi et al., 2012; Roth et al., 2023;
Yan et al., 2023; Yang et al., 2023; Fabian et al., 2023; Pratt et al., 2023; Novack et al., 2023;
Mao et al., 2023; Tian et al., 2023; Zheng et al., 2023; Zhang et al., 2024; Liu et al., 2024b) or
fail to effectively exploit the inherent hierarchy within the knowledge (Maniparambil et al., 2023;
Wang et al., 2024; Hsieh et al., 2024; Liu et al., 2024a). Our approach (TAP ) addresses these
limitations by introducing a novel method to distill a knowledge graph from LLMs in a top-down
manner, transitioning from class names (concepts) to visual attributes (e.g., color, shape) and further
to detailed descriptions of each attribute, forming a structured Tree of Attributes (ToA). To fully
leverage the ToA, we propose a bottom-up integration pipeline. We introduce vision-conditional
pooling (VCP) layers to aggregate descriptions into attribute-level features, effectively mitigating
potential noise in the generated descriptions. The alignment between attributes and introduced visual
expert tokens is then refined through this hierarchical structure. This integration enables the model to
exploit structured relationships within the ToA, enhancing both the granularity and interpretability of
vision-text alignment.

3 METHODOLOGY

3.1 PRELIMINARY

CLIP. Our approach is built on the pre-trained vision-language model, CLIP (Radford et al., 2021).
Formally, let (x, c) denote the dataset, where x is an image and c ∈ {1, . . . , C} are the class
labels. For an image x, the vision encoder hI (·) transforms it into a feature vector f v
x = hI (x).
Simultaneously, each class label c is mapped to a text prompt tc = a photo of a {c}, and
converted into textual feature vectors f t

c = hT (tc). The predicted class ˆy is given by:
x , f t
c )

cos(f v

ˆy = argmax

c

(1)

where cos(·) denotes cosine similarity.

Image classification with class descriptions. To improve the model’s understanding of the categories
in the transfer datasets, previous works (Menon & Vondrick, 2023; Roth et al., 2023) use more detailed

3

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Figure 2: Overview of the proposed TAP method. TAP uses a bottom-up approach to aggregate the
generated Tree of Attribute. The vision-conditional pooling (VCP) layer aggregates descriptions into
attribute-level features, which are aligned with visual expert tokens focusing on specific attributes
(e.g., color, texture). These attribute-level features are then combined to make class predictions via a
weighted sum of logits from each attribute, fully leveraging the hierarchical structure within the tree.

descriptions from Large Language Models (LLMs) instead of the simple "a photo of a {c}"
to prompt the CLIP text encoder. Under this approach, a convoluted set of descriptions is generated for
a class c as Dc : {"c, which is/has/etc description." }, e.g. c="television"
and description="black or grey". This classification is reformulated as

ˆy = argmax

c

1
|Dc|

(cid:88)

d∈Dc

cos(hI (x), hT (d))

(2)

3.2 OVERALL FRAMEWORK

We rethink the descriptions by LLM Dc as nodes in knowledge graphs. While previous methods
generate an unstructured set of descriptions, we distill structured knowledge graphs for each class c
from LLM, in which the root node is the class name c, capturing the highest level semantics, and the
leaf nodes are the detailed descriptions capturing fine-grained details. In this framework, previous
paradigms only generate the leaf nodes of the graph, with the edges and graph structure missing,
where the rich and inherent structure from the descriptions is overlooked. To address this limitation,
we formulate our approach as a Tree of Attribute, which follows the “concept - attribute - description”
structures, as illustrated in Fig. 1 (c).

Besides weighting the descriptions equally, previous works align descriptions that describe images
from different aspects and at different granularities with a singular CLS token from the image encoder.
However, while the use of a single CLS token is effective in certain contexts, we note that the CLS
token is designed to capture the global information of an input image x (Dosovitskiy et al., 2021).
As a result, even though this helps to further inform global understanding, it may fail to effectively
capture the nuances and variances at the attribute level, which leads to suboptimal use of the rich
descriptions. We address this by introducing a set of learnable prompt tokens that serve as domain
experts in the vision branch, each of which aligns with a specific attribute-level textual embedding.

Additionally, close inspection of the LLM-generated descriptions indicates limited contextual rele-
vance and a high degree of diversity. Previous works (Roth et al., 2023) reflect the issue of descriptions
that are likely not co-occurring e.g. “steam” and “fried”. We further identify cases where the de-
scriptions are technically correct but irrelevant to certain images, such as describing “long tail” in
frontal images of cats, underscoring the need for a selective pooling mechanism. Thus, we introduce
a vision-conditional pooling layer to extract instance-specific text features for each attribute for
selecting the most applicable descriptions.

Overall, TAP leverages the tree structure in two key ways: first, a top-down process generates
attributes and corresponding descriptions for each class in a structured and contextually relevant
manner. This ensures that the descriptions are structured and contextually relevant. Second, a bottom-
up process aggregates information from the leaf nodes (descriptions) into attribute-level features,
which are aligned with visual expert tokens. These expert tokens focus on fine-grained visual

4

Text Encoder ❄······Vision Encoder ❄············++I1•T1I1•T2I1•T3I1•TCI1•T1I1•T2I1•T3I1•TCI1•T1I1•T2I1•T3I1•TC𝑉𝐶𝑃!large, alert ears that are wide at the baseminimal furnishing on the insidetaper smoothly to a pointed 6p.ruddy brown coat with 6ckingblue 6cked pa:ernfawn 6cked pa:ernalmond-shaped and green eyesgold eyesdark eyeliner appearanceFur Pa:ernEar ShapeEye Pa:ern·········𝑉𝐶𝑃"𝑉𝐶𝑃#🔥🔥🔥🔥🔥🔥🔥𝑝!$🔥🔥···𝑣!!𝑝!%𝑝"%𝑝#%𝑝"$𝑝&$𝑣"!𝑣#!𝑣$!···𝑣!"𝑣""𝑣#"𝑣$"···𝑣!%𝑣"%𝑣#%𝑣$%      Text Context                      Token               Vision Expert             Token                 Pooled text feature           for 𝑎&’ A8r.           and 𝑐&’ class𝑝’$𝑝(%𝑣)(🔥🔥𝐼𝑑𝑒𝑛𝑡𝑖𝑡𝑦𝑊!𝑊"𝑣01𝑎∈1,…,𝐴𝑐∈1,…,𝐶𝐶𝑜𝑛𝑐𝑎𝑡.𝐿𝑒𝑎𝑟𝑛𝑎𝑏𝑙𝑒𝐹𝑟𝑜𝑧𝑒𝑛𝑉𝐶𝑃(🔥𝒟"!{}······𝒟!!{}𝒟#!{}𝒟$!{}···𝒟!"{}𝒟"%{}𝒟!%{}𝒟#%{}𝒟$%{}𝒟""{}𝒟#"{}𝒟$"{}···𝒟01{}🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥·········🔥🔥🔥🔥🔥🔥🔥🔥🔥𝑝(%𝑑01,!···𝑑01,3𝑑01,":{{Under review as a conference paper at ICLR 2025

attributes, such as color or shape. Finally, the aggregated attribute-level features contribute to class
predictions using a weighted sum of prediction logits, fully utilizing the hierarchical relationships
within the tree. This dual approach allows TAP to capture both high-level structure and fine-grained
details, leading to enhanced alignment of visual and textual data and improved model performance
and interpretability. Inspired by CoOP (Zhou et al., 2022b), we also incorporate textual contextual
tokens in the text encoder. The overall framework is presented in Fig. 2.

3.3 TREE OF ATTRIBUTE GENERATION BY LLMS

We redefine the process of integrating LLM-generated descriptions by introducing a knowledge graph
Gc = {Vc, Ec} for each class c, where Vc denotes the set of nodes, and Ec denotes the edges that
capture the semantic relationship between nodes. In previous works, Vc is the set of descriptions
Dc, while Ec is missing. We argue that such methods overlook the inherent structure among the
descriptions and thus do not exploit the richness of these descriptions effectively. To better leverage
knowledge from LLMs, we introduce an attribute layer to link the root node class name, and the leaf
node descriptions. The attribute nodes include visual attributes generated by LLMs, such as color and
shape, for systematically guiding description generation as illustrated in Fig. 1 (c). Each branch of
this “tree” represents a specific attribute, with the subsequent “leaves” fleshing out the descriptions
with finer details. In this framework, Gc includes the class name which is the root node, the set of
attributes such as color and shape being the intermediate layer, and lastly the set of descriptions
under each attribute node. Ec includes the edges that build up the hierarchy. This structure allows
for a nuanced representation of class information, spanning from general concepts down to specific
attributes and detailed descriptions.

To this end, we introduce the Tree of Attribute (ToA), where we use a tree structure to model the
relationship and structure of the descriptions. Let Ac denote the set of attributes, and for each attribute
ac ∈ Ac, we denote its leaf nodes as Da
c contains descriptions that specifically pertain
to attribute a for class c, which is denoted as

c . Each set Da

Da

c = {da,1

c

, da,2
c

, . . . , da,n

c },

(3)

where da,i
per attribute.

c

represents the i-th description for attribute a of class c and n is the number of descriptions

The process of generating a Tree of Attribute (ToA) unfolds in three steps: 1) Attribute Generation:
We first query LLMs with the dataset information and ask it to generate a set of attributes A which are
considered relevant and characteristic of the dataset. 2) Example Generation: We then ask LLMs to
generate descriptions for a randomly sampled class in the dataset, using the attributes A identified in
the previous step. Each description takes the format of “class, which {is/has/etc} {description}”. 3)
Description Generation for All Classes: Building upon the Q&A template from the previous step,
the LLM is then tasked with generating descriptions for all classes in the dataset.

Additionally, we incorporate a “global context” attribute which is aligned with the CLS token in the
vision encoder. The descriptions are the 7 standard templates provided in (Radford et al., 2021).

3.4 LEARNING TAP WITH LEARNABLE EXPERT TOKENS

To fully exploit the structured Tree of Attribute, we introduce learnable visual expert tokens pv
a in the
vision branch to learn from each of the attribute nodes a ∈ A. Unlike traditional methods that rely
on a single CLS token for alignment, these expert tokens enable focused learning on specific image
attributes, such as color or shape, enhancing the model’s performance and interpretability.
We denote the set of introduced visual expert tokens as Pv = {pv
a|a ∈ A}. Akin to the idea of
visual prompt tuning (VPT) (Jia et al., 2022), we insert Pv into the input sequence of the vision
encoder, forming the prompted input sequences ˜Xp = {eCLS, Pv, Epatch}, where eCLS is the input
CLS token, and Epatch denotes the embedded patch tokens. To further boost the model’s capacity
for nuanced attribute representation, we employ deep prompting by introducing a zero-initialized
layer residual for each prompt token across transformer layers, which provides more explicit attribute
In parallel, we adopt a set of m learnable context tokens
guidance across transformer layers.
Pt = {pt
j|j ∈ {1, 2, ..., m}} for the text encoder shared across all descriptions, similar to (Zhou
et al., 2022b).

5

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Under review as a conference paper at ICLR 2025

3.5 VISION-CONDITIONAL POOLING

To mitigate issues of misalignment and potential misleading information from the broad spectrum of
LLM-generated descriptions, we proposed an adaptive vision-conditional pooling layer, applicable to
each set of attribute descriptions Da shared across all classes to dynamically pool the most applicable
descriptions based on the visual content of the image x using its corresponding visual expert token
denoted as pv
a,x. For ease of expression, we will proceed without explicitly mentioning x, though it’s
important to note that both the expert token and the resulting attribute-level embeddings are dependent
on the visual information. Intuitively, VCP uses attention to calculate the similarity between pv
a and
all embedded descriptions in attribute Da, which are then used as weights for a weighted sum of the
original description embeddings. Formally, for each attribute a and its associated expert token pv
a,
the pooled attribute-level embedding va

c for class c and attribute a is:

Query = Wq · pv
a,

Key = Wk · Emb(Da
Attention Score = softmax(Query · KeyT ),
c = Attention Score · Emb(Da
va
c ),

c ),

(4)

where Wq and Wk are learnable weights ∈ Rd×d, Emb(·) denotes the embedding function, and
softmax(·) is the Softmax function. This layer mirrors cross-attention but omits Wv to maintain
the output within the CLIP V-L space.

3.6 TRAINING AND INFERENCE

Training objective. During training, each visual expert token pv
attribute-level embedding va

c , trained with the following contrastive objective:

a is aligned with its associated

Lcon(pv

a, va

c ) = −

1
N

N
(cid:88)

i=1

log

a, va
exp(cos(pv
c=1 exp(cos(pv

y )/τ )
a, va

c )/τ )

(cid:80)C

,

(5)

where N represents the number of training samples, and τ is the learned temprature of CLIP. The
total classification loss Lclass is the average of the contrastive loss from each expert token as well as
the CLS token, defined as:

Lclass =

1
|A|

(cid:18) (cid:88)

a∈A

Lcon(pv

a, va

c ))

(cid:19)

,

(6)

Similar to (Khattak et al., 2023b) and (Bulat & Tzimiropoulos, 2023), we regularize the vision CLS
token, text feature, and the prediction logits from each attribute using the vanilla CLIP model. We
denote the regularization loss as Lreg, where the details can be found in Appendix. The overall
training objective is Ltotal = Lclass + Lreg.

Prediction fusion. During inference, we integrate the prediction by each attribute expert pair by a
weighted sum, formulated as follows:

(cid:18)

˜y = argmax

c

α cos(f v

CLS, vCLS

c

) +

1 − α
|A| − 1

(cid:88)

(cid:19)

cos(pv

a, va
c )

a∈A\{CLS}

(7)

where α is a hyperparameter that signifies the weight assigned to the global context provided by the
CLS token, balancing its contribution with that of the attribute-specific expert prompts.

4 EXPERIMENTS

We extensively evaluate our method in three settings: 1) Base-to-novel class generalization, where the
datasets are equally split into base and novel classes. We train the model on the base classes only and
evaluate on both base and novel classes; 2) Cross-dataset transfer, where we train on ImageNet with
16 shots per class, and directly evaluate on other datasets in zero-shot; and 3) Few-shot classification
with 16 shots per class.

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Table 1: Comparison with state-of-the-art methods in base-to-novel generalization. The model is
trained on the base class, and evaluated on the unseen novel classes in zero-shot. TAP demonstrates
strong generalization performance. HM: harmonic mean (Xian et al., 2017).

(a) Average

(b) ImageNet

(c) Caltech101

(d) OxfordPets

Base Novel HM

Base Novel HM

Base Novel HM

Base Novel HM

69.34 74.22 71.70
CLIP
82.69 63.22 71.66
CoOp
80.47 71.69 75.83
Co-CoOp
82.48 70.75 76.16
ProGrad
81.13 75.00 77.78
RPO
LoGoPrompt 84.47 74.24 79.03
PromptSRC 84.26 76.10 79.97
84.75 77.63 81.04
TAP

72.43 68.14 70.22
CLIP
76.47 67.88 71.92
CoOp
75.98 70.43 73.10
Co-CoOp
77.02 66.66 71.46
ProGrad
76.60 71.57 74.00
RPO
LoGoPrompt 76.74 70.83 73.66
PromptSRC 77.60 70.73 74.01
77.97 70.40 73.99
TAP

96.84 94.00 95.40
CLIP
98.00 89.81 93.73
CoOp
97.96 93.81 95.84
Co-CoOp
98.02 93.89 95.91
ProGrad
97.97 94.37 96.03
RPO
LoGoPrompt 98.19 93.78 95.93
PromptSRC 98.10 94.03 96.02
98.90 95.50 97.17
TAP

91.17 97.26 94.12
CLIP
93.67 95.29 94.47
CoOp
95.20 97.69 96.43
Co-CoOp
95.07 97.63 96.33
ProGrad
94.63 97.50 96.05
RPO
LoGoPrompt 96.07 96.31 96.18
PromptSRC 95.33 97.30 96.30
95.80 97.73 96.76
TAP

(e) StanfordCars

(f) Flowers102

(g) Food101

(h) FGVCAircraft

Base Novel HM

Base Novel HM

Base Novel HM

Base Novel HM

63.37 74.89 68.65
CLIP
78.12 60.40 68.13
CoOp
70.49 73.59 72.01
Co-CoOp
77.68 68.63 72.88
ProGrad
73.87 75.53 74.69
RPO
LoGoPrompt 78.36 72.39 75.26
PromptSRC 78.27 74.97 76.58
80.70 74.27 77.35
TAP

72.08 77.80 74.83
CLIP
97.60 59.67 74.06
CoOp
94.87 71.75 81.71
Co-CoOp
95.54 71.87 82.03
ProGrad
RPO
94.13 76.67 84.50
LoGoPrompt 99.05 76.52 86.34
PromptSRC 98.07 76.50 85.95
97.90 75.57 85.30
TAP

90.10 91.22 90.66
CLIP
88.33 82.26 85.19
CoOp
90.70 91.29 90.99
Co-CoOp
90.37 89.59 89.98
ProGrad
RPO
90.33 90.83 90.58
LoGoPrompt 90.82 91.41 91.11
PromptSRC 90.67 91.53 91.10
90.97 91.83 91.40
TAP

27.19 36.29 31.09
CLIP
40.44 22.30 28.75
CoOp
33.41 23.71 27.74
Co-CoOp
40.54 27.57 32.82
ProGrad
RPO
37.33 34.20 35.70
LoGoPrompt 45.98 34.67 39.53
PromptSRC 42.73 37.87 40.15
44.40 36.50 40.06
TAP

(i) SUN397

(j) DTD

(k) EuroSAT

(l) UCF101

Base Novel HM

Base Novel HM

Base Novel HM

Base Novel HM

69.36 75.35 72.23
CLIP
80.60 65.89 72.51
CoOp
79.74 76.86 78.27
Co-CoOp
81.26 74.17 77.55
ProGrad
RPO
80.60 77.80 79.18
LoGoPrompt 81.20 78.12 79.63
PromptSRC 82.67 78.47 80.52
82.87 79.53 81.17
TAP

53.24 59.90 56.37
CLIP
79.44 41.18 54.24
CoOp
77.01 56.00 64.85
Co-CoOp
77.35 52.35 62.45
ProGrad
RPO
76.70 62.13 68.61
LoGoPrompt 82.87 60.14 69.70
PromptSRC 83.37 62.97 71.75
84.20 68.00 75.24
TAP

56.48 64.05 60.03
CLIP
92.19 54.74 68.69
CoOp
87.49 60.04 71.21
Co-CoOp
90.11 60.89 72.67
ProGrad
RPO
86.63 68.97 76.79
LoGoPrompt 93.67 69.44 79.75
PromptSRC 92.90 73.90 82.32
90.70 82.17 86.22
TAP

70.53 77.50 73.85
CLIP
84.69 56.05 67.46
CoOp
82.33 73.45 77.64
Co-CoOp
84.33 74.94 79.35
ProGrad
RPO
83.67 75.43 79.34
LoGoPrompt 86.19 73.07 79.09
PromptSRC 87.10 78.80 82.74
87.90 82.43 85.08
TAP

Datasets and baslines. For all of the three settings, we follow previous works (Zhou et al., 2022b;a),
using 11 image recognition datasets, including: ImageNet (Deng et al., 2009) and Caltech101 (Fei-Fei
et al., 2004) for generic object recognition; OxfordPets (Parkhi et al., 2012), StanfordCars (Krause
et al., 2013), Flowers102 (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014), and
FGVCAircraft (Maji et al., 2013) for fine-grained classification; SUN397 (Xiao et al., 2010) for scene
recognition; UCF101 (Soomro et al., 2012) for action recognition; DTD (Cimpoi et al., 2014) for
texture classification; and EuroSAT (Helber et al., 2019) for satellite image analysis. We benchmark
against several leading methods, including CLIP (Radford et al., 2021), CoOp (Zhou et al., 2022b),
Co-CoOP (Zhou et al., 2022a), ProGrad (Zhu et al., 2023), RPO (Lee et al., 2023), LoGoPrompt (Shi
& Yang, 2023), and the state-of-the-art PromptSRC (Khattak et al., 2023b).

Implementation details. A pre-trained CLIP model with a ViT-B/16 vision backbone is used in
all of our experiments and results are averaged over 3 runs. We use GPT-3.5-turbo (Ouyang et al.,
2022) for attribute and description generation. We initialize the text context tokens with the word
embedding of "a photo of a." During training, we iteratively train the vision and text encoders
with 5 epochs for vision and 1 epoch for text schedule. We train a total of 60, 24, and 120 epochs for
base-to-novel generalization, cross-dataset transfer, and few-shot classification respectively. We set
α = 0.4 for all datasets. We also use a Gaussian Prompt Weighting (GPA) following (Khattak et al.,
2023b), with a mean of 0.9N , std of 0.1N , where N represents the total number of epochs, for all
tasks. Refer to the Appendix for additional implementation details.

4.1 BASE-TO-NOVEL GENERALIZATION

In base-to-novel generalization, we equally split the classes into base and novel classes. Initial
training and evaluations are conducted on the seen base classes, followed by evaluation on the unseen
novel classes in a zero-shot manner. TAP surpasses prior state-of-the-art models in terms of the
base and novel class accuracy, as well as their harmonic mean across most of the 11 datasets, with
an average increase of 1.53% in the zero-shot novel class prediction, and a 1.07% increase in the
overall harmonic mean in average, as detailed inTable 1. Notably, our method improves unseen class
prediction without compromising base class performance, exhibiting an average performance boost

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Table 2: Comparison with state-of-the-art methods in cross-dataset transfer evaluation. The model is
trained on the source dataset and evaluated on the target datasets in zero-shot.

Source

ImageNet

71.51
71.02
71.27

72.30

CoOp
CoCoOp
PSRC

TAP

Target

Caltech101

93.70
94.43
93.60

Pets
89.14
90.14
90.25

Cars
64.51
65.32
65.70

Flowers

68.71
71.88
70.25

Food101

85.30
86.06
86.15

Aircraft

18.47
22.94
23.90

N397
SU

64.15
67.36
67.10

D
T
D

41.92
45.73
46.87

EuroSAT

46.39
45.37
45.50

CF101

U

66.55
68.21
68.75

Average

63.88
65.74
65.81

94.30

90.70

65.60

70.93

86.10

24.57

68.30

50.20

46.00

68.90

66.56

Table 3: Comparison with state-of-the-art methods in few shot classification results with 16 shots.

16-Shot Classification

Average

78.79
79.89
74.90
82.87

ImageNet

67.31
71.87
70.83
73.17

CLIP
CoOp
CoCoOp
PSRC

TAP

83.37

73.76

Caltech101

95.43
95.57
95.16
96.07

96.73

Pets
85.34
91.87
93.34
93.67

Cars
80.44
83.07
71.57
83.83

Flowers

97.37
97.07
87.84
97.60

Food101

82.90
84.20
87.25
87.50

Aircraft

45.36
43.40
31.21
50.83

N397
SU

73.28
74.67
72.15
77.23

D
T
D

69.96
69.87
63.04
72.73

EuroSAT

87.21
84.93
73.32
92.43

CF101

U

82.11
82.23
78.14
86.47

93.90

85.37

98.10

87.53

50.43

77.30

74.90

91.90

87.17

of 0.49%. In the challenging fine-grained tasks such as DTD, EuroSAT, and UCF101, TAP achieves
significant improvements in novel class prediction by 5.03%, 8.27%, and 3.63% respectively. These
results underscore the robust generalizability and efficacy of our method across diverse scenarios.

4.2 CROSS-DATASET TRANSFER

To further investigate the generalization capability of TAP , we train on ImageNet with 16 shots per
class, and directly test on the other 10 datasets under zero-shot without further tuning. As shown in
Table 2, TAP demonstrates better generalizability on 8/10 target datasets compared to PromptSRC
(Khattak et al., 2023b), and achieves an average performance increase of 0.75%. Additionally, while
the performance increase of previous methods on target datasets come with costs on the source
dataset (−0.49% for CoCoOP and −0.24% for PromptSRC) as compared to CoOP (Zhou et al.,
2022b), TAP also outperform previous methods on the source dataset with 1.03% increase compared
to PromptSRC (0.79% incrase compared to CoOP), demonstrating TAP ’s robustness in domain
generalization without sacrifice on source dataset performance.

4.3 FEW-SHOT CLASSIFICATION

In few-shot classification, TAP also outperforms existing methods in 9 out of the 11 datasets. Detailed
in Table 3, we achieve an average accuracy of 83.37 across the 11 datasets, surpassing the previous
state-of-the-art methods by 0.5%, further demonstrating the effectiveness of our method.

4.4 ABLATION STUDY

Effects of Tree of Attribute. A core inquiry is whether structuring descriptions into a Tree of
Attribute (ToA) offers advantages over an unstructured aggregation of LLM-generated descriptions.
To evaluate, we revert to aligning a mixed, unstructured set of descriptions with the CLS token - a
common practice in prior studies (Mao et al., 2023; Maniparambil et al., 2023; Liu et al., 2024b;
Wang et al., 2024; Tian et al., 2023; Zheng et al., 2023), while keeping the same number of visual
prompt tokens. According to Table 4, substituting the ToA with an unstructured set results in
significant performance decreases of 1.86%, 2.31%, and 2.11% across the average base, novel, and
their harmonic mean performances, respectively. This stark contrast underscores the ToA’s critical
role in enhancing model efficacy.

8

Under review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Table 4: Effects of the Tree of At-
tributes.

Des. Org. Unstructured

Ours

Base
Novel
HM

82.89
75.32
78.93

84.75
77.63
81.04

Table 5: Effects of domain experts.

Align. Token

CLS

Ours

Base
Novel
HM

83.89
76.85
80.22

84.75
77.63
81.04

Figure 3: Visualization of the class activation maps.

Table 6: Effects of α

Table 7: Effects of the number of experts.

α

1.0

0.4

# Attrs.

1

2

3

4

5

6

7

8

Ours

Base
Novel
HM

81.54
73.85
77.51

84.75
77.63
81.04

83.20 83.97 84.10 84.41 84.45 84.62 84.66 84.74 84.75
Base
74.90 76.20 76.35 77.06 77.13 77.17 77.35 76.67 77.63
Novel
HM 78.83 79.90 80.04 80.57 80.63 80.72 80.84 80.50 81.04

Effects of Learning through Domain Experts. Further, we examine the impact of substituting the
CLS token with visual expert tokens for learning fine-grained attributes, commonly adopted in in
previous works (Mao et al., 2023; Lee et al., 2023; Tian et al., 2023; Zheng et al., 2023). Our findings
(Table 5) reveal improvements of 0.89%, 0.78%, and 0.82% in the average base, novel, and harmonic
mean accuracies, respectively, upon integrating visual expert tokens. These results support the notion
that domain-specific, learnable tokens enhance the model’s ability to grasp fine-grained details by
focusing on distinct aspects of the image, as opposed to the CLS token’s global focus.

Effects of fusion coefficient α. α in Eq. (7) balance global and local information. We compare
the performance of using CLS token only (i.e. α = 1.0) for making the final prediction against
our proposed prediction fusion with α = 0.4. As shown in Table 6, using CLS token decreases
the performance significantly on both base and novel classes. This result further demonstrates the
limitations of using a singular CLS token which focuses on global information, and supports the
effectiveness of the use of expert tokens which focus on local information.

Effects of Number of Attributes. In our framework, the selection of attributes is dynamically
determined by LLMs, leading to variability across different datasets. This adaptability stands in
contrast to a static approach where the number of attributes is uniformly set across all datasets. To
understand the impact of this variability, we explore how altering the number of attributes from 1 to 8
influences model performance. Our findings, detailed in Table 7, reveal a performance improvement
trend as the number of attributes increases, with an optimal peak at 7 attributes before a slight decline
at 8. However, crucially, across all fixed-attribute scenarios, none matched the performance achieved
through our method’s dynamic attribute determination. These results underscore the importance of
an adaptive approach to attribute selection, as opposed to a one-size-fits-all strategy.

Design choice of the vision-conditional pooling layer. Lastly, we ablate the design of the pooling
layer, starting from the naive training-free average pooling, to the attention-based pooling mechanism
with condition on the input image. Compared to average pooling, VCP demonstrates a performance
gain of 1.08% in the average harmonic mean. Furthermore, when compared with attention-based max
pooling, which selects a single description per attribute according to the attention score in Eq. (4),
VCP maintains a superior advantage of 1.55% in average harmonic mean. These outcomes attest to
the VCP layer’s integral role in finetuning attribute relevance to the visual context, substantiating its
design and implementation within our model.

9

Fur Pa’ernEar Pa’ernEye Pa’ernWheel DesignGrille StyleHeadlight ShapeColorPetalStem Characteris=csImageImageImageUnder review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Figure 4: Visualization of the attention weights in the VCP layer for an example “dumplings” image.

Table 8: Ablation on design choice of the VCP layer. Our cross-attention based pooling mechanism
demonstrates the best performance among other variants.

Pooling Method

Base Acc. Novel Acc.

HM

Attn. Max Pooling
Average Pooling

VCP (Ours)

82.90
83.18

84.75

76.36
76.98

77.63

79.49
79.96

81.04

4.5 VISUALIZATION

Expert tokens focus on attribute-related regions. We further investigate the effects of vision
domain experts by visualizing their class activation maps from three illustrative examples using
GradCAM (Selvaraju et al., 2017), as shown inFig. 3. These visualizations underscore the precision
with which each expert token concentrates on the image regions pertinent to its designated attribute.
Take the first cat image as an example. The “fur pattern” expert distinctly highlights the animal’s fur
texture, whereas the “ear” and “eye” experts focus precisely on the respective anatomical features.
This pattern of attribute-specific attention is consistent across the evaluated examples, reinforcing the
conceptualization of expert tokens as dedicated “domain experts” within the visual field.

VCP layer pools the most applicable descriptions. The inherently interpretable nature of the VCP
layer, thanks to its attention mechanism, allows for insightful visualizations of its operational process.
Through the examination of attention weights assigned by the VCP layer to different attributes
in a given image, we elucidate the layer’s capability to discern and prioritize the most applicable
descriptions. As illustrated in Fig. 4 with a “dumplings” image, the VCP layer adeptly allocates
higher attention weights to descriptions accurately reflecting the observed instance (e.g., assigning
weights of 0.92 to “round with a pleated edge” under the “Shape” attribute and 0.95 to “soft and
chewy texture” under the Texture”). In contrast, less relevant descriptions for the specific image
context (e.g., “crescent-shaped” for Shape and “crispy texture from pan-frying” for Texture) receive
significantly lower weights. This discernment is crucial, given the class dumplings” encompasses a
broad variety of appearances based on cooking methods, yet not all descriptions are fitting for every
instance. These visualizations compellingly demonstrate the VCP layer’s effectiveness in refining
description relevance, thereby enhancing the model’s interpretative alignment with the visual content.

5 CONCLUSION

This paper introduces Tree of Attribute Prompt learning (TAP), a novel method that integrates detailed,
LLM-generated descriptions within VLMs, achieving state-of-the-art performance in base-to-novel
generalization, cross-dataset transfer, and few-shot image classification tasks across 11 diverse
datasets. TAP leverages a hierarchical "Tree of Attribute" framework, distilling structured knowledge
graphs from LLMs for nuanced representation of visual concepts, and employs learnable "domain
expert" tokens and a vision-conditional pooling module for optimal image-text alignment. While
promising, we note that the reliance on LLMs presents challenges in fine-grained datasets where
similar classes require nuanced differentiation, in which cases LLMs generate identical descriptions
for distinct classes, impacting novel class prediction performance. It highlights the current limitations
of LLMs in discerning highly fine-grained distinctions. Addressing this challenge through enhanced
LLM capabilities or alternative strategies will be a key focus of future research.

10

• served steamed in a bamboo basket• pan-fried to a crispy finish and served with a dipping saucePresentation• pale beige color • golden-brown hue from pan-frying or deep-fryingColor• soft and chewy texture• crispy texture on the bottom from pan-fryingTexture0.810.190.950.050.880.120.920.08Shape• round with a pleated edge• crescent-shaped, with a fold in the doughUnder review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modify-
ing pixel space to adapt pre-trained models. arXiv preprint arXiv:2203.17274, 3:11–12, 2022.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative compo-
nents with random forests. In Computer Vision–ECCV 2014: 13th European Conference, Zurich,
Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 446–461. Springer, 2014.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.

Adrian Bulat and Georgios Tzimiropoulos. Lasp: Text-to-text optimization for language-aware
soft prompting of vision & language models. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pp. 23232–23241, 2023.

Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Prompt

learning with optimal transport for vision-language models. In ICLR, 2023.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ-
ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern
recognition, pp. 3606–3613, 2014.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale
hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition,
pp. 248–255. Ieee, 2009.

Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor G Turrisi da Costa, Cees GM
Snoek, Georgios Tzimiropoulos, and Brais Martinez. Bayesian prompt learning for image-language
model generalization. In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pp. 15237–15246, 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas
Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit,
and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale.
In International Conference on Learning Representations, 2021. URL https://openreview.
net/forum?id=YicbFdNTTy.

Zalan Fabian, Zhongqi Miao, Chunyuan Li, Yuanhan Zhang, Ziwei Liu, Andrés Hernández, Andrés
Montes-Rojas, Rafael Escucha, Laura Siabatto, Andrés Link, et al. Multimodal foundation models
for zero-shot animal species recognition in camera trap images. arXiv preprint arXiv:2311.01064,
2023.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training
examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference
on computer vision and pattern recognition workshop, pp. 178–178. IEEE, 2004.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset
and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected
Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019.

Yu-Guan Hsieh, Cheng-Yu Hsieh, Shih-Ying Yeh, Louis Béthune, Hadi Pouransari, Pavan Ku-
mar Anasosalu Vasu, Chun-Liang Li, Ranjay Krishna, Oncel Tuzel, and Marco Cuturi. Graph-
based captioning: Enhancing visual descriptions by interconnecting region captions. arXiv preprint
arXiv:2407.06723, 2024.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung,
Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with
noisy text supervision. In International conference on machine learning, pp. 4904–4916. PMLR,
2021.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and
Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709–727.
Springer, 2022.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz
Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pp. 19113–19122, 2023a.

Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan
Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without
forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),
pp. 15190–15200, October 2023b.

Gahyeon Kim, Sohee Kim, and Seokju Lee. Aapl: Adding attributes to prompt learning for vision-
language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 1572–1582, 2024.

Jae Myung Kim, A Koepke, Cordelia Schmid, and Zeynep Akata. Exposing and mitigating spurious
correlations for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pp. 2584–2594, 2023.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained
categorization. In Proceedings of the IEEE international conference on computer vision workshops,
pp. 554–561, 2013.

Dongjun Lee, Seokwon Song, Jihee Suh, Joonmyeong Choi, Sanghyeok Lee, and Hyunwoo J Kim.
Read-only prompt optimization for vision-language few-shot learning. In Proceedings of the
IEEE/CVF International Conference on Computer Vision, pp. 1401–1411, 2023.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt

tuning, 2021.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.

Mingxuan Liu, Subhankar Roy, Wenjing Li, Zhun Zhong, Nicu Sebe, and Elisa Ricci. Democratizing
fine-grained visual recognition with large language models. In The Twelfth International Confer-
ence on Learning Representations, 2024a. URL https://openreview.net/forum?id=
c7DND1iIgb.

Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt
tuning can be comparable to fine-tuning universally across scales and tasks. CoRR, abs/2110.07602,
2021. URL https://arxiv.org/abs/2110.07602.

Xin Liu, Jiamin Wu, Wenfei Yang, Xu Zhou, and Tianzhu Zhang. Multi-modal attribute prompting
for vision-language models. IEEE Transactions on Circuits and Systems for Video Technology,
2024b.

Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution
learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pp. 5206–5215, 2022.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained

visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.

Mayug Maniparambil, Chris Vorster, Derek Molloy, Noel Murphy, Kevin McGuinness, and Noel E
O’Connor. Enhancing clip with gpt-4: Harnessing visual descriptions as prompts. In Proceedings
of the IEEE/CVF International Conference on Computer Vision, pp. 262–271, 2023.

Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang, Xin Wang, and Carl
Vondrick. Doubly right object recognition: A why prompt for visual rationales. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2722–2732, 2023.

Sachit Menon and Carl Vondrick. Visual classification via description from large language models.

ICLR, 2023.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number
of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp.
722–729. IEEE, 2008.

Zachary Novack, Julian McAuley, Zachary Lipton, and Saurabh Garg. Chils: Zero-shot image
classification with hierarchical label sets. In International Conference on Machine Learning
(ICML), 2023.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow
instructions with human feedback. Advances in Neural Information Processing Systems, 35:
27730–27744, 2022.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012

IEEE conference on computer vision and pattern recognition, pp. 3498–3505. IEEE, 2012.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga,

and A. Lerer. Automatic differentiation in PyTorch. In NeurIPS Autodiff Workshop, 2017.

Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? gener-
ating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pp. 15691–15701, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,
Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual
models from natural language supervision. In International conference on machine learning, pp.
8748–8763. PMLR, 2021.

Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz
Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 6545–6554, 2023.

Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep
Akata. Waffling around for performance: Visual classification with random words and broad
concepts. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),
pp. 15746–15757, October 2023.

Shuvendu Roy and Ali Etemad. Consistency-guided prompt learning for vision-language models.
In The Twelfth International Conference on Learning Representations, 2024. URL https:
//openreview.net/forum?id=wsRXwlwx4w.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh,
and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local-
ization. In Proceedings of the IEEE international conference on computer vision, pp. 618–626,
2017.

Cheng Shi and Sibei Yang. Logoprompt: Synthetic text images can be good visual prompts for
vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pp. 2932–2941, 2023.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions

classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.

Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. Argue: Attribute-guided prompt tuning for

vision-language models. arXiv preprint arXiv:2311.16494, 2023.

Dongsheng Wang, Miaoge Li, Xinyang Liu, MingSheng Xu, Bo Chen, and Hanwang Zhang. Tuning
multi-mode token-level prompt alignment across modalities. In Thirty-seventh Conference on
Neural Information Processing Systems, 2023a. URL https://openreview.net/forum?
id=A253n2EXCd.

Wenhao Wang, Yifan Sun, Wei Li, and Yi Yang. TransHP: Image classification with hierarchical
prompting. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL
https://openreview.net/forum?id=vpQuCsZXz2.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Yubin Wang, Xinyang Jiang, De Cheng, Dongsheng Li, and Cairong Zhao. Learning hierarchical
prompt with structured linguistic knowledge for vision-language models. In Proceedings of the
AAAI Conference on Artificial Intelligence, volume 38, pp. 5749–5757, 2024.

Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren,
Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for
rehearsal-free continual learning. In European Conference on Computer Vision, pp. 631–648.
Springer, 2022a.

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent
Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings
of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139–149, 2022b.

Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4582–4591,
2017.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database:
Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on
computer vision and pattern recognition, pp. 3485–3492. IEEE, 2010.

Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, Peng Wang, and Yanning
Zhang. Dual modality prompt tuning for vision-language pre-trained model. IEEE Transactions
on Multimedia, pp. 1–13, 2023. doi: 10.1109/TMM.2023.3291588.

An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo
Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3090–3100,
2023.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li,
Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint
arXiv:2407.10671, 2024.

Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark
Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image
classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 19187–19197, 2023.

Yi Zhang, Ce Zhang, Ke Yu, Yushun Tang, and Zhihai He. Concept-guided prompt learning for
generalization in vision-language models. Proceedings of the AAAI Conference on Artificial
Intelligence, 38(7):7377–7386, Mar. 2024. doi: 10.1609/aaai.v38i7.28568. URL https://ojs.
aaai.org/index.php/AAAI/article/view/28568.

Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search.

arXiv preprint

arXiv:2206.04673, 2022.

Zhaoheng Zheng, Jingmin Wei, Xuefeng Hu, Haidong Zhu, and Ram Nevatia. Large language models
are good prompt learners for low-shot image classification. arXiv preprint arXiv:2312.04076,
2023.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for
vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pp. 16816–16825, 2022a.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-

language models. International Journal of Computer Vision, 130(9):2337–2348, 2022b.

Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for
prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
15659–15669, 2023.

14

Under review as a conference paper at ICLR 2025

A APPENDIX

A.1 MODEL REGULARIZATION

Denote the frozen image feature from CLIP vision encoder as f v, the frozen text feature for description
d from CLIP text encoder as f t
d, and the zero-shot logit prediction from CLIP as ˆy. Additionally,
denote the trained image feature as ˜f v, the trained text feature for description d as ˜f t
d, and the logit
prediction from attribute a after training as ˜ya. The losses are as follows:

LL1−V = ||f v − ˜f v||1

Lcon−T = −

(cid:18) 1
2

(cid:88)

d∈D

log

(cid:80)

k∈Ds

exp(cos(f t

d, ˜f t
d))
d, ˜f t
exp(cos(f t
(cid:18) (cid:88)

k))

LKL−attr =

1
|A|

a∈A

+

1
2

log

exp(cos(f t

d, ˜f t
d))
k, ˜f t
exp(cos(f t

d))

k∈Ds

(cid:80)

(cid:19)

DKL(ˆy, ˜ya)

The regularization loss is then:

Lreg = µ1LL1−V + µ2LKL−attr + µ3Lcon−T ,

Our overall training objective is thus given by:

Ltotal = Lclass + Lreg

(cid:19)

(8)

(9)

(10)

(11)

(12)

To investigate the effectiveness of model regularization, we compare TAP against existing methods
with and without regularization. As evidenced in Table 9, the proposed model regularization helps in
both base and novel performance, with an increase of 1.62% in average harmonic mean. Comparing
to existing methods, TAP is consistently better than other baselines in both settings, demonstrating
the robustness of our method.

Table 9: Effectiveness of model regularization. TAP achieves favorable results under both settings.

Regularization

Base

Novel

HM

PSRC-reg
MaPLe
TAP-reg

PSRC
TAP

×
×
×
✓
✓

84.21
82.28
83.37

84.26
84.75

71.79
75.14
75.82

76.10
77.63

77.51
78.55
79.42

79.97
81.04

A.2 ADDITIONAL IMPLEMENTATION DETAILS

Because the number of attributes vary across the 11 datasets which results in different number of
learnable parameters, we group the datasets into two and apply two sets of learning rates to balance
generalizability and performance. For DTD, Oxford Flowers, Stanford Cars, UCF101, and Caltech101
datasets, which have fewer attributes, we use a low learning rate of 0.002 for the text encoder to avoid
overfitting and a high learning rate of 0.006 for the vision encoder to facilitate the learning process. A
high µ3 = 3 is also used to regularize the text encoder for preventing overfitting. For the remaining 6
datasets, which have more attributes, the learning rates for both text and vision encoders are set as
0.004, with µ3 = 1.5. µ1 = 10, and µ2 = 2.5 are used for all datasets.

For base-to-novel generalization and few-shot classification evaluations, we use an adaptive approach
for generating the attributes, in which the attributes vary across datasets. Although it turns out to be
better than using a fixed set of attributes as shown in Table 7, it is not applicable to the cross-dataset
transfer experiment as both VCP layers and visual expert tokens are specific to their corresponding
attributes. Therefore, for cross-dataset transfer, we use the following fixed set of 4 attributes that are
applicable to all 11 datasets: Pattern, Texture, Shape, and Context.

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

We use PyTorch Paszke et al. (2017) to implement all experiments on a single NVIDIA A100-80GB
GPU. Our code is developed based on the implementation of CoOp Zhou et al. (2022b), which is
available at https://github.com/KaiyangZhou/CoOp and released under the MIT license. Our code is
also released under the MIT license. Baseline results for the three tasks are taken from their respective
publications. For the “global context” attribute which is aligned with the CLS token in the vision
encoder, we use the following 7 selected templates provided in Radford et al. (2021).

"itap of a {class}."

"a bad photo of the {class}."

"a origami {class}."

"a photo of the large {class}."

"a {class} in a video game."

"art of the {class}."

"a photo of the small {class}."

A.3 ROBUSTNESS OF LLMS

To investigate the robustness of our methods against different LLMs, we additionally generate the
descriptions using a locally-served small LLM - Qwen-2-7B-Instruct (Yang et al., 2024), in which
the results are comparable.

Table 10: Robustness against different LLMs.

LLMs

Base Acc. Novel Acc.

HM

Qwen-2-7B-Instruct
GPT-3.5-Turbo

84.68
84.75

77.31
77.63

80.83
81.04

A.4 PROMPTS FOR TREE-OF-ATTRIBUTE GENERATION

As introduced in Section 3.3, we generate the Tree-of-Attribute with the following three steps: 1)
Attribute Generation, 2) In-Context Example Generation, and 3) Description Generation for All
Classes. The prompts for each step are as follows:

1) Attribute Generation:

{Dataset Description.}

Visual attributes refer to observable, describable features of the images that can include color, shape,
size, texture, and any specific patterns or markings, which can help differentiate between classes for
the dataset. They should be consistently observable across multiple images of the same class. Your
task is to generate a list of visual attributes (less than 10) for the {Dataset Name} dataset. Ensure
this list is clear, concise, and specific to the dataset’s needs. Avoid generic attributes that do not
contribute to distinguishing between classes.

2) In-Context Example Generation

Describe describe what a "{Random Class Name}" class in the {Dataset Name} dataset look like
using the generated visual attributes.

You must follow the following rules:

1. For each visual attribute, describe all possible variations as separate sentences. This approach
allows for a detailed and clear presentation of each attribute’s range.

2. Provide a maximum of five descriptions for each visual attribute to maintain focus and relevance.
Also, aim to provide at least two descriptions to ensure a comprehensive overview of the attribute.

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

3. The descriptions should provide clear, distinguishable features of each class to support image
classification tasks.

4. Descriptions for each attribute are independent from each other, and they should not serve as
context for each other.

5. Each description describes an image independetly. If certain description is possible for a class,
please just list that description, and do not use words like "may have" or "sometimes have".

6. Reply descriptions only. Do not include any explanation before and after the description.

7. The descriptions should follow the format of "classname, which ...", where "..." is the description
of the visual attribute.

3) Description Generation for All Classes

{Dataset Description.}

Your task is to write detailed descriptions for various classes within the {Dataset Name} dataset, using
the provided visual attributes such as color and shape. These descriptions will help in accurately
classifying and understanding the unique features of each class.

You must follow the following rules:

1. For each visual attribute, describe all possible variations as separate sentences. This approach
allows for a detailed and clear presentation of each attribute’s range.

2. Provide a maximum of five descriptions for each visual attribute to maintain focus and relevance.
Also, aim to provide at least two descriptions to ensure a comprehensive overview of the attribute.

3. The descriptions should provide clear, distinguishable features of each class to support image
classification tasks.

4. Descriptions for each attribute are independent from each other, and they should not serve as
context for each other.

5. Each description describes an image independetly. If certain description is possible for a class,
please just list that description, and do not use words like "may have" or "sometimes have".

6. Reply descriptions only. Do not include any explanation before and after the description.

7. The descriptions should follow the format of "classname, which ...", where "..." is the description
of the visual attribute.

Q: Describe what a "{Random Class Name}" in the {Dataset Name} look like using the following
visual attributes: {Visual Attributes from Step 1.}

A: {Answer from Step 2.}

Q: Describe what a "{Target Class Name}" in the {Dataset Name} look like using the following visual
attributes: {Visual Attributes from Step 1.}

A:

In the prompt templates, "Dataset Description" is the description of the dataset from their official
website, "Random Class Name" is a randomly sampled class name in the dataset for in-context
example generation, and "Target Class Name" is the class name of interest for the current query.
While step 1 and 2 are made in two consecutive calls to provide contexts which are queried once
per dataset, step 3 is queried independently for each of the remaining classes in the dataset. Our
carefully designed prompts for step 1 and 2 guide the LLM in generating high-quality examples.
Human review further confirms that the generated in-context examples from these prompts are of
high quality even without any manual intervention.

A.5 ATTRIBUTE SETS

The attribute sets generated by LLMs are shown in Table 11.

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

17

Under review as a conference paper at ICLR 2025

Table 11: Attribute sets generated by LLMs for the 11 datasets.

Dataset

ImageNet

Caltech101

StanfordCars

Flowers102

Food101

FGVCAircraft

SUN397

DTD

EuroSAT

UCF101

Attributes

Orientation
Shape
Pattern
Texture
Pose
Context

Dominant Feature
Shape
Texture
Color
Size

Body Type
Wheel Design
Grille Style
Headlight Shape
Rear Taillight Design
Roof Style

Color
Petal
Center structure
Stem characteristics

Color
Shape
Texture
Ingredients
Presentation Style

Wing Configuration
Winglet Presence
Engine Configuration
Number of Engines
Fuselage Length
Fuselage shape
Wingspan

Indoor/Outdoor
Color
Dominant elements
Environment
Architectural style
Patterns

Texture
Pattern
Repetition
Contrast

Contrast
Texture
Orientation
Edge
Size
Color
Symmetry

Action Pose
Number of People
Background Setting
Objects Present
18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

