Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning

 Cristina Menghini

Brown University

cristina_menghini@brown.edu &Andrew Delworth

Brown University

adelwort@cs.brown.edu &Stephen H. Bach

Brown University

sbach@cs.brown.edu

###### Abstract

Fine-tuning vision-language models (VLMs) like CLIP to downstream tasks is often necessary to optimize their performance. However, a major obstacle is the limited availability of labeled data. We study the use of pseudolabels, i.e., heuristic labels for unlabeled data, to enhance CLIP via prompt tuning. Conventional pseudolabeling trains a model on labeled data and then generates labels for unlabeled data. VLMs' zero-shot capabilities enable a "second generation" of pseudolabeling approaches that do not require task-specific training on labeled data. By using zero-shot pseudolabels as a source of supervision, we observe that learning paradigms such as semi-supervised, transductive zero-shot, and unsupervised learning can all be seen as optimizing the same loss function. This unified view enables the development of versatile training strategies that are applicable across learning paradigms. We investigate them on image classification tasks where CLIP exhibits limitations, by varying prompt modalities, e.g., textual or visual prompts, and learning paradigms. We find that (1) unexplored prompt tuning strategies that iteratively refine pseudolabels consistently improve CLIP accuracy, by 19.5 points in semi-supervised learning, by 28.4 points in transductive zero-shot learning, and by 15.2 points in unsupervised learning, and (2) unlike conventional semi-supervised pseudolabeling, which exacerbates model biases toward classes with higher-quality pseudolabels, prompt tuning leads to a more equitable distribution of per-class accuracy. The code to reproduce the experiments is at BatsResearch/menghini-neurips23-code.

## 1 Introduction

Large pre-trained vision-language models (VLMs) [31, 43, 17] achieve remarkable accuracy without task-specific training but still require adaptation for optimal performance. Prompt-tuning [13, 18] is an approach to efficiently enhance VLMs performance on downstream tasks by learning inputs to the model. While learning prompts with a few labeled data can yield significant improvements [48, 2], a broader range of learning settings such as semi-supervised, transductive zero-shot, and unsupervised learning are still underexplored. All of these settings share access to unlabeled data, and the versatile zero-shot classification abilities of VLMs make pseudolabeling a natural approach to leveraging it. This paper investigates how the use of out-of-the-box pseudolabels assigned by CLIP can contribute to improving CLIP's own performance. To this end, we conduct an extensive exploration of learning scenarios by varying prompt modalities, learning paradigms, and training strategies. We present empirical evidence showcasing the effectiveness of iterative prompt-training strategies that leverage CLIP-based pseudolabels, regardless of learning paradigms and prompt modalities, resulting in significant improvements in CLIP's image classification performance across different settings.

Pseudolabels are heuristic labels assigned by a model to unlabeled data, which are leveraged to further train the model [20]. Successful training with pseudolabels relies on two factors: the qualityof the labels and how they are used during training. To address the first, conventional methods assign labels to instances with high-confidence predictions [36]. For pseudolabeling using CLIP, Huang et al. propose to select the most confident samples for each class [15], mitigating CLIP's bias [38] and miscalibration [22] (see Section 3). To assign pseudolabels, we rely on this approach and address the second point by exploring how to make the best use of them. We design a broad space of analysis considering three dimensions: prompt modalities, which are the model inputs we learn; learning paradigms, which define the data we have available; and training strategies, which describe the process used to optimize performance (Figure 1).

Research on prompt tuning has demonstrated that training strategies used for learning prompts in one modality can be transferred to learning prompts in a different modality. For instance, Visual Prompt Tuning [18] was originally designed to effectively fine-tune large vision models but can be adapted to efficiently fine-tune CLIP using the same training strategy as standard textual prompt tuning [48, 34, 44]. On the contrary, different learning paradigms with limited labeled data typically require distinct approaches specifically tailored to extract information from the available data [27, 12]. However, we observe that this changes by using VLM's generated pseudolabels. Unlike conventional pseudolabeling approaches that bootstrap off labeled data and are used as semi-supervised learning techniques [36, 3, 40], VLMs can generate pseudolabels in any learning setting. This offers a significant advantage, expanding the scope of pseudolabeling beyond semi-supervised learning, and making it a promising approach for other settings, such as transductive zero-shot and unsupervised learning. By using CLIP-based pseudolabels as a source of supervision, we can view these settings as optimizing the same loss function, which is simply a weighted sum of the errors on labeled data, if available, and pseudolabeled data. Given that we can express different settings as the same problem, we can propose training strategies, i.e., the way of using pseudolabels, that suit them all.

By standardizing the training strategies across various prompt modalities and learning settings, we can conduct experiments on different applications of pseudolabels for various combinations of prompt modalities, learning paradigms, and training strategies, as illustrated in Figure 1. To the best of our knowledge, only one potential path has been explored thus far; specifically, fine-tuning textual prompts in an unsupervised learning context using a few pseudolabels [15]. Rather than relying on a fixed set of pseudolabels, we propose iterative training techniques that allow for the ongoing refinement and expansion of the pool of pseudolabeled data used during training. With each iteration, we progressively enhance CLIP's pseudolabeling ability, allowing us to extend the set of pseudolabeled data while maintaining the high quality of the initial pseudolabels.

We conduct experiments on six tasks where CLIP has been observed to underperform [31], such as satellite-image classification, flower-species identification, and texture-image recognition, among others. Our findings reveal that iterative approaches effectively fine-tune prompts irrespective of their modality and learning paradigms. Recent studies have identified the "Matthew effect" as a potential issue for semi-supervised models that use pseudolabels [49, 38]. This phenomenon causes models to perform well on classes with accurate pseudolabels but poorly on those with inaccurate ones, thereby reinforcing the model's original bias towards certain classes. Our analysis reveals that using pseudolabels generated by CLIP for prompt-tuning with iterative strategies not only improves CLIP's overall performance but also corrects its natural bias towards certain classes.

We summarize the main takeaways of our work:

Figure 1: Our design space to explore the effect of leveraging pseudolabels in a unified way across prompt modalities, learning paradigms, and training strategies. The green (dashed) path has already been explored [15], while the red (solid) lines are the unexplored combinations for prompt tuning.

 * General purpose zero-shot learners used as general purpose pseudolabelers open the opportunity to develop training strategies that leverage pseudolabeled data beyond semi-supervised learning. We point out that different learning paradigms, such as semi-supervised, transductive zero-shot, and unsupervised learning, can be all considered as special cases of a single objective function, by using pseudolabels as a source of supervision.
* We demonstrate that simple iterative training strategies for refining pseudolabels are highly effective approaches for limited-label prompt tuning. In fact, regardless of the prompt modality and learning setting, these strategies improve CLIP, by on average 19.5 points in semi-supervised learning, 28.4 in transductive zero-shot learning, and 15.2 in unsupervised learning.
* We show that prompts learned with iterative strategies help mitigate the "rich get richer, poor get poorer" effect observed in semi-supervised approaches leveraging pseudolabels. By redistributing the quality of pseudolabels across different classes, we observe a "Robin Hood effect" where the extremely rich classes' accuracy stays the same or decreases, while poorer classes get richer, leading to a more equitable distribution of per-class accuracy.

## 2 Background and related work

Vision-language modelsVision-language models such as CLIP [31], ALIGN [17], and Florence [43] are models that align images and text. We focus on CLIP, which is composed of two components: a text encoder, \(\psi\), and an image encoder, \(\phi\), which are jointly trained using a contrastive loss to learn a multi-modal embedding space which aligns the representations of similar text and image inputs. This pre-training enables CLIP to perform zero-shot image classification. Given an image \(x\) and a set of classes \(\mathcal{Y}=\{y_{1},...,y_{C}\}\), CLIP classifies \(x\) by measuring the similarity between the image representation \(z=\phi(x)\) and each class representation \(w_{i}=\psi(\pi_{i})\), based on their cosine distance in the shared embedding space. Here, \(\pi_{i}\) is a natural language prompt such as "a photo of a [CLASS\({}_{i}\)]", where CLASS\({}_{i}\) is the specific class name, such as "orange dahlia," "forest" or "Boeing 737". The image \(x\) gets assigned to the class with the highest similarity score. In this work, we study how to learn better prompts that enhance CLIP by leveraging pseudolabels.

Prompt tuningPrompt tuning is a technique that enhances the practical application of large pre-trained models like CLIP [31] and GPT [32; 7]. It involves providing task-specific information to the model during inference through textual or visual inputs, leading to improved performance on downstream tasks [1; 33; 6; 7]. While discrete prompts are manually crafted natural language descriptions of classes that guide the model, they may not yield optimal results [47]. Soft prompting [21; 24], on the other hand, optimizes prompts as continuous vectors. These can be optimized by backpropagating through the frozen pre-trained model, resulting in better performance. Soft prompts can be learned for various modalities, e.g., text or image, [48; 13; 17; 2; 44; 19] and applications [34; 27; 12; 28] by training on a small number of labeled examples per class. If only unlabeled data is accessible, it is possible to learn textual soft prompts by leveraging CLIP-based pseudolabels [15]. Expanding on this concept, we further investigate the use of pseudolabels across a broader range of prompt modalities and learning approaches, and we introduce unexplored training strategies to leverage pseudolabels more effectively.

Learning from pseudolabelsPseudolabeling is the practice of assigning labels to unlabeled data based on the prediction of a model [20]. Then, pseudolabels are used to improve the performance of the model itself. There are different ways to obtain and use pseudolabels and each impacts the final predictions of the model [41; 45; 16; 35]. Some approaches use confidence thresholds [36; 3; 40] and others average predictions from multiple augmentations [4]. Pseudolabeling is a semi-supervised learning technique, and it is rarely used in transductive zero-shot learning [42; 5; 25]. Applying such techniques requires a few labeled examples related to the target task to learn a baseline model capable of pseudolabeling. However, this limitation has been overcome by VLMs, which are capable of pseudolabeling examples without task-specific training. The conventional pseudolabeling scheme based on confidence threshold is not effective if we assign pseudolabels based on CLIP. In fact CLIP is miscalibrated [22] and has imbalanced predictions [38] which may induce noise in the pseudolabels. An alternative approach selects the top-K most confident examples per class to improve performance [15]. In our analysis, we rely on this scheme (Section 3).

Design space

Our analysis encompasses the design space consisting of various combinations of prompt modalities, learning paradigms, and training strategies (Figure 1). Within this space, two key components remain constant: the pseudolabeling scheme and a unified loss function. This section begins by introducing these components and subsequently delves into a comprehensive discussion of each dimension within the design space to be explored.

Pseudolabeling schemeThe use of CLIP to generate pseudolabels has been investigated in [15]. Given unlabeled data \(X_{u}\) with target classes \(\{y_{1},...,y_{C}\}\), the goal is to assign labels to data points in which the model is most confident. Typically, pseudo labeling schemes use a confidence threshold (\(P(y|x)>\tau\)) to select instances to pseudolabel. However, this approach does not work well for CLIP due to its miscalibration [22] and imbalanced predictions [38]. Instead, one can use a top-K pseudo labeling approach, where the top-K most confident examples per class are used as pseudolabeled data [15]. The pseudolabel assignment consists of (1) computing the similarity scores of each datapoint with classes' textual prompts, and (2) select for each class the \(K\) datapoints with the highest similarity score to the class. In this way, we always get \(K\) pseudolabels per class, effectively addressesing the natural bias in CLIP's pseudolabels [38]. In Appendix A.1, we provide more details about pseudolabel assignment corner cases.

This top-K pseudolabeling scheme is applicable to unlabeled data, regardless of the availability of labeled data. As a result, we can extend the use of pseudolabels to any learning setting that involves unlabeled data. We observe that by treating pseudolabeled examples as true labeled data, we can view all learning settings as optimizing the same objective function.

Unified objective functionConsider a \(C\)-class image classification task, where \(X_{L}\) and \(Y_{L}\) represent the image representations and labels of the labeled data, and \(X_{U}\) and \(\tilde{Y}_{U}\) denote the image representations and pseudolabels for the unlabeled data. We define a loss function that combines two cross-entropy losses, one accounting for the error on the labeled data points and the other accounting for the error on pseudolabeled data:

\[\mathcal{L}=\mathcal{L}_{CE}(X_{L},Y_{L})+\lambda\,\mathcal{L}_{CE}(X_{U}, \tilde{Y}_{U}) \tag{1}\]

where \(\gamma\) and \(\lambda\) define the training balance between the errors on labeled and pseudolabeled data.

### Prompt modalities

Learning prompts is the process of training a set of vectors \(\mathrm{P}=[\mathrm{p}]_{1}\ldots[\mathrm{p}]_{K}\) that are prepended to the textual or visual inputs of the encoders within the CLIP architecture. By prepending these vectors to specific inputs, we can learn _textual_ prompts, _visual_ prompts, or _multimodal_ prompts when applying a set of vectors to both inputs simultaneously. We provide a technical and detailed explaination in Appendix A.1.

In our exploration, we consider all three types of prompts. The efficacy of prompts can vary depending on the task. Text prompt tuning may be most beneficial when image features are well-separated by class but may not be aligned with the corresponding textual prompt. Visual prompts rearrange the image features within the projection space, and it has the potential to improve CLIP when the pre-trained image features are not well separated by class. Finally, multimodal prompts allows for beneficial interaction between the two separate modalities, which might lead to both separable visual features, and text classifiers that are well-aligned with the corresponding visual features.

### Learning paradigms

By adjusting the values of parameters \(\gamma\) and \(\lambda\) and using the appropriate sets of labeled and pseudolabel data, the unified objective loss can be customized for each learning paradigm. We note that the redundancy in the use of parameters is for notation clarity in descriptions of the different strategies below. One can simply use \(\lambda\) as balancing factor between labeled and pseudolabeled data.

Semi-supervised learningIn the semi-supervised learning (SSL) scenario we have access to a limited number of labeled data for all the target classes \(D_{L}=\{(x,y)\}\) where \(x\) is an input feature and \(y\in\mathcal{Y}=[C]\) is the corresponding label. In addition, we have access to unlabeled data \(X_{U}=\{x\}\), where \(x\) is an image in the target domain \(\mathcal{Y}\). From \(X_{U}\), we get \(\mathcal{D}_{PL}=\{(x,\tilde{y})\}\), where \(\tilde{y}\in[C]\) is \(x\)'s pseudolabel. When using the unified loss in this setting, we set \(\gamma\) to \(|\mathcal{D}_{PL}|/|\mathcal{D}_{L}|\). As \(|\mathcal{D}_{L}|\) is much smaller than \(|\mathcal{D}_{PL}|\), \(\gamma\) acts as an upweighting factor for the few-labeled instances, thus counterbalancing the learning effect of pseudolabels (\(\lambda\)=1).

Transductive zero-shot learningIn transductive zero-shot learning (TRZSL), we are provided with labeled data \(D_{L}=\{(x,y)\}\) for some target classes \(S\) (referred to as _seen_ classes), where \(x\) represents input features, and \(y\in[S]\) is the corresponding label. Additionally, we have access to unlabeled data \(X_{U}=\{x\}\) for a disjoint set of classes \(U\) (referred to as _unseen_ classes). Using \(X_{U}\), we obtain \(\mathcal{D}_{PL}=(x,\tilde{y})\), where \(\tilde{y}\in[U]\) denotes the pseudolabels for \(x\). The value of \(\lambda\) in the unified loss is set to \(|\mathcal{D}_{L}|/|\mathcal{D}_{PL}|\), which makes the weight of the pseudolabel loss equivalent to that of the labeled data (\(\gamma=1\)). This is necessary because an imbalance in the number of labeled and pseudolabeled samples can result in a skewed training distribution, leading to better performance on seen classes while the performance on unseen classes may either remain stagnant or degrade. Studying this setting is interesting beyond transductive zero-shot learning. In fact, it has the potential to generalize to scenarios where the target task involves unseen classes, while the seen classes consist of auxiliary labeled data from the same domain but different task [30].

Unsupervised learningIn the unsupervised learning (UL) setting, we have access only to unlabeled data \(X_{U}=\{x\}\), from which we obtain \(\mathcal{D}_{PL}=(x,\tilde{y})\), where \(\tilde{y}\in[C]\) denotes the pseudolabel for \(x\). In this case, \(\gamma\) is set to 0, as there is no labeled data, and \(\lambda=1\). The use of this setting was initially explored in [15], who leveraged a few pseudolabels per class to learn textual prompts. In this paper, we build on their work by investigating a variety of training strategies and prompt modalities.

Supervised learningIn supervised learning (SL), we are only provided with labeled data \(D_{L}=(x,y)\), where \(x\) represents an input feature, and \(y\in[C]\) is the corresponding label. If we set \(\lambda\) to 0, the unified loss function is equivalent to the objective functions of default prompt-tuning approaches that optimize the prompts using a few labeled instances per target class. This setting is not strictly part of our design space. However, we will refer to it to define baselines in Section 4.

### Training strategies

The unified objective function enables the development of training strategies broadly applicable across various learning paradigms. We explore three distinct learning strategies to effectively use pseudolabels in this context. The first strategy uses pseudolabels in a static manner. The other two strategies, which are unexplored for prompt tuning, involve the dynamic use of pseudolabeled data.

Few-pseudolabels (FPL)We select \(K\) pseudolabels per target class, resulting in a pseudolabeled dataset of size \(K\cdot C\). We learn the prompts by minimizing the objective function via backpropagation through CLIP's encoders. This strategy aligns with Unsupervised Prompt Learning (UPL) in [15]. We refer to it as few-pseudolabels (FPL) to encompass its applicability for learning prompts of diverse modalities across learning paradigms.

Iterative Refinement of FPL (IFPL)Similar to FPL, we obtain the top-K pseudolabels for each target class. These pseudolabels are then used to train a new task-specific prompt. After completing the training, we use the learned prompt to compute the top-K pseudolabels per class again. Subsequently, we reinitialize the prompt and repeat this entire process for a total of \(I\) iterations. With this iterative approach, if training with the initial pseudolabel set leads to an improvement in the model's performance, the model itself can become a more effective pseudolabeler, refining the pseudolabels in each subsequent iteration.

Grow and Refine Iteratively Pseudolabels (GRIP)Although IFPL can improve the quality of the \(K\times C\) pseudolabels used for training, it still limits learning to a few examples per target class. To overcome this constraint, we explore a method similar to IFPL, but with a key difference. In each iteration, we progressively increase the value of \(K\). Specifically, during the \(i\)-th iteration, we use \(K=(i\times\frac{|X_{U}|}{I})/C\) of the unlabeled data to perform the steps in the iterative process. GRIP maintains class balance by selecting the top-K samples at each iteration, with \(K\) increasing progressively. Similar to IFPL, both prompts and pseudolabels are reinitialized with every iteration, in order to avoid accumulating errors from earlier iterations. In other words, learning progresses from pseudolabels to new prompts to new pseudolabels, and so on. The rationale behind this strategy is that as the model's accuracy in generating pseudolabels improves, we can increase the total number of pseudolabels without introducing excessive noise.

## 4 Experiments

We explore the design space outlined in Section 3 to understand the effectiveness of leveraging pseudolabels for limited-label prompt tuning. We show that (1) iterative strategies significantly improve CLIP's performance across prompt modalities and learning settings, (2) using CLIP-based pseudolabels with iterative strategies induces a more equitable distribution of per-class accuracy.

DatasetsWe conduct the analysis on six tasks, covering specialized and fine-grained domains, where CLIP shows deficiencies [31]. We call this set of tasks FRAMED, and it includes Flowers102 [29], RESICS45 [9], FGVC-Aircraft [26], MNIST [11], EuroSAT [14], and DTD [10]. For each dataset we use the training and test splits provided in [23]. For the transductive zero-shot learning setting we randomly generate three splits of seen and unseen classes with a 62-38 ratio. Further details are in Appendix A.2.

BaselinesTo evaluate the effectiveness of the training strategies described in Section 3.3, we compare the performance of CLIP when queried with the learned soft prompts to CLIP zero-shot with default prompts such as "a photo of a [CLASS]." In addition, we compare with default supervised prompt-tuning baselines, for which we only use the available labeled data: CoOp [48] for textual prompts, VPT [18] for visual prompts, and UPT [44] for multimodal prompts. We defer to Appendix A.1 the technical details of these methods.

Evaluation metricsWe assess the performance of each method by measuring the accuracy of the test set, averaging the results over five runs. In the case of TRZSL, we report the harmonic of the accuracies of seen and unseen classes to account for their potentially imbalanced performance [39].

Training settingsFor all experiments, datasets, and learning strategies, we use ViT-B/32 as the vision backbone. For both visual and textual prompt learning, we set the prefix size to 16 [48; 18]. Multimodal prompts have length 8 [44]. We use SGD as the optimizer and train for 150 epochs. We use 5 warmup epochs at a learning rate of 0.0001, and then set the learning rate to \(l\), which is decayed by the cosine annealing rule. For textual and visual prompt learning, \(l=0.1\), while for multimodal prompt learning, \(l=0.01\). In SSL, we use 2 labeled samples per class to assess the impact of pseudolabels in the scenario of very few labeled data and abundant unlabeled data. The number of iterations \(I\) is \(10\). FPL and IFPL have the number of pseudolabels per class fixed to 16 since it is indicated as the optimal \(K\) in the previous research on pseudolabeling with CLIP [15]. In general, \(K\) is a hyperparameter that may require optimization in practical cases. We decide to be consistent with the literature and apply this fixed value of \(K\) in order to reduce the addition of more confounding factors in our analysis.

### Exploring the design space

GRIP consistently enhances CLIP across prompt modalities and learning settings Table 1 reports the performance of GRIP, the best performing among the training strategies in Section 3.3, compared to CLIP and prompt-tuning baselines. Overall, GRIP consistently improves the performance of CLIP and the baselines across prompt modalities and learning settings. By tuning textual prompts, the average improvement over CLIP is 20.7 points in SSL, 14.9 in UL, and 32.4 in TRZSL, while the improvement on CoOp is 9.6 points in SSL, and 26.6 in TRZSL. Similar results for the visual prompts show that GRIP improves CLIP by 18.2 points in SSL, 15.7 in UL, and 30.8 in TRZSL, and VPT by 12.9 points in SSL, and 20.8 in TRZSL. We note that CoOp and VPT applied to the SSL setting correspond to learning only on the labeled data, and we do not run them in the UL setting as there is no labeled data. Results are similar for multimodal prompts. We defer them to Appendix A.3, due to space constraints.

No prompt modality is clearly superiorUsing pseudolabels dynamically is beneficial for each modality. However, determining the clear superiority of one prompt modality over the other is challenging, as it depends on the specific tasks. For example, visual prompts work better for EuroSAT, while textual prompts excel in Flowers102. Despite intuitive explanations (Section 3.1), the scientific consensus remains elusive [44]. Hence, we prefer to emphasize that the dynamic use of pseudolabels consistently improves performance for each prompt modality, without declaring one modality as definitively better than the other.

Unsupervised learning is equivalent or more robust than learning with very few shotsThe accuracy of GRIP when applied to the fully unsupervised setting is either higher or equivalent to the accuracy of VPT, which is trained using two labeled instances per class (Table 1). This shows that pseudolabeled data can substitute very few labeled examples for prompt tuning. However, the significant improvement of GRIP over CoOp and VPT in the semi-supervised setting (see Table 1) suggests that leveraging unlabeled data through pseudolabeling is advantageous in scenarios where labeled data is scarce but there is an abundance of unlabeled data.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multicolumn{1}{c}{\multirow{2}{*}{Method}} & \multicolumn{3}{c}{Flowers102} & \multicolumn{3}{c}{RESICS45} & \multicolumn{3}{c}{FGVC Aircraft} \\ \cline{2-10}  & SSL & UL & TRZSL & SSL & UL & TRZSL & SSL & UL & TRZSL \\ \hline CLIP & \(63.67_{0.00}\) & \(63.40_{0.00}\) & \(54.48_{0.00}\) & \(54.46_{0.00}\) & & \(17.58_{0.00}\) & & \(17.86_{0.00}\) & \\ CoOp & \(76.76_{1.11}\) & - & \(63.22_{0.00}\) & \(58.53_{0.01}\) & - & \(63.37_{0.02}\) & & \(14.91_{3.22}\) & - & \(21.70_{0.03}\) \\ GRIP & \(\mathbf{83.6_{0.68}}\) & \(\mathbf{69.84_{1.06}}\) & \(\mathbf{86.26_{0.00}}\) & \(\mathbf{74.11_{0.08}}\) & \(\mathbf{70.55_{0.58}}\) & \(\mathbf{81.07_{0.00}}\) & & \(16.98_{0.02}\) & \(15.22_{0.71}\) & \(\mathbf{26.08_{0.00}}\) \\ \hline \(\Delta\) CLIP & \(\uparrow 19.93\) & \(\uparrow 6.17\) & \(12.28\) & \(\uparrow 19.63\) & \(\uparrow 16.07\) & \(2\cdot 26.61\) & \(\uparrow 0.6\) & \(\downarrow 2.36\) & \(\uparrow 8.22\) \\ \(\Delta\) CoOp & \(\uparrow 6.84\) & - & \(\uparrow 23.04\) & \(\uparrow 15.58\) & - & \(\uparrow 17.70\) & \(\uparrow 2.07\) & - & \(\uparrow 4.38\) \\ \hline \multicolumn{10}{c}{MNIST} & \multicolumn{3}{c}{EuroSAT} & \multicolumn{3}{c}{DTD} \\ \cline{2-10}  & \(2.51_{0.00}\) & \(27.70_{0.00}\) & \(32.88_{0.00}\) & \(30.54_{0.00}\) & & \(43.24_{0.00}\) & & \(43.45_{0.00}\) \\ CoOp & \(56.42_{0.00}\) & - & \(21.50_{0.00}\) & \(\mathbf{59.51_{4.55}}\) & - & \(49.68_{0.00}\) & & \(37.10_{4.56}\) & - & \(46.30_{0.00}\) \\ GRIP & \(\mathbf{71.78_{5.59}}\) & \(\mathbf{67.88_{2.76}}\) & \(\mathbf{74.06_{0.00}}\) & \(\mathbf{58.66_{0.64}}\) & \(\mathbf{57.21_{1.77}}\) & \(\mathbf{92.33_{0.00}}\) & & \(\mathbf{56.07_{0.85}}\) & \(\mathbf{46.09_{1.06}}\) & \(\mathbf{65.30_{0.01}}\) \\ \hline \(\Delta\) CLIP & \(\uparrow 4.68\) & \(\uparrow 42.78\) & \(\uparrow 53.29\) & \(\uparrow 25.78\) & \(\uparrow 24.33\) & \(\uparrow 61.79\) & \(\uparrow 12.83\) & \(\uparrow 2.85\) & \(\uparrow 21.85\) \\ \(\Delta\) CoOp & \(\uparrow 15.36\) & - & \(\uparrow 52.91\) & \(\downarrow 0.85\) & - & \(\uparrow 42.65\) & \(\uparrow 18.97\) & - & \(\uparrow 19.00\) \\ \hline \hline \multicolumn{10}{c}{**Visual prompts**} & \multicolumn{3}{c}{Flowers102} & \multicolumn{3}{c}{RESICS45} & \multicolumn{3}{c}{FGVC Aircraft} \\ \cline{2-10}  & SSL & UL & TRZSL & SSL & UL & TRZSL & SSL & UL & TRZSL \\ \hline CLIP & \(63.67_{0.00}\) & \(63.40_{0.00}\) & \(54.48_{0.00}\) & & \(54.46_{0.00}\) & & \(17.58_{0.00}\) & & \(17.86_{0.00}\) \\ VPT & \(63.73_{1.52}\) & - & \(64.77_{0.00}\) & \(60.80_{0.00}\) & & \(67.06_{0.00}\) & & \(17.76_{0.68}\) & & \(26.69_{0.00}\) \\ GRIP & \(\mathbf{67.95_{1.2}}\) & \(63.09_{0.55}\) & \(\mathbf{77.18_{0.00}}\) & \(\mathbf{71.22_{0.77}}\) & \(\mathbf{68.43_{0.61}}\) & \(\mathbf{82.19_{0.00}}\) & & \(\mathbf{19.45_{0.5}}\) & \(17.51_{0.61}\) & \(26.42_{0.00}\) \\ \hline \(\Delta\) CLIP & \(\uparrow 4.28\) & \(\downarrow 0.58\) & \(\uparrow 13.78\) & \(\uparrow 16.74\) & \(\uparrow 13.95\) & \(\uparrow 27.73\) & \(\uparrow 1.85\) & \(\downarrow 0.07\) & \(\uparrow 8.56\) \\ \(\Delta\) VPT & \(\uparrow 4.22\) & - & \(\uparrow 12.47\) & \(\uparrow 10.42\) & - & \(\uparrow 15.13\) & \(\uparrow 1.67\) & - & \(\downarrow 0.27\) \\ \hline \multicolumn{10}{c}{MNIST} & \multicolumn{3}{c}{EuroSAT} & \multicolumn{3}{c}{DTD} \\ \cline{2-10}  & \(25.10_{0.00}\) & \(20.77_{0.00}\) & \(32.88_{0.00}\) & & \(30.54_{0.00}\) & & \(43.24_{0.00}\) & & \(43.45_{0.00}\) \\ VPT & \(\mathbf{42.53_{1.13}}\) & \(25.51_{0.05}\) & \(47.13_{1.34}\) & & \(\mathbf{62.24_{0.00}}\) & \(36.41_{21.7}\) & - & \(44.16_{0.01}\) \\ GRIP & \(\mathbf{69.66_{1.51}}\) & \(\mathbf{68.04_{1.11}}\) & \(\mathbf{69.54_{0.50}}\) & \(\mathbf{63.48_{0.00}}\) & \(\mathbf{63.68_{0.32}}\) & \(\mathbf{96.97_{0.70}}\) & \(\mathbf{54.57_{0.86}}\) & \(\mathbf{50.51_{0.50}}\) & \(\mathbf{62.78_{0.00}}\) \\ \hline \(\Delta\) CLIP & \(\uparrow 4.56\) & \(\uparrow 42.94\) & \(\uparrow 48.77\) & \(\uparrow 30.60\) & \(\uparrow 30.80\) & \(\uparrow 66.43\) & \(\uparrow 11.33\) & \(\uparrow 7.27\) & \(\uparrow 19.33\) \\ \(\Delta\) VPT & \(\uparrow 27.14\) & - & \(\uparrow 44.03\) & \(\uparrow 16.35\) & - & \(\uparrow 34.73\) & \(\uparrow 18.16\) & - & \(\uparrow 18.62\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: For each learning paradigm, we compare the accuracy of GRIP with CLIP zero-shot (ViT-B/32), CoOp, and VPT. Results are for SSL, UL, and TRZSL on FRAMED. We average the accuracy on 5 seeds and report the standard deviation. \(\Delta\) METHOD is the difference between the accuracy of GRIP and METHOD. We note that for UL we can not apply CoOp and VPT since no labeled data is available.

Figure 2: Balance of seen and unseen accuracies vs. model’s overall accuracy. Points close to 0 indicate a good balance. Negatives represent better accuracy for the seen classes.

**Transductive zero-shot learning effectively transfers knowledge** In the TRZSL setting, GRIP improves over CLIP and the baselines by a large margin (Table 1). Figure 2 displays the balance of seen and unseen classes of each method alongside its accuracy. The _class balance_ is \((acc_{unseen}-acc_{seen})/acc_{seen}\), where values close to zero indicate a good balance, negative values indicate better accuracies for seen classes, and positive values indicate better accuracies for unseen classes. Methods employing an iterative usage of pseudolabels maintain a good balance, as opposed to CoOp/VPT and FPL. This balance in accuracy is likely a combined effect of the quality of the pseudolabels and the transfer of knowledge from the seen to the unseen classes. The latter point is significant because it implies that even if we only possess unlabeled data for a specific target task, we can still use labeled data from related classes [30] within the same domain to enhance CLIP's performance.

**There is a trade-off between quality and quantity of pseudolabels** Table 2 shows the performance of CLIP employing prompts learned with different training strategies, all leveraging pseudolabels (Section 3.3). Iterative strategies are more effective than FPL which, similarly to [15], use a static set of a few pseudolabels for one iteration.

On Flowers102, RESICS45, and DTD, IFPL improves on average FPL by 5.6 points in SSL, 1.7 in UL, and 5.6 in TRZSL. GRIP boosts the performance even more by on average 7.8 points in SSL, 3.1 in UL, and 9.7 in TRZSL. Results on the other tasks are comparable or larger and we report them in Appendix A.3 due to space constraints.

Figure 3 shows the progression of pseudolabels quality for the iterative learning of textual prompts. IFPL maintains a fixed set of 16 pseudolabels, improving their quality with each iteration (top x-axis).

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multicolumn{1}{c}{**Textual prompts**} & \multicolumn{6}{c}{**Flowers102**} & \multicolumn{6}{c}{**RESICS45**} & \multicolumn{3}{c}{DTD} \\ \hline Method & SSL & UL & TRZSL & SSL & UL & TRZSL & SSL & UL & TRZSL \\ \hline FPL & \(75.96_{0.74}\) & \(65.67_{0.23}\) & \(80.97_{0.00}\) & \(68.13_{0.55}\) & \(63.07_{0.38}\) & \(72.71_{0.00}\) & \(37.10_{0.45}\) & \(44.96_{0.55}\) & \(46.3_{0.03}\) \\ IFPL & \(78.60_{0.75}\) & \(65.61_{0.28}\) & \(82.08_{0.00}\) & \(70.52_{1.24}\) & \(64.11_{0.18}\) & \(75.51_{0.00}\) & \(\mathbf{55.24}_{0.07}\) & \(\mathbf{47.77}_{1.15}\) & \(59.14_{0.02}\) \\ GRIP & \(\mathbf{83.60_{0.48}}\) & \(\mathbf{69.84_{0.46}}\) & \(\mathbf{86.26_{0.00}}\) & \(\mathbf{74.14_{0.68}}\) & \(\mathbf{70.58_{0.81}}\) & \(\mathbf{81.00_{0.05}}\) & \(\mathbf{56.07_{0.58}}\) & \(\mathbf{46.09}_{0.06}\) & \(\mathbf{65.30}_{0.01}\) \\ \hline \(\Delta\) IFPL & \(\uparrow\) 2.72 & \(\uparrow\) 3.89 & \(\uparrow\) 1.11 & \(\uparrow\) 2.39 & \(\uparrow\) 1.07 & \(\uparrow\) 3.4 & \(\uparrow\) 18.14 & \(\uparrow\) 2.81 & \(\uparrow\) 12.84 \\ \(\Delta\) GRIP & \(\uparrow\) 7.64 & \(\uparrow\) 4.17 & \(\uparrow\) 5.29 & \(\uparrow\) 5.89 & \(\uparrow\) 7.48 & \(\uparrow\) 8.96 & \(\uparrow\) 18.97 & \(\uparrow\) 1.13 & \(\uparrow\) 19.00 \\ \hline \hline \multicolumn{1}{l}{**Visual prompts**} & \multicolumn{6}{c}{**Visual prompts**} & \multicolumn{6}{c}{**Visual prompts**} & \multicolumn{6}{c}{**Visual prompts**} & \multicolumn{6}{c}{**Visual prompts**} & \multicolumn{6}{c}{**Visual prompts**} \\ \hline FPL & \(67.03_{0.65}\) & \(\mathbf{65.50_{0.41}}\) & \(71.94_{0.60}\) & \(65.14_{0.25}\) & \(62.24_{0.22}\) & \(67.85_{0.00}\) & \(47.60_{0.40}\) & \(47.60_{0.48}\) & \(52.43_{0.00}\) \\ IFPL & \(\mathbf{68.69_{0.45}}\) & \(\mathbf{66.12_{0.46}}\) & \(76.91_{0.00}\) & \(67.11_{1.19}\) & \(62.93_{1.23}\) & \(73.53_{0.50}\) & \(\mathbf{51.65}_{0.00}\) & \(\mathbf{50.34}_{0.65}\) & \(57.86_{0.01}\) \\ GRIP & \(\mathbf{67.95_{1.2}}\) & \(63.00_{0.56}\) & \(\mathbf{77.18_{0.00}}\) & \(\mathbf{71.22_{0.77}}\) & \(\mathbf{68.50_{0.51}}\) & \(\mathbf{82.19_{0.00}}\) & \(\mathbf{54.57_{1.86}}\) & \(\mathbf{50.51_{0.59}}\) & \(\mathbf{62.78}_{0.00}\) \\ \hline \(\Delta\) IFPL & \(\uparrow\) 1.66 & \(\downarrow\) 0.38 & \(\uparrow\) 4.97 & \(\uparrow\) 1.97 & \(\uparrow\) 0.69 & \(\uparrow\) 5.68 & \(\uparrow\) 4.05 & \(\uparrow\) 2.65 & \(\uparrow\) 5.43 \\ \(\Delta\) GRIP & \(\uparrow\) 0.92 & \(\downarrow\) 3.41 & \(\uparrow\) 5.24 & \(\uparrow\) 6.08 & \(\uparrow\) 6.19 & \(\uparrow\) 14.34 & \(\uparrow\) 6.97 & \(\uparrow\) 2.82 & \(\uparrow\) 10.35 \\ \hline \hline \multicolumn{1}{l}{**Multimodal prompts**} & \multicolumn{6}{c}{**Multimodal prompts**} & \multicolumn{6}{c}{**Visual prompts**} & \multicolumn{6}{c}

On the other hand, GRIP and CLIP expand pseudolabels by incorporating an additional decile of unlabeled data in each iteration (bottom x-axis). Initially, GRIP maintains accuracy, but as it nears completion the quality tends to decrease, while a larger dataset with good-quality pseudolabels becomes available.

Comparing GRIP and CLIP, GRIP's expanded pseudolabels exhibit superior quality and performs better (Table 1). Even though IFPL's pseudolabel accuracy surpasses GRIP in the final iteration, GRIP's overall performance remains better due to training on a larger number of pseudolabels (Table 2). This suggests that numerous, slightly noisier pseudolabels can yield better results, highlighting a trade-off and offering insights for future approaches.

GRIP benefits adaptation even for larger image encodersWe measure how much the effect of the iterative strategies changes if we consider a larger pre-trained image encoder. In Table 3, we report the average improvements of GRIP on CLIP for Flowers102, RESICS45, and DTD. The magnitude of improvements slightly decreases when using a larger image encoder. However, we still see significant benefits for both modalities. The smaller relative improvements with respect to smaller visual encoders align with our expectations. Larger encoders possess a stronger base knowledge, making it relatively more challenging to attain further improvements on top of it. In Table 10, we break down the accuracy of CLIP with different backbones. The performance of the larger backbone is higher, indicating higher quality pseudolabels.

### The Robin Hood effect

Although training models with pseudolabels can lead to good performance, it can also result in biased predictions and generate disparate impacts on sub-populations, i.e., the "Matthew effect" [49; 8]. Particularly, the use of pseudolabels can lead to improved performance in _well-behaved_ (high accuracy) classes but can cause stagnation or decreased performance in _poorly behaved_ (low accuracy) classes. As we explore the use of pseudolabels, we investigate how the accuracy of the analyzed approaches distributes across classes. Figure 4 show an opposite scenario from typical SSL. The solid line represents the sorted per-class accuracies of CLIP. The arrows indicate the per-class accuracies of GRIP. For all learning paradigms, the iterative training strategies increase the accuracy of classes where CLIP is not proficient, while maintaining or decreasing the accuracy of initially well-behaved classes. This effect, which we call the "Robin Hood effect," is very interesting because it shows how CLIP can mitigate its own bias toward certain classes by learning from itself.

To understand the roots of the Robin Hood effect, we examine two factors: (1) the role of pseudolabels generated by CLIP, and (2) the role of prompt tuning. To disentangle these factors, we explore the variation in per-class accuracy of a basic linear classifier trained on CLIP's ViT-B/32 image representation.

"Second generation" pseudolabels are a good treatment for class disparityWe train the linear classifier in the SSL setting on 2 labeled examples per class and pseudolabels. The pseudolabels are obtained through conventional methods, where a threshold of.95 is applied, or by using CLIP to generate 16 pseudolabels per class.

Figure 4: Improvements of FPL and GRIP on CLIP’s per-class accuracies (RESICS45). The x-axis is the ranked class index, while y-axis is the accuracy.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Textual prompts** & & & & \\ \cline{2-4}  & SSL & UL & TRZSL \\ \hline Avg. \(\Delta\) CLIP (ViT-B/32) & \(17.46_{12.83}\) & \(8.36_{28.85}\) & \(23.77_{2.51}\) \\ Avg. \(\Delta\) CLIP (ViT-L/14) & \(15.85_{44.4}\) & \(8.16_{1.22}\) & \(19.96_{19.19}\) \\ \hline
**Visual prompts** & & & \\ \cline{2-4}  & SSL & UL & TRZSL \\ \hline Avg. \(\Delta\) CLIP (ViT-B/32) & \(10.78_{-42.8}\) & \(6.88_{-0.58}\) & \(20.28_{13.78}\) \\ Avg. \(\Delta\) CLIP (ViT-L/14) & \(7.61_{3.27}\) & \(4.89_{-0.48}\) & \(16.14_{11.13}\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Average improvement of GRIP with different backbones on Flowers102, RESICS45, and DTD. \(\Delta\) CLIP is the difference between the accuracy of GRIP and CLIP. Alongside the average, we provide the minimum improvement across tasks.

We find that both approaches yield similar overall accuracies. However, we observe the "Matthew effect" when using the first approach. In contrast, when using CLIP-based pseudolabels, the class disparity of the regressor trained solely on seen classes is reduced. Particularly, we see a significant improvement on initially poor classes, together with a significant diminishing of the accuracy of well-behaved classes. We observe a clear manifestation of the "Robin Hood effect." We present plots illustrating this effect in Appendix A.4.

Prompt tuning retains the accuracy of already rich classes better than linear probingTo evaluate the role of prompt tuning in the "Robin Hood effect," we train a linear classifier and textual prompts in the UL setting using GRIP's training strategy. Comparing the per-class accuracies of the two approaches, GRIP on prompts shows an average improvement of 22.85 points for the poor classes across tasks, along with a slight average decrease of 0.3 points for the rich classes. On the other hand, linear probing determines a 14.42 points improvement for the poor classes, but it results in an average decrease of 9.39 points in accuracy for the rich classes (Appendix A.4).

## 5 Conclusions

We show that prompt tuning using pseudolabels generated by CLIP itself is a successful approach to enhance CLIP across various learning settings. Training strategies that iteratively refine pseudolabels turn out to be effective ways of leveraging pseudolabeled data. These approaches not only enhance CLIP's accuracy but also mitigate model biases toward certain classes. We hope this work lays a solid groundwork for reducing reliance on labeled data when adapting pre-trained vision-language models like CLIP to new tasks.

LimitationsThe effectiveness of the training strategies examined in this paper depends on both the strategies themselves and the quality of pseudolabels. The latter is particularly crucial. If CLIP performs poorly on a task, we may struggle to obtain a reliable set of pseudolabels to begin with, potentially diminishing CLIP's performance. Despite this potential risk, we have not observed any relevant failure of GRIP, even in tasks where CLIP's initial accuracy is extremely low (such as FGVC Aircraft). Also, the pseudolabeling strategy we adopt involves selecting \(K\) pseudolabels per class, which can create a strong assumption about the distribution of the training data if we attempt to cover all unlabeled data. During the final iteration, it is as if we assume a uniform class balance.

Another important consideration is the efficiency of the explored methods. Repeating the training process multiple times brings impressive improvements at the cost of a non-negligible increase of computation time. At each iteration, we generate pseudolabels for the unlabeled data from scratch. While we parallelized the pseudolabeling procedure to cut some of the cost, reducing those for the iterative training presents more significant challenges. We decided to mainly focus on the analysis of qualitative and quantitative effects of pseudolabels in prompt tuning. Future research should address budget constraints and investigate optimal stopping criteria for the iterative process, considering the possibility of reaching a plateau or decreased pseudolabel quality after a certain point, to maximize efficiency while maintaining performance.

## Acknowledgments and Disclosure of Funding

We are thankful to our reviewers for the fruitful and insightful discussions that contributed to the refinement of the paper. We also thank Reza Esfandiarpoor and Zheng-Xin Yong for the comments on our drafts. This material is based on research sponsored by Defense Advanced Research Projects Agency (DARPA) and Air Force Research Laboratory (AFRL) under agreement number FA8750-19-2-1006. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Defense Advanced Research Projects Agency (DARPA) and Air Force Research Laboratory (AFRL) or the U.S. Government. We gratefully acknowledge support from Google and Cisco. Disclosure: Stephen Bach is an advisor to Snorkel AI, a company that provides software and services for data-centric artificial intelligence.

## References

* Bach et al. [2022] Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. PromptSource: An integrated development environment and repository for natural language prompts. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics, 2022.
* Bahng et al. [2022] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. _arXiv preprint arXiv:2203.17274_, 2022.
* Berthelot et al. [2020] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In _International Conference on Learning Representations_, 2020.
* Berthelot et al. [2019] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In _Advances in Neural Information Processing Systems_, 2019.
* Bo et al. [2021] Liu Bo, Qiulei Dong, and Zhanyi Hu. Hardness sampling for self-training based transductive zero-shot learning. _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 16494-16503, 2021.
* Bommasani et al. [2021] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa D'Oumbuya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajah, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel J. Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramer, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. _ArXiv_, abs/2108.07258, 2021.
* Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In _Advances in Neural Information Processing Systems_, 2020.
* Chen et al. [2022] Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, and Mingsheng Long. Debiased self-training for semi-supervised learning. In _Advances in Neural Information Processing Systems_, 2022.
* Cheng et al. [2017] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. _Proceedings of the IEEE_, 2017.

* [10] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. _2014 IEEE Conference on Computer Vision and Pattern Recognition_, 2013.
* [11] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. _IEEE Signal Processing Magazine_, 2012.
* [12] Yunhe Gao, Xingjian Shi, Yi Zhu, Hongya Wang, Zhiqiang Tang, Xiong Zhou, Mu Li, and Dimitris N. Metaxas. Visual prompt tuning for test-time domain adaptation. _ArXiv_, abs/2210.04831, 2022.
* [13] Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. _ArXiv_, abs/2202.06687, 2022.
* [14] Patrick Helber, Benjamin Bischke, Andreas R. Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_, 2017.
* [15] Hao Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. _ArXiv_, abs/2204.03649, 2022.
* [16] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 5065-5074, 2019.
* [17] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In _International Conference on Machine Learning_, 2021.
* [18] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In _ECCV 2022: 17th European Conference on Computer Vision_, 2022.
* [19] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. _ArXiv_, abs/2210.03117, 2022.
* [20] Dong-Hyun Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. _ICML 2013 Workshop : Challenges in Representation Learning (WREPL)_, 2013.
* [21] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, 2021.
* [22] Will LeVine, Benjamin Pikus, Pranav Vishnu Raja, and Fernando Amat. Enabling calibration in the zero-shot inference of large vision-language models. In _ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML_, 2023.
* [23] Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, and Jianfeng Gao. Elevater: A benchmark and toolkit for evaluating language-augmented visual models. In _Advances in Neural Information Processing Systems_, 2022.
* [24] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, 2021.
* [25] Bo Liu, Lihua Hu, Qiulei Dong, and Zhanyi Hu. An iterative co-training transductive framework for zero shot learning. _IEEE Transactions on Image Processing_, 30:6943-6956, 2021.
* [26] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. _ArXiv_, abs/1306.5151, 2013.

* [27] Shu Manli, Nie Weili, Huang De-An, Yu Zhiding, Goldstein Tom, Anandkumar Anima, and Xiao Chaowei. Test-time prompt tuning for zero-shot generalization in vision-language models. In _NeurIPS_, 2022.
* [28] Nihal V. Nayak, Peilin Yu, and Stephen Bach. Learning to compose soft prompts for compositional zero-shot learning. In _The Eleventh International Conference on Learning Representations_, 2023.
* [29] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. _2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing_, 2008.
* [30] Wasu Piriyakulkij, Cristina Menghini, Ross Briden, Nihal V. Nayak, Jeffrey Zhu, Elaheh Raisi, and Stephen H. Bach. TAGLETS: A system for automatic semi-supervised learning with auxiliary data. In _Conference on Machine Learning and Systems (MLSys)_, 2022.
* [31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In _International Conference on Machine Learning_, 2021.
* [32] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
* [33] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhabalani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesth Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In _International Conference on Learning Representations_, 2022.
* [34] Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E. Gonzalez, Kurt Keutzer, and Trevor Darrell. Multitask vision-language prompt tuning. _arXiv preprint arXiv:2211.11720_, 2022.
* [35] Weiwei Shi, Yihong Gong, C. Ding, Zhiheng Ma, Xiaoyu Tao, and Nanning Zheng. Transductive semi-supervised deep learning using min-max features. In _European Conference on Computer Vision_, 2018.
* [36] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In _Advances in Neural Information Processing Systems_, 2020.
* [37] Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. _ArXiv_, 2022.
* [38] Xudong Wang, Zhi-Li Wu, Long Lian, and Stella X. Yu. Debiased learning from naturally imbalanced pseudo-labels. _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, 2022.
* [39] Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning -- the good, the bad and the ugly. _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, 2017.
* [40] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In _Advances in Neural Information Processing Systems_, 2020.
* [41] Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In _International Conference on Machine Learning_, 2021.

* [42] Yunlong Yu, Zhong Ji, Xi Li, Jichang Guo, Zhongfei Zhang, Haibin Ling, and Fei Wu. Transductive zero-shot learning with a self-training dictionary approach. _IEEE Transactions on Cybernetics_, 48(10):2908-2919, 2018.
* [43] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel C. F. Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. _ArXiv_, abs/2111.11432, 2021.
* [44] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. _ArXiv_, abs/2210.07225, 2022.
* [45] Bowen Zhang, Yidong Wang, Wenxin Hou, HAO WU, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In _Advances in Neural Information Processing Systems_, 2021.
* [46] X. Zhang, Yusuke Iwasawa, Yutaka Matsuo, and Shixiang Shane Gu. Domain prompt learning for efficiently adapting clip to unseen domains. 2021.
* [47] Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. 2021.
* [48] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. _International Journal of Computer Vision_, 2021.
* [49] Zhaowei Zhu, Tianyi Luo, and Yang Liu. The rich get richer: Disparate impact of semi-supervised learning. In _International Conference on Learning Representations_, 2022.

Appendix

We include here extra information that supports the results presented in the main body of the paper.

ReproducibilityWe have provided the code to run the experiments as supplementary material for the submission. However, we plan to release it as an open repository upon acceptance.

### Trainable Prompts

Text Prompt TuningThe primary objective of text prompt tuning is to improve the alignment between the class token and the image features extracted by the image encoder. This is achieved by adding learnable vectors, i.e., _prefix_, before the \(\mathtt{CLASS}\) token to create a contextualized representation. Specifically, the sequence

\[\mathbf{t}=[\mathrm{V}]_{1}[\mathrm{V}]_{2}\ldots[\mathrm{V}]_{M}[\mathrm{CLASS}]\]

is fed into the textual encoder, where each vector \([\mathrm{V}]_{m}\) (\(m\in 1,\ldots,M\)) has the same dimension as word embeddings, and \(M\) is a hyperparameter that determines the length of the prefix.

Context Optimization (CoOp) [48] was the first work to explore continuous prompts for VLMs. Follow-up works have experimented with different training strategies to enhance the generalizability of the learned prompts while preserving the core concept of continuous vector tuning [34, 12, 27, 46, 13, 37].

Tuning the text prefix vector changes the resulting \(n\) linear weight vectors \(w_{i}=\psi(p_{i})\), while leaving the image features unchanged. Therefore, text prompt tuning may be most beneficial when image features are well-separated by class but may not be aligned with the corresponding textual prompt. Conversely, text prompt tuning may not be as effective when the image features are poorly separated, as in specialized or novel domains where CLIP may lack sufficient training data.

Visual Prompt TuningInstead of tuning the text prompts, one can also tune the inputs of the vision encoder. In this case, a learnable visual prefix is prepended to the image tokens as input to the image transformer as follows:

\[\mathbf{\hat{I}}=[\mathrm{p}]_{1}\ldots[\mathrm{p}]_{K}[\mathrm{I}]_{1}\ldots[ \mathrm{I}]_{P}\]

where \(p\) represents a sequence of \(K\) learnable prefix vectors, and \([\mathrm{I}]_{1}\ldots[\mathrm{I}]_{P}\) are the image tokens from the corresponding \(P\) patches of the input images. The new sequence \(\mathbf{\hat{I}}\) is the input to the image encoder \(\phi\).

Visual Prompt Tuning (VPT) was introduced in the context of efficiently adapting pre-trained vision transformers to downstream tasks [18]. However, the approach has since been applied in the context of VLM [34].

Whereas text prompt tuning does not alter the image features, visual prompt tuning does. By rearranging the image features within the projection space, VPT has the potential to improve CLIP when the image features are not well separated by class, such as in specialized domains.

Multimodal Prompt TuningThe previous approaches are unimodal, as they either involve modifying the text or visual input, but never both. This choice may be suboptimal as it does not allow the flexibility to dynamically adjust both representations on a downstream task. Recently, multimodal prompt tuning has been introduced [44, 19]. We focus on Unified Prompt Tuning (UPT) [44] which essentially learns a tiny neural network to jointly optimize prompts across different modalities. UPT learns a set of prompts \(\mathbf{U}=[\mathbf{U}_{T},\mathbf{U}_{V}]\in\mathbb{R}^{d\times n}\) with length \(n\), where \(\mathbf{U}_{T}\in\mathbb{R}^{d\times n_{T}},\mathbf{U}_{V}\in\mathbb{R}^{d\times n_{V}}\). \(\mathbf{U}\) is transformed as follows:

\[\mathbf{U}^{\prime} =\mathrm{SA}(\mathbf{U})+\mathrm{LN}(\mathbf{U})\] \[\mathbf{\hat{U}} =\mathrm{FFN}\left(\mathrm{LN}\left(\mathbf{U}^{\prime}\right)\right)+ \mathrm{LN}\left(\mathbf{U}^{\prime}\right)\]

where \(\mathrm{SA}\) is the self-attention operator, \(\mathrm{LN}\) is the layer normalization operator, and \(\mathrm{FFN}\) is a feed forward network. After transformation, we obtain \(\mathbf{\hat{U}}=\left[\mathbf{\hat{U}}_{T},\mathbf{\hat{U}}_{V}\right]\in\mathbb{R}^{d \times n}\), such that \(\mathbf{\hat{U}}_{T}\) is to be used as a text prompt, and \(\mathbf{\hat{U}}_{V}\) is to be used as a visual prompt.

The author of UPL argue that self-attention allows for beneficial interaction between the two separate modalities, which leads to both separable visual features, and text classifiers that are well-aligned with the corresponding visual features [44].

Prompts initializationWe initialize textual and visual prompts from a normal distribution of mean 0 and variance 0.02. We note that we learn shallow visual prompts by modifying only the input to the image encoder. Multimodal prompts are initialized from a uniform distribution. We found that the latter was not working properly for textual and visual prompts.

Additional training settingsFor training, the batch size is 64.

Additional details about pseudolabels assignmentIt can happen that if \(K\) is too large and the unlabeled dataset is smaller than \(K\times C\), we cannot assign \(K\) samples per class. In this case, we can reduce \(K\) accordingly. Also, we the same sample might get assigned to the pseudolabel lists of different classes. It rarely happens in our experiments cases. Hoever, this is a characteristic of the pseudolabeling strategy proposed in [15], and it can be an object of study for future work motivated by the effectiveness of self-training.

### Datasets details

We use six datasets from specialized or fine-grained domains. Here we provide a description of each of them. In Table 4, we report the details about the number of classes and data available for each dataset. For each dataset, we also show CLIP's prediction distribution over classes Figure 5.

Flowers102 [29]It is a dataset collecting images for 102 flower categories commonly occurring in the United Kingdom. For each class we have between 40 and 258 images. Figure 4(a) shows that CLIP's predictions are skewed toward certain classes, which are predicted more often than what we would expect according to the real class distribution on the test set.

RESICS45 [9]This is a publicly available benchmark for Remote Sensing Image Scene Classification. It collects 45 kind of scenes. Figure 4(b) shows that CLIP predicts more often a subset of classes.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & Num. classes (\(|Y|\)) & Num. seen classes (\(S\)) & Num. unseen classes (\(U\)) & Size training data & Avg. labeled data per class & Size test \\ \hline Flowers102 & 102 & 63 & 39 & 2040 & 16 & 6149 \\ RESICS45 & 45 & 27 & 18 & 6300 & 110 & 25200 \\ FOVC- Aircraft & 100 & 62 & 38 & 6667 & 53 & 3333 \\ MNIST & 10 & 6 & 4 & 60000 & 4696 & 10000 \\ EuroSAT & 10 & 6 & 4 & 27000 & 2200 & 5000 \\ DTD & 47 & 29 & 18 & 3760 & 64 & 1880 \\ \hline \hline \end{tabular}
\end{table}
Table 4: For each dataset we report the number of classes, the number of seen and unseen classes in the TRZSL setting, the size of training data (including both labeled and unlabeled data), the average number of labeled examples per class, and the size of the test set which is the same across learning paradigms. We recall that we use the datasets gathered by the recent ELEVATER [23] benchmark for vision-language models.

Figure 5: For each dataset we show the distribution of CLIP’s predictions over classes on the test set. The blue dots represent the true class distribution.

FGVC-Aircraft [26]It describes the fine-grained task of categorizing aircraft. We consider the task of classifying aircrafts into 100 variants. Also for this task, CLIP assigns images to a reduced set of classes (Figure 4(c)).

Mnist[11]MNIST is a database of handwritten digits. The digits are size-normalized and centered in a fixed-size image. We observe that CLIP never predicts 6 out of 10 classes (Figure 4(c)).

EuroSAT[14]EuroSAT represents the task of categorizing satellite images of scenes. It consists of 10 classes. In Figure 4(e), we show CLIP's predictions distribution over the classes.

Dtd[10]DTD stands for Describable Textures Dataset. It is an evolving collection of textural images in the wild, and it is annotated relying on human-centric attributes, inspired by the perceptual properties of textures. The zero-shot CLIP predictions show the model's bias toward certain classes (Figure 4(f)).

### Experiments

In this section, we report tables and plots that complement the results presented in Section 4.

The effect of GRIP on multimodal promptsTable 5 shows the improvements of GRIP on CLIP and Unified Prompt Tuning (UPL) [44]. Similar to the results in Table 1, GRIP consistently improves CLIP with respect to the baselines. The improvements on CLIP are by 18.2 in semi-supervised learning, 14.8 in unsupervised learning, and 30.7 in transductive zero-shot learning. While GRIP outperforms UPL by 4.7 in semi-supervised learning, and 19.5 in transductive zero-shot learning.

Comparison across iterative strategiesIn Table 6, we report a comparison between FPL and the iterative strategies (IFPL and GRIP) on MNIST, EuroSAT, and FGVC-Aircraft. Results on the other tasks can be found in the main body of the paper Section 4.1. While GRIP largely and consistently outperforms FPL by on average 16.7 points in accuracy, IFPL is not robust and it leads to performances that are inferior to FPL by on average 4.4 points in accuracy.

The evolving accuracy of dynamic pseudolabelsFigure 6 represents the evolution of pseudolabels accuracy during training for all datasets, but Flowers102 and RESICS45 presented in Figure 3. We observe that the accuracy of the pseudolabels characterizes the overall performance of the models reported in Table 6. For instance, IFPL for EuroSAT in the TRZSL setting is highly variable, explaining the low average accuracy of the model on the test set (Table 6). Similarily, for MNIST in the TRZSL we observe that after the first iteration, the pseudolabels get very noisy. When we increase the amount of pseudolabeled data, the accuracy of CLIP is not necessarily constant. This is because as we increase K, we are effectively selecting pseudolabels with lower similarities to the

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multicolumn{6}{l}{**Multimodal prompts**} \\ \hline \hline \multicolumn{6}{l}{**Multimodal prompts**} \\ \hline \hline  & \multicolumn{3}{c}{Flowers102} & \multicolumn{3}{c}{RESICS45} & \multicolumn{3}{c}{FGVC Aircraft} \\ \hline Method & SSL & UL & TRZSL & SSL & UL & TRZSL & SSL & UL & TRZSL \\ \hline CLIP & 63.67\({}_{0.00}\) & 61.40\({}_{0.00}\) & 54.48\({}_{0.00}\) & 54.46\({}_{0.00}\) & **17.58\({}_{0.00}\)** & **17.86\({}_{0.00}\)** & **17.86\({}_{0.00}\)** \\ UPT & 68.03\({}_{1.29}\) & - & 61.05\({}_{0.64}\) & 62.84\({}_{1.06}\) & - & 58.79\({}_{0.00}\) & 11.13\({}_{4.98}\) & - & 15.89\({}_{0.00}\) \\ GRIP & **74.56\({}_{0.22}\)** & 64.82\({}_{1.63}\) & **82.01\({}_{0.10}\)** & **73.68\({}_{0.91}\)** & **63.70\({}_{0.61}\)** & **82.17\({}_{0.76}\)** & 17.36\({}_{0.14}\) & 14.73\({}_{0.08}\) & **17.85\({}_{0.30}\)** \\ \hline \(\Delta\) CLIP & \(\uparrow\) 10.89 & \(\uparrow\) 1.15 & \(\uparrow\) 18.61 & \(\uparrow\) 19.2 & \(\uparrow\) 14.89 & \(\uparrow\) 27.71 & \(\downarrow\) 0.22 & \(\downarrow\) 2.85 & \(\downarrow\) 0.01 \\ \(\Delta\) UPT & \(\uparrow\) 6.53 & - & \(\uparrow\) 20.96 & \(\uparrow\) 10.84 & - & \(\uparrow\) 22.38 & \(\uparrow\) 6.23 & - & \(\uparrow\) 1.96 \\ \hline \hline \multicolumn{6}{l}{**MNIST**} & \multicolumn{3}{c}{EuroSAT} \\ \hline CLIP & 25.10\({}_{0.00}\) & 20.77\({}_{0.00}\) & 32.88\({}_{0.00}\) & 30.54\({}_{0.00}\) & 43.24\({}_{0.00}\) & 43.45\({}_{0.00}\) \\ UPT & **64.44\({}_{0.66}\)** & - & 63.59\({}_{0.11}\) & **68.85\({}_{0.92}\)** & - & 60.43\({}_{0.00}\) & 43.71\({}_{2.18}\) & - & 36.91\({}_{0.00}\) \\ GRIP & **65.94\({}_{2.23}\)** & **68.18\({}_{\infty}\)** & **73.75\({}_{2.93}\)** & **60.38\({}_{2.77}\)** & **61.52\({}_{3.04}\)** & **59.52\({}_{0.40}\)** & **54.07\({}_{2.25}\)** & **47.37\({}_{0.7}\)** & **63.42\({}_{0.00}\)** \\ \hline \(\Delta\) CLIP & \(\uparrow\) 40.84 & \(\uparrow\) 43.08 & \(\uparrow\) 52.98 & \(\uparrow\) 27.5 & \(\uparrow\) 28.64 & \(\uparrow\) 64.98 & \(\uparrow\) 10.83 & \(\uparrow\) 4.13 & \(\uparrow\) 19.97 \\ \(\Delta\) UPT & \(\uparrow\) 2.35 & - & \(\uparrow\) 10.16 & \(\downarrow\) 8.47 & - & \(\uparrow\) 35.09 & \(\uparrow\) 10.36 & - & \(\uparrow\) 26.51 \\ \hline \hline \end{tabular}
\end{table}
Table 5: For each learning paradigm, we compare the accuracy of GRIP with CLIP zero-shot (ViT-B/32), and UPT. Results are for SSL, UL, and TRZSL on FRAMED. We average the accuracy on 5 seeds and report the standard deviation. \(\Delta\) METHOD is the difference between the accuracy of GRIP and METHOD. We note that for UL we can not apply UPL since no labeled data is available.

classes, resulting in a reduction in their accuracy, as shown in the plot. This observation aligns with previous findings in [15].

GRIP performance on transductive zero-shot learningWe show how the effectiveness of GRIP is consistent over the three random splits of seen and unseen classes which we randomly generated. The splits are reported in Table 9. Table 8 gathers the accuracy of seen and unseen classes, along with the harmonic mean for all three splits using textual prompts. Beyond the consistent improvement induced by GRIP training strategy, we observe that the accuracy of GRIP on the seen classes is often lower than the accuracy of CoOp on the same set of classes. During training, for large lambda, the loss component of unlabeled data (unseen classes) is the first to decrease, while the loss on the seen classes reduces later. Thus, we hypothesize that extra training steps might be needed to complete the learning on the labeled data.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multicolumn{1}{c}{**Textual prompts**} & \multicolumn{6}{c}{EuroSAT} & \multicolumn{6}{c}{FGVC Aircraft} \\ \hline Method & SSL & UL & TRZSL & SSL & UL & TRZSL & SSL & UL & TRZSL \\ \hline FPL & \(66.06_{1.10}\) & \(40.03_{2.63}\) & \(73.93_{4.05}\) & \(\mathbf{62.05_{1.64}}\) & \(48.96_{4.49}\) & \(53.70_{58.87}\) & \(\mathbf{20.02_{0.77}}\) & \(\mathbf{16.62_{0.87}}\) & \(17.55_{30.37}\) \\ IFPL & \(59.14_{3.43}\) & \(28.94_{2.05}\) & \(0.00_{0.00}\) & \(\mathbf{61.28_{1.59}}\) & \(\mathbf{56.46_{4.26}}\) & \(14.36_{28.18}\) & \(18.00_{0.35}\) & \(13.80_{0.67}\) & \(21.72_{0.77}\) \\ GRIP & \(\mathbf{71.78_{5.59}}\) & \(\mathbf{67.82_{7.76}}\) & \(\mathbf{74.06_{2.95}}\) & \(56.66_{2.64}\) & \(\mathbf{57.21_{7.17}}\) & \(\mathbf{92.33_{40.09}}\) & \(16.98_{2.02}\) & \(15.20_{1.21}\) & \(\mathbf{26.08_{0.25}}\) \\ \hline \(\Delta\) IFPL & \(\downarrow 6.92\) & \(\downarrow 11.09\) & \(\downarrow 9.73\) & \(\downarrow 0.77\) & \(\uparrow 7.50\) & \(\downarrow 39.34\) & \(\downarrow 2.02\) & \(\downarrow 2.82\) & \(\uparrow 4.17\) \\ \(\Delta\) GRIP & \(\uparrow 5.72\) & \(\uparrow 27.85\) & \(\uparrow 64.33\) & \(\downarrow 3.39\) & \(\uparrow 8.25\) & \(\uparrow 38.63\) & \(\downarrow 3.04\) & \(\downarrow 1.40\) & \(\uparrow 8.53\) \\ \hline \hline \multicolumn{1}{c}{**Visual prompts**} & \multicolumn{6}{c}{} & \multicolumn{6}{c}{} & \multicolumn{6}{c}{} & \multicolumn{6}{c}{} & \multicolumn{6}{c}{} & \multicolumn{6}{c}{} & \multicolumn{6}{c}{} \\ \hline FPL & \(42.84_{4.86}\) & \(39.62_{5.33}\) & \(31.82_{7.53}\) & \(52.43_{5.87}\) & \(48.79_{3.60}\) & \(\mathbf{20.46_{2.06}}\) & \(\mathbf{18.28_{0.33}}\) & \(16.28_{0.45}\) \\ IFPL & \(52.91_{69.79}\) & \(37.17_{67.28}\) & \(38.34_{31.81}\) & \(\mathbf{57.85_{5.82}}\) & \(52.52_{10.00}\) & \(48.13_{11.13}\) & \(18.77_{46.16}\) & \(16.30_{0.37}\) & \(19.29_{0.36}\) \\ GRIP & \(\mathbf{69.66_{5.61}}\) & \(\mathbf{68.04_{1.11}}\) & \(\mathbf{69.54_{3.14}}\) & \(\mathbf{63.48_{2.09}}\) & \(\mathbf{63.63_{2.92}}\) & \(\mathbf{96.97_{3.07}}\) & \(\mathbf{19.45_{30.50}}\) & \(\mathbf{17.51_{0.16}}\) & \(\mathbf{26.42_{0.30}}\) \\ \hline \(\Delta\) IFPL & \(\uparrow 10.07\) & \(\downarrow 2.45\) & \(\uparrow 6.56\) & \(\uparrow 5.38\) & \(\downarrow 16.27\) & \(\downarrow 20.55\) & \(\downarrow 1.37\) & \(\downarrow 1.92\) & \(\uparrow 3.01\) \\ \(\Delta\) GRIP & \(\uparrow 26.82\) & \(\uparrow 28.42\) & \(\uparrow 37.72\) & \(\uparrow 11.01\) & \(\uparrow 14.89\) & \(\uparrow 28.29\) & \(\downarrow 0.71\) & \(\downarrow 0.77\) & \(\uparrow 10.14\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: For each learning paradigm, we compare FPL, IFPL, and GRIP on MNIST, EuroSAT, and FGVC Aircraft. We average across 5 runs and report the standard deviation. \(\Delta\) METHOD is the difference between the accuracy of FPL and METHOD.

Figure 6: We plot the evolution of dynamic-pseudolabels accuracy during training. The rows refer to SSL, UL, and TRZSL, in order. IFPL refers to the top x-axis, while CLIP and GRIP to the bottom.

### The Robin Hood effect

The Robin Hood effect on all tasksFor each dataset, we provide the per-class accuracy distribution of GRIP compared with CLIP, Figure 8. The Robin Hood effect characterizes all the tasks. We observe that for GRIP the increase in overall accuracy corresponds to consistent improvements in the predictions of initially poor classes. By comparing Figure 7 with Figure 8, we see that GRIP reinforces the Robin Hood effect already visible when using FPL in certain cases.

The importance of good quality pseudolabels to mitigate the Matthew effect in SSLIn the SSL setting, we train a logistic regression on top of the visual feature extracted by CLIP's image encoder (ViT-B/32). In Figure 9, we show the per-class accuracy of the final model trained by combining labeled data with either pseudolabels assigned with the conventional scheme (threshold at.95) or 16 CLIP-generated pseudolabels. We compare the two distribution with the per-class accuracy of the model trained solely on the few labeled examples per class (2 instances).

The different impact of prompt tuning and linear probing on the Robin Hood effectWe investigate if there is any difference in the Robin Hood effect when adapting CLIP via prompt tuning or linear probing. We train both relying on the iterative training strategy that grows the set of pseudolabels at each iteration by using the top-\(K\) scheme (Section 3). We consider the UL setting.

Among the set of target classes, we distinguish between _poor_ and _rich_ classes. A class is _poor_, if CLIP's accuracy on that class is lower than its overall accuracy on the task. Otherwise, the class is considered _rich_. Table 7 reports the accuracy of the two approaches, and the accuracy on the poor and rich classes, while highlighting the average effect with respect to CLIP. Training with prompt tuning retains more knowledge of the rich classes than linear probing. Prompt tuning reduces the accuracy on the rich classes by on average 0.3 points, while linear probing has an average deterioration of 9.4. the reduction of accuracy of rich classes is on average 30 times larger than the reduction observed using prompts. Overall, GRIP works better than linear probing. We note that the lower accuracy of linear probing is characterized by a worse ability to correctly predict the rich classes, i.e., "rich get poorer." This is surprising, as we would have expected the errors to concentrate on the poor classes compared to CLIP.

Figure 7: Per-class accuracy of FPL compared to CLIP’s per-class accuracy on Flowers102, FGVC-Aircraft, MNIST, EuroSAT, DTD. **X-axis** is the ranked class index, while the **y-axis** is the accuracy.

Figure 8: Per-class accuracy of GRIP compared to CLIP’s per-class accuracy on Flowers102, FGVC-Aircraft, MNIST, EuroSAT, and DTD. **X-axis** is the ranked class index, while the **y-axis** is the accuracy.

Figure 9: Per-class accuracy of a logistic classifier using conventional pseudolabels (first row) and CLIP-based pseudolabels (second row). The solid orange line represents the per-class accuracy of a logistic regression trained on 2-shots per class. **X-axis** is the ranked class index, while the **y-axis** is the accuracy. We present results for Flowers102, RESICS45, FGVC-Aircraft, MNIST, EuroSAT, and DTD, in order.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & Flowers102 & RESICS45 & FGVC-Aircraft & MNIST & EuroSAT & DTD & Avg. \(\Delta\) \\ \hline Linear probe (LP) & 41.01 & 58.79 & 61.94 & 50.52 & 51.37 & 10.17 & - \\ GRIP & **46.09** & **70.55** & **69.84** & **57.21** & **67.88** & **15.22** & - \\ \hline Rich CLIP & **67.81** & 75.47 & 85.16 & 65.26 & 65.14 & **45.93** & - \\ Rich LP & 52.87 & 69.01 & 79.55 & 67.53 & 50.34 & 29.12 & - \\ Rich GRIP & 56.05 & **78.81** & **86.40** & **71.73** & **77.84** & 31.95 & - \\ \hline \(\Delta\) LP & \(\downarrow 14.92\) & \(\downarrow 6.47\) & \(\downarrow 5.61\) & \(\uparrow 2.26\) & \(\downarrow 14.79\) & \(\downarrow 16.81\) & \(\downarrow\)**9.39** \\ \(\Delta\) GRIP & \(\downarrow 11.76\) & \(\uparrow 3.33\) & \(\uparrow 1.24\) & \(\uparrow 6.46\) & \(\uparrow 12.70\) & \(\downarrow 13.98\) & \(\downarrow 0.33\) \\ \hline \hline Poor CLIP & 25.63 & 35.60 & 27.98 & 11.10 & 3.18 & 5.35 & - \\ Poor LP & 26.50 & 42.77 & 36.25 & 28.34 & 56.76 & 4.77 & - \\ Poor GRIP & **35.03** & **56.85** & **42.82** & **39.88** & **65.08** & **6.31** & - \\ \hline \(\Delta\) LP & \(\uparrow 0.87\) & \(\uparrow 7.18\) & \(\uparrow 8.27\) & \(\uparrow 17.24\) & \(\uparrow 53.58\) & \(\downarrow 0.58\) & \(\uparrow 14.43\) \\ \(\Delta\) GRIP & \(\uparrow 9.4\) & \(\uparrow 21.26\) & \(\uparrow 14.84\) & \(\uparrow 28.78\) & \(\uparrow 61.9\) & \(\uparrow 0.96\) & \(\uparrow\)**22.86** \\ \hline \hline \end{tabular}
\end{table}
Table 7: For each task we report the overall accuracy of linear probing (LP) and GRIP textual along with the accuracy on _poor_ and _rich_ classes. \(\Delta\) METHOD is the difference between the accuracy of CLIP and METHOD. For an overall evaluation of the difference between linear probing and prompt tuning, we report the average difference of LP and GRIP with respect to CLIP on poor and rich classes.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multicolumn{2}{l}{**Split 1**} & \multicolumn{4}{c}{Flowers102} & \multicolumn{4}{c}{RESICS45} & \multicolumn{4}{c}{FGVCAircraft} \\ \hline Method & S & U & H & S & U & H & S & U & H \\ \hline CLIP & \(64.26_{0,00}\) & \(62.56_{0,00}\) & \(63.4_{0,00}\) & \(54.85_{0,00}\) & \(54.85_{0,00}\) & \(54.46_{0,00}\) & \(16.27_{0,00}\) & \(19.79_{0,00}\) & \(17.86_{0,00}\) \\ CoOp & **91.52\({}_{36}\)** & \(48.35_{2.96}\) & \(63.22_{2,60}\) & **84.66\({}_{1.01}\)** & \(50.73_{2.8}\) & \(63.37_{2.23}\) & **34.18\({}_{1.56}\)** & \(16.28_{3.69}\) & \(21.70_{3.45}\) \\ GRIP & \(90.31_{0.51}\) & **82.57\({}_{1.26}\)** & **86.26\({}_{0.81}\)** & \(82.68_{0.47}\) & **79.53\({}_{0.72}\)** & **81.07\({}_{0.37}\)** & 22.25\({}_{0.07}\) & **31.51\({}_{0.59}\)** & **26.08\({}_{0.25}\)** \\ \hline \(\Delta\) CLIP & \(\uparrow 26.05\) & \(\uparrow 20.01\) & \(\uparrow 22.86\) & \(\uparrow 27.83\) & \(\uparrow 25.45\) & \(\uparrow 26.61\) & \(\uparrow 5.98\) & \(\uparrow 11.72\) & \(\uparrow 8.22\) \\ \(\Delta\) CoOp & \(\downarrow 1.21\) & \(\uparrow 34.22\) & \(\uparrow 23.04\) & \(\downarrow 1.98\) & \(\uparrow 28.8\) & \(\uparrow 17.7\) & \(\downarrow 11.93\) & \(\uparrow 15.23\) & \(\uparrow 4.38\) \\ \hline \hline  & \multicolumn{4}{c}{MNIST} & \multicolumn{4}{c}{EuroSAT} & \multicolumn{4}{c}{DTD} \\ \hline CLIP & \(31.74_{0,00}\) & \(15.43_{0,00}\) & \(20.77_{0,00}\) & \(22.33_{0,00}\) & \(48.3_{0,00}\) & \(30.54_{0,00}\) & \(42.5_{0,00}\) & \(44.40_{0,00}\) & \(43.45_{0,00}\) \\ CoOp & **94.68\({}_{6.64}\)** & \(15.43_{7.75}\) & \(21.15_{12.18}\) & \(82.91_{8.81}\) & \(46.02_{2.23}\) & \(58.64_{5,86}\) & **69.67\({}_{1.17}\)** & \(34.81_{3.44}\) & \(46.32_{2.92}\) \\ GRIP & **95.13\({}_{1.11}\)** & **60.63\({}_{0.44}\)** & **74.06\({}_{0.29}\)** & **91.75\({}_{0.53}\)** & **92.91\({}_{0.91}\)** & **92.33\({}_{0.70}\)** & **68.26\({}_{0.69}\)** & **62.61\({}_{1.87}\)** & **65.30\({}_{1.03}\)** \\ \hline \(\Delta\) CLIP & \(\uparrow 63.39\) & \(\uparrow 45.2\) & \(\uparrow 53.29\) & \(\uparrow 69.42\) & \(\uparrow 44.61\) & \(\uparrow 61.79\) & \(\uparrow 25.76\) & \(\uparrow 18.17\) & \(\uparrow 21.85\) \\ \(\Delta\) CoOp & \(\uparrow 0.45\) & \(\uparrow 45.2\) & \(\uparrow 52.91\) & \(\uparrow 8.84\) & \(\uparrow 46.89\) & \(\uparrow 33.69\) & \(\downarrow 1.41\) & \(\uparrow 27.8\) & \(\uparrow 19.00\) \\ \hline \hline \multicolumn{2}{l}{**Split 2**} & \multicolumn{4}{c}{Flowers102} & \multicolumn{4}{c}{RESICS45} & \multicolumn{4}{c}{FGVCAircraft} \\ \hline Method & S & U & H & S & U & H & S & U & H \\ \hline CLIP & \(65.38_{0.00}\) & \(60.64_{0.00}\) & \(62.92_{0.00}\) & \(59.5_{0.00}\) & \(47.06_{0.00}\) & \(52.55_{0.00}\) & \(17.30_{0.00}\) & \(18.12_{0.00}\) & \(17.70_{0.00}\) \\ CoOp & **91.8\({}_{1.32}\)** & \(47.75_{3.86}\) & \(62.77_{3.31}\) & **86.54\({}_{1.92}\)** & \(48.00_{3.01}\) & \(61.70_{2.17}\) & **33.59\({}_{1.12}\)** & \(19.57_{1.37}\) & **24.63\({}_{0.63}\)** \\ GRIP & \(88.84_{0.75}\) & **70.93\({}_{2.08}\)** & **78.86\({}_{1.26}\)** & \(84.47_{0.41}\) & **84.09\({}_{1.01}\)** & **84.28\({}_{0.73}\)** & \(22.13_{0.24}\)** & \(28.32_{0.33}\) & **24.84\({}_{0.05}\)** \\ \hline \(\Delta\) CLIP & \(\uparrow 23.46\) & \(\uparrow 10.29\) & \(\uparrow 15.94\) & \(\uparrow 27.83\) & \(\uparrow 25.45\) & \(\uparrow 26.61\) & \(\uparrow 4.83\) & \(\uparrow 10.20\) & \(\uparrow 7.14\) \\ \(\Delta\) CoOp & \(\downarrow 2.96\) & \(\uparrow 23.18\) & \(\uparrow 16.09\) & \(\downarrow 2.07\) & \(\uparrow 36.09\) & \(\uparrow 22.58\) & \(\downarrow 11.46\) & \(\uparrow 8.75\) & \(\uparrow 0.21\) \\ \hline \hline  & \multicolumn{4}{c}{MNIST} & \multicolumn{4}{c}{EuroSAT} & \multicolumn{4}{c}{DTD} \\ \hline CLIP & \(15.99_{0.00}\) & \(39.18_{0.00}\) & \(22.71_{0.00}\) & \(32.47_{0.00}\) & \(33.10_{0.00}\) & \(32.78_{0.00}\) & \(45.43_{0.00}\) & \(39.72_{0.00}\) & \(42.39_{0.00}\) \\ CoOp & **90.6\({}_{1.02}\)** & \(18.77_{9.12}\) & \(30.29_{1.28}\) & \(86.43_{2.23}\) & \(47.16_{11.17}\) & \(60.53_{8.42}\) & \(\mathbf{70.41}_{9.99}\) & \(32.53_{4.58}\) & \(44.42_{4.63}\) \\ GRIP & **95.71** & **97.50** & **96.59** & \(\mathbf{91.08}_{0.02}\) & \(\mathbf{92.02}_{0.98}\) & \(\mathbf{91.55}_{0.47}\) & \(66.69_{0.53}\) & \(\mathbf{56.19}_{1.18}\) & **60.99\({}_{0.69}\)** \\ \hline \(\Delta\) CLIP & \(\uparrow 85.12\) & \(\uparrow 50.76\) & \(\uparrow 79.32\) & \(\uparrow 58.61\) & \(\uparrow 58.92\) & \(\uparrow 58.77\) & \(\uparrow 21.26\) & \(\uparrow 16.47\) & \(\uparrow 18.6\) \\ \(\Delta\) CoOp & \(\uparrow 6.11\) & \(\uparrow 71.20\) & \(\uparrow 57.19\) & \(\uparrow 4.65\) & \(\uparrow 44.86\) & \(\uparrow 31.02\) & \(\downarrow 3.71\) & \(\uparrow 23.66\) & \(\uparrow 16.57\) \\ \hline \hline  & \multicolumn{4}{c}{MNIST} & \multicolumn{4}{c}{EuroSAT} & \multicolumn{4}{c}{DTD} \\ \hline CLIP & \(10.59_{0.00}\) & \(46.74_{0.00}\) & \(17.27_{0.00}\) & \(41.47_{0.00}\) & \(19.60_{0.00}\) & \(26.62_{0.00}\) & \(45.52_{0.00}\) & \(39.58_{0.00}\) & \(42.34_{0.00}\) \\ CoOp & \(89.6_{8.08}\) & \(26.31_{2.88}\) & \(39.4_{16.61}\) & \(79.39_{3.97}\) & \(

\begin{table}
\begin{tabular}{l l l} \hline \hline

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multicolumn{1}{c}{**Textual prompts**} & & & & & & & & & & \\ \hline \hline \multicolumn{1}{c}{} & \multicolumn{3}{c}{Flowers102} & & & & & & & & & \\ \hline Method & SSL & UL & TRZSL & SSL & UL & TRZSL & SSL & UL & TRZSL \\ \hline CLIP (ViT-B/32) & \(63.67\) & \(63.67\) & \(63.40\) & \(54.48\) & \(54.48\) & \(54.46\) & \(43.24\) & \(43.24\) & \(43.45\) \\ GRIP (ViT-B/32) & \(83.60\) & \(69.84\) & \(86.26\) & \(74.11\) & \(70.55\) & \(81.07\) & \(56.07\) & \(46.09\) & **65.30** \\ \hline CLIP (ViT-L/14) & \(73.98\) & \(73.98\) & \(73.05\) & \(62.67\) & \(62.67\) & \(62.13\) & \(52.45\) & \(52.45\) & \(51.61\) \\ GRIP (ViT-L/14) & **94.21** & **82.33** & **96.18** & **81.53** & **76.86** & **86.88** & **60.91** & **54.40** & \(64.92\) \\ \hline \multicolumn{1}{c}{**Visual prompts**} & & & & & & & & & \\ \hline CLIP (ViT-B/32) & \(63.67\) & \(63.67\) & \(63.40\) & \(54.48\) & \(54.48\) & \(54.46\) & \(43.24\) & \(43.24\) & \(43.45\) \\ GRIP (ViT-B/32) & \(67.95\) & \(63.09\) & \(77.18\) & \(71.22\) & \(68.43\) & \(82.19\) & \(54.57\) & \(50.51\) & **62.78** \\ \hline CLIP (ViT-L/14) & \(73.98\) & \(73.98\) & \(73.05\) & \(62.67\) & \(62.67\) & \(62.13\) & \(52.45\) & \(52.45\) & \(51.61\) \\ GRIP (ViT-L/14) & **78.68** & \(73.50\) & **85.85** & **77.53** & **76.00** & **86.63** & **55.72** & **54.27** & \(62.74\) \\ \hline \hline \end{tabular}
\end{table}
Table 10: Performance of CLIP and GRIP with different backbones on Flowers102, RESICS45, and DTD, for all the learning settings SSL, UL, TRZSL.

Response:
## Review

### Summary

This paper investigates the use of CLIP for pseudo-labeling across various tasks, including semi-supervised learning (SSL), transductive zero-shot learning (TZSL), and unsupervised learning (UL). The authors propose three training strategies—FPL, IFPL, and GRIP—differentiated by the static or dynamic nature of the pseudo-labels. The experimental results on six datasets demonstrate significant improvements in performance, highlighting the potential of the proposed methods to enhance image classification capabilities while addressing inherent biases in CLIP's pseudo-labels. The findings emphasize the importance of prompt tuning and effective pseudo-labeling techniques in leveraging CLIP for diverse learning scenarios.

### Strengths

- Well-written paper with a clear structure, making it easy to understand.
- Extensive exploration of a broad design space, including prompt modalities, learning paradigms, and training strategies.
- Demonstrates significant performance improvements across multiple datasets and tasks.
- Empirical study shows the effectiveness of self-training with CLIP-based models, including debiasing effects.
- Experimental results indicate that the proposed strategy holds considerable merit.

### Weaknesses

- Writing lacks clarity in certain details, including typos and inconsistencies in abbreviations.
- Absence of ablation studies to analyze the balance between labeled and unlabeled data.
- The proposed approach may be overly naive, lacking detailed analysis of hyper-parameter choices.
- Limited novelty as the use of CLIP's pseudo-labels is not new, and the proposed training strategies are commonly used.
- Need for further analysis of results to explain unexpected performance outcomes.

### Questions

- Does 'line 217' mean reinitializing the prompts at the beginning of each iteration?
- Is the process of pseudo-labeling done online or offline? If offline, is it time-consuming with large unlabeled data?
- Are abbreviations TZSL and TRZSL referring to the same concept?
- What are the reasons for slight decreases in performance observed in some experimental results?
- What happens if there are no samples for a class during the IFPL process?
- Why do results for larger image encoders show reduced performance?
- Could the authors clarify the differences between IFPL and GRIP?

### Soundness

**Score:** 3

**Description:** 3 = good; the methodology is technically sound, with clear results but some weaknesses in the approach.

### Presentation

**Score:** 2

**Description:** 2 = fair; while the paper is generally well-structured, it requires improvements in clarity and detail.

### Contribution

**Score:** 3

**Description:** 3 = good; the contributions are meaningful and impactful within the context of using CLIP for pseudo-labeling.

### Rating

**Score:** 6

**Description:** 6 = Weak Accept; the paper is technically solid with moderate-to-high impact but requires some clarifications and improvements.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper presents a solid exploration of using CLIP for pseudo-labeling, with significant experimental results. While there are some weaknesses in clarity and the novelty of the approach, the contributions to the field and the positive reception from reviewers justify acceptance. The authors should address the highlighted issues in their revisions.