weakness (insight and contribution), my initial rating is borderline. Strengths:
+ The problem of adapting CLIP under few-shot setting is recent. Compared to the baseline method CoOp, the improvement of the proposed method is significant.
+ The ablation studies and analysis in Section 4.4 is well organized and clearly written. It is easy to follow the analysis and figure our the contribution of each component. Also, Figure 2 is well designed and clear to illustrate the pipeline.
+ The experimental analysis is comprehensive. The analysis on computation time and inference speed is also provided. Weakness:
- (major concern) The contribution is somehow limited. The main contribution is applying optimal transport for few-shot adaptation of CLIP. After reading the paper, it is not clear enough to me why Optimal Transport is better than other distance. Especially, the insight behind the application of Optimal Transport is not clear. I would like to see more analysis and explanation on why Optimal Transport works well. Otherwise, it seems that this work is just an application work on a specific model and a specific task, which limits the contribution.
- The recent related work CoCoOp [1] is not compared in the experiments. Although it is a CVPR'22 work that is officially published after the NeurIPS deadline, as the extended version of CoOp, it is necessary to compare with CoCoOp in the experiments.
- In the approach method, there lacks a separate part or subsection to introduce the inference strategy, i.e., how to use the multiple prompts in the test stage.
- Table 2 mixed different ablation studies (number of prompts, visual feature map, constraint). It would be great if the table can be split into several tables according to the analyzed component.
- The visualization in Figure 4 is not clear. It is not easy to see the attention as it is transparent. References
[1] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
After reading the authors' response and the revised version, my concerns (especially the contribution of introducing the optimal transport distance for fine-tuning vision-language models) are well addressed and I am happy to increase my rating.
- The recent related work CoCoOp [1] is not compared in the experiments. Although it is a CVPR'22 work that is officially published after the NeurIPS deadline, as the extended version of CoOp, it is necessary to compare with CoCoOp in the experiments.