The motivation for all V&L experiments looks weak because it stems from the following single observation “if cast VQA 2.0 into a zero-shot image-to-text retrieval task, we only observe chance performance. Thus, we propose to integrate CLIP’s visual encoder with previous V&L models.” The prompt engineering considered for VQA is not persuasive and does not imply the same straightforward introduction of CLIP-based backbones to VLN / Image Captioning tasks. Also, I think it is necessary to explain the “chance performance” for VQA, at least for the binary yes/no question type. E.g., does 0.037 mean that one can simply flip the model’s answers?
“Unfreezing” the Visual Backbone helps if done correctly – it is a well-known fact supported by BUTD-Res101 “unfreezing” experiment in the paper. However, CLIP-Res50 benefits more from pre-training than BUTD-Res101. This observation requires further elaboration.
Most of the models considered in their respective tasks seem outdated as of 2021 (e.g., Pythia 2019, MCAN 2019). At the same time, the authors avoid direct comparison to the methods that require large-scale “supervised” or “self-supervised” pre-training, e.g., VLN-BERT, although CLIP itself can be considered as such.
The authors may want to include findings on localization ability of transformers from the following papers in the later versions of the manuscript: - Do Vision Transformers See Like Convolutional Neural Networks? arxiv 2021 - Dynamic Head: Unifying Object Detection Heads with Attentions, CVPR 2021 - Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions, ICCV 2021
- Do Vision Transformers See Like Convolutional Neural Networks? arxiv 2021 - Dynamic Head: Unifying Object Detection Heads with Attentions, CVPR 2021 - Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions, ICCV 2021