# Semi-Supervised Image Captioning with CLIP

Chuanyang Jin
New York University

###### Abstract

Image captioning, a fundamental task in vision-language understanding, seeks
to generate accurate natural language descriptions for provided images. The
CLIP model, with its rich semantic features learned from a large corpus of
image-text pairs, is well-suited for this task. In this paper, we present a
two-stage semi-supervised image captioning approach that exploits the
potential of CLIP encoding. Our model comprises a CLIP visual encoder, a
mapping network, and a language model for text generation. In the initial
stage, we train the model using a small labeled dataset by contrasting the
generated captions with the ground truth captions. In the subsequent stage, we
continue the training using unlabeled images, aiming to maximize the image-
caption similarity based on CLIP embeddings. Remarkably, despite utilizing
less than 2% of the COCO-captions, our approach delivers a performance
comparable to state-of-the-art models trained on the complete dataset.
Furthermore, the captions generated by our approach are more distinctive,
informative, and in line with human preference.

##  1 Introduction

A large polar bear swimming in a pool of water.

A man riding a wave on top of a surfboard.

A city skyline and a large body of water in the sunset.

A small brown squirrel standing on the grass.

Figure 1: Our approach generalizes well to a broad range of images and
consistently generates high-quality captions.

Image captioning aims to describe images with syntactically and semantically
meaningful sentences, thereby bridging the gap between vision and language.
Modern image captioning approaches, however, face two primary challenges:
their resource-intensive nature and reliance on the quality of reference
captions.

First, these models have a large number of trainable parameters, thereby
demanding considerable substantial training time and computational resources.
They also heavily rely on vast amounts of captioned images for training.
Sometimes, these models even require multiple reference captions and
additional annotations [32].

Second, most models are trained with reference-based similarity objectives,
hence the performance ceiling of these models is effectively dictated by the
quality of reference captions used during training. Our research shows that
current state-of-the-art models have reached this limit, producing captions of
similar, if not marginally better, quality than the reference captions.
Furthermore, models trained on text similarity objectives tend to overlook
specific details that set one image apart from others, as public datasets’
reference captions typically describe only the most conspicuous and common
objects [8].

Our research proposes a semi-supervised captioning approach to address these
issues simultaneously. We begin with supervised training on a minimal set of
captioned images, then leverage the CLIP objective for unsupervised learning,
drastically reducing the reliance on reference captions and minimizing the
training duration. When presented with new unlabeled data, we can routinely
update the model, enhancing its performance and applicability.

Figure 2: Overview of the model architecture of ClipCap.

##  2 Related Work

#### Image Captioning.

Large research efforts have been devoted to image captioning [32]. Typically,
an image captioning approach first encodes an image into a visual
representation, which is then decoded to produce the final captions. The
visual encoder extracts the representation from either classification networks
[4] [7] [12] [40], or object detection networks [2] [22] [43], with the latter
providing more expressive features. Some models introduce a self-attention
mechanism [14] or Vision Transformers [11] to make better use of visual cues.
When it comes to the textual decoder, different models use LSTM variants [6]
[36] or transformer-based architectures [14] [26]. The emergence of large
language models like BERT [10] and GPT [3][30][31] presents promising
alternatives.

Following the success of Vision-Language (VL) models, there has been
substantial progress in image captioning. This can be largely attributed to
their ability to harness abundant online image-caption pairs using Vision
Language Pre-training. It allows the creation of a shared latent space of both
vision and text for image captioning [22] [24] [33] [41]. We will delve
further into a CLIP-based method.

#### CLIP.

CLIP (Contrastive Language-Image Pre-training) [29] effectively learns visual
concepts from natural language supervision, by maximizing the similarity of
the matched image-text pairs and minimizing that of the unmatched pairs. By
training on a large dataset of image-text pairs, CLIP learns meaningful
representations from both modalities and achieves state-of-the-art performance
in downstream tasks, such as detection [13] [21], segmentation [20] [39], and
video understanding [25] [37]. Our research incorporates CLIP’s image encoding
for generating captions and employs the CLIP image-caption similarity for
unsupervised finetuning.

#### ClipCap.

ClipCap [27], a state-of-the-art image captioning model, leverages the power
of CLIP to interconnect visual and textual representations. First, it employs
CLIP’s visual encoder to generate an encoded image representation. Next, a
mapping network—either a transformer [34] or MLP—translates the CLIP embedding
to the GPT space, producing a prefix to the caption. Finally, the GPT auto-
regressively generates the predicted image captions.

|  |  |  |  |
---|---|---|---|---|---
Reference |  A puppy rests on the street next to a bicycle |  A traffic light and street sign surrounded by buildings |  A giraffe bending over while standing on green grass |  A person up in the air, upside down while outside |  A very cute cat near a bunch of birds
Supervised |  A puppy sleeping on the street ($S_{\text{CLIP}}=0.02$) |  A city in the fog ($S_{\text{CLIP}}=0.68$) |  A giraffe that is standing in the grass ($S_{\text{CLIP}}=0.04$) |  A man flying through the air while riding a snowboard ($S_{\text{CLIP}}=1$) |  A white cat standing next to a bunch of pigeons ($S_{\text{CLIP}}=0.98$)
Ours |  A white dog laying on the street next to a bike ($S_{\text{CLIP}}=0.15$) |  A black and white photo of a traffic light ($S_{\text{CLIP}}=0.99$) |  A giraffe eating grass in a grassy field ($S_{\text{CLIP}}=0.16$) |  A person on a snowboard jumping over a snow-covered slope ($S_{\text{CLIP}}=1$) |  A cat approaching a bunch of pigeons ($S_{\text{CLIP}}=0.97$)
Figure 3: Uncurated captions generated through both the supervised approach
and our semi-supervised approach for five images in the COCO-captions dataset.

##  3 Approach

###  3.1 Semi-Supervised Framework

Our approach to captioning images builds upon the model architecture of
ClipCap [27], which first extracts CLIP image embeddings from given images,
then transforms these embeddings and feeds them into a GPT model to generate
captions. To mitigate the model’s reliance on captioned images, we seek to
propose an unsupervised training approach that utilizes the CLIP image-caption
similarity as the training objective. Nonetheless, in the absence of initial
signals, our generated text is purely random and meaningless, which obstructs
further optimization.

To overcome this challenge, we adopt a two-stage Semi-Supervised Learning
(SSL) framework [44]. Initially, a supervisory signal is established using a
minimal dataset of 10,000 captioned images, constituting less than 2$\%$ of
the typical training division in the COCO-captions dataset [5] [23]. Training
over 10 epochs on this small dataset requires only a few minutes on a single
GPU. The model then undergoes unsupervised training to refine its performance.

The supervised stage trains the model by comparing the generated captions with
the provided reference captions. When given an image ${\boldsymbol{x}^{i}}$,
the CLIP model produces its visual embedding ${\boldsymbol{v}^{i}}$, which is
subsequently transformed by the mapping network into a series of prefix
embeddings $p_{1}^{i},\dots,p_{k}^{i}$. These are then combined with the
caption embeddings $c_{1}^{i},\dots,c_{l}^{i}$ and passed into the language
model. Training the model with a batch size of $N$, we utilize a next-token
prediction task with a cross-entropy loss:

$\mathcal{L}_{\text{supv}}=-\sum_{i=1}^{N}\sum_{j=1}^{l}\log
p_{\theta}(c_{j}^{i}|p_{1}^{i},\dots,p_{k}^{i},c_{1}^{i},\dots,c_{j-1}^{i})$

The subsequent stage involves unsupervised training of the model. We develop a
cohesive framework that integrates both stages, thereby enabling smooth
transitions between stages throughout the training cycles.

###  3.2 Unsupervised Stage with CLIP Objective

In the unsupervised stage, each training step involves two key steps. First,
our model generates captions for a given batch of images. Then, these
generated captions are used to optimize the model, with the similarity between
the images and their captions serving as the objective function. Typically,
this stage necessitates around 10 epochs of training, a process that can be
accomplished within a few hours on a single GPU.

Our loss function is designed to be the inverse of the image-caption
similarity determined by CLIP embeddings. Suppose we have a set of images
$\\{\boldsymbol{x}^{i}\\}_{i=1}^{N}$ with corresponding CLIP visual embeddings
$\\{\boldsymbol{v}^{i}\\}_{i=1}^{N}$, and a set of candidate captions with
corresponding CLIP textual embeddings $\\{\boldsymbol{c}^{i}\\}_{i=1}^{N}$, we
compute our loss function as:

$\mathcal{L}_{\text{unsupv}}=-\sum_{i=1}^{N}\cos(\boldsymbol{v}^{i},\boldsymbol{c}^{i})$

By seeking to maximize the cosine similarity as derived from the contrastive
CLIP model, our model is encouraged to generate more distinctive and
informative captions.

Our language model generates candidate captions by sequentially predicting
each token. For every token, the model outputs a probability distribution over
all possible tokens in the vocabulary. These probabilities then determine the
following token. The process is typically accomplished through either a greedy
approach or beam search, both of which frequently use the Softmax operation.
However, the discrete sampling operation following Softmax is non-
differentiable, thus interrupting the flow of gradients. To overcome this
challenge, we utilize the Gumbel-Softmax method [16], which provides a
reparameterization trick for discrete variables. This technique allows us to
transform the non-differentiable operation into a differentiable one, thereby
enabling our gradient-based optimization.

###  3.3 CLIP Score for Evaluation

Current prevailing methods for evaluating the quality of text generation, such
as image captioning, are primarily reference-based n-gram overlap techniques,
such as BLEU [28], METEOR [9], and CIDEr [35]. Some more advanced reference-
based metrics like SPICE [1] and TIGEr [17], seek to go beyond simple overlap
and incorporate more sophisticated models of similarity between reference and
candidate texts. The fundamental assumption in all these approaches is that
the reference captions possess high quality, making it challenging for the
model-generated captions to reach a similar level of quality. However, our
research demonstrates that the captions generated by recent models often
exceed the quality of reference captions, providing more accurate
descriptions. Consequently, we need metrics that no longer solely concentrate
on the proximity to the reference caption.

Recent works including BERTScore [42], ViLBERTScore [19], UMIC [18], and
CLIPScore [15], propose using relevance scores computed by multimodal models.
Pretrained on large datasets, these models acquire a rich semantic
understanding of the relationship between images and texts. Given that our
test set includes reference captions, we develop our own CLIP Score to ensure
a more stable evaluation and a comprehensive understanding of our method’s
performance relative to these reference captions. The CLIP Score is calculated
as follows:

$S_{\text{CLIP}}=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{M_{i}}\sum_{j=1}^{M_{i}}\mathbf{1}_{{g_{i}\geq
r_{ij}}}\right)$ (1)

In this equation, $g_{i}$ represents the CLIP score between the image $i$ and
the caption generated by the model. $r_{ij}$ represents the CLIP score between
the image $i$ and the reference caption $j$. The indicator function,
$\mathbf{1}{{g{i}\geq r_{ij}}}$, equals 1 if $g_{i}\geq r_{ij}$, and equals 0
otherwise. $M_{i}$ is the total number of reference captions for image $i$,
and $N$ represents the total number of images in the dataset.

In this context, a CLIP score of 0.5 or higher signifies that our model
generates captions that are, on average, of equal or better quality when
compared with reference captions. Our findings suggest that state-of-the-art
models exhibit the capability to generate captions that excel in quality,
surpassing the requirement of merely resembling reference captions. This
underscores the importance of exploring evaluation metrics that take into
account factors beyond mere textual similarity.

##  4 Experiments

###  4.1 Experimental Results

We carry out experiments on both our semi-supervised approach and the entirely
supervised approach proposed by Mokady et al [27]. Mokady et al’s method
leverages the full breadth of the COCO-captions dataset [23] for training the
model. In contrast, our semi-supervised approach utilizes merely 10,000
captioned images in the initial supervised training stage, accounting for less
than 2% of the overall dataset. Thereafter, our unsupervised stage
incorporates the remaining images from the dataset without any accompanying
captions.

Figure 4: BLEU@4 and CLIP scores during the unsupervised stage.

Figure 4 demonstrates the performance in terms of BLEU@4 and CLIP scores
during our unsupervised training stage. In the beginning, the performance of
our model lags due to the significantly smaller size of the training dataset
in the supervised stage. Despite this, significant performance improvement is
observed throughout the unsupervised training. Although there are some
fluctuations in the performance increase, the overall trend is consistently
upward, reaching a point of stabilization after 10 epochs. Remarkably, our
approach not only marginally surpasses the BLEU@4 score achieved by the
supervised method, but also significantly outperforms it in terms of the CLIP
score.

###  4.2 Impact of the Supervised Stage

Figure 5: Final BLEU@4 and CLIP scores with the variation in the number of
images used in the supervised stage.

Without the initial supervised training stage, the text produced by the model
during the unsupervised stage would be entirely random and incoherent. As a
result, the similarity between the image captions calculated during this stage
would be extremely low. This situation creates a significant challenge in
refining the model, given that the search space for potential inputs is
exceptionally vast and discrete. Under such conditions, the loss is prone to
fluctuating wildly and shows no signs of converging to an optimal value.

To tackle this problem, we introduce an initial supervised training stage
before the unsupervised training stage. This supervised stage equips the model
with an essential set of preliminary signals, thereby establishing a baseline
proficiency that can be further refined during the unsupervised training
stage. In Figure 5, we depict the relationship between the size of the labeled
training dataset used during the supervised stage and the model’s final
performance. Notably, even a modest dataset of 100 images for supervised
training is sufficient to guide the model effectively through the subsequent
unsupervised stage. As we increase the size of the supervised training set,
the model’s performance improves consistently. However, beyond an approximate
limit of 10,000 images, adding more training data does not lead to a
significant enhancement of the model’s performance.

|  |  |  |  |
---|---|---|---|---|---
Reference |  A man with a red helmet on a small moped on a dirt road |  A young girl inhales with the intent of blowing out a candle |  A man on a bicycle riding next to a train |  A wooden cutting board topped with sliced up food |  A kitchen is shown with a variety of items on the counters
Oscar [22] |  A man riding a motorcycle down a dirt road ($S_{\text{CLIP}}=0.22$) |  A woman sitting at a table with a plate of food ($S_{\text{CLIP}}=0$) |  A woman riding a bike down a street next to a train ($S_{\text{CLIP}}=0.02$) |  A woman sitting at a table with a plate of food ($S_{\text{CLIP}}=0.42$) |  A kitchen with a sink, dishwasher and a window ($S_{\text{CLIP}}=0.41$)
Ours |  A man riding a motorbike on a dirt mountain road ($S_{\text{CLIP}}=0.37$) |  A woman is blowing out a candle on a piece of cake ($S_{\text{CLIP}}=0.91$) |  A man is riding a bike next to a train ($S_{\text{CLIP}}=0.15$) |  A lot of wooden spoons on a wooden table ($S_{\text{CLIP}}=1$) |  A kitchen with a sink, a stove and a window ($S_{\text{CLIP}}=0.31$)
Figure 6: Uncurated captions generated through both Oscar and our approach for
five images in the COCO-captions dataset.

###  4.3 Comparative Analysis of Different Models

Figure 7: Comparative evaluation of six models: four variants of our model and
two state-of-the-art models. The sizes of the circles represent the model
sizes.

We experiment with four model configurations using our approach: two that
involve training the mapping network with a frozen language model, and two
that include the finetuning of the language model. The mapping network is
either architected as an MLP or a Transformer network. We refer to these as
MLP, Transformer, MLP + GPT finetuning, and Transformer + GPT finetuning.

Our default Transformer configuration delivers the highest BLEU@4 score and
excellent CLIP score among our four configurations while maintaining a
relatively small model size. The MLP, on the other hand, presents the least
compelling results, likely due to its limitation in expressiveness. Both the
MLP + GPT tuning and Transformer + GPT tuning yield decent results, but they
do not distinctly outperform the Transformer, despite involving more trainable
parameters and requiring more training time. This observation suggests that
finetuning the language model in conjunction with using a Transformer mapping
network could lead to an excess of expressive power.

In the subsequent section, we compare our models with other state-of-the-art
methods, specifically the Oscar [22] and the Vision-Language Pre-training
model (VLP) [43]. Our default model operates with fewer parameters and
requires substantially less training time compared to these alternatives. In
terms of performance, our approach yields slightly lower BLEU@4 scores,
closely mirroring the results of the purely supervised ClipCap approach.
However, our model outperforms these models by achieving remarkably higher
CLIP scores.

Though it might initially appear biased to evaluate our CLIP-based model using
the CLIP score, a closer inspection of specific examples reveals that a higher
CLIP score often corresponds to better alignment with human preferences. As
evidenced in examples, such as Figure 3 and Figure 6, reference-based methods
often use more commonplace words in their descriptions, leading to a better
match with the references.

In contrast, our approach generates more semantically rich and “human-like”
interpretations. While these may not necessarily echo the phrasing of the
reference captions and might contain occasional errors, they largely resonate
with human intuition and common sense. Moreover, unlike reference-based
approaches that primarily focus on describing the most prominent and
frequently encountered objects, our approach goes beyond that to include
descriptions of additional objects within the scene. Consequently, our model
creates captions that stand out for their distinctiveness, descriptiveness,
and compatibility with human aesthetics. This demonstrates a marked advantage
over models that prioritize similarity objectives.

##  5 Conclusion

We introduce a semi-supervised approach to image captioning that leverages
CLIP encoding and CLIP objectives. Our quick and resource-efficient method
demonstrates not only impressive performance when assessed via similarity-
based metrics but also excels by producing state-of-the-art results in CLIP-
based evaluations. Unlike existing methods, our approach significantly reduces
the heavy reliance on captioned images and generates captions that are more
distinctive, informative, and aligned with human preference.

## Acknowledgements

We thank Professor Saining Xie, Professor He He, Professor Tianmin Shu, Hao
Zhu, and Peiqi Liu for their valuable insights shared during our discussions.

## References

  * [1] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.  SPICE: semantic propositional image caption evaluation.  CoRR, abs/1607.08822, 2016.
  * [2] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and VQA.  CoRR, abs/1707.07998, 2017.
  * [3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  CoRR, abs/2005.14165, 2020.
  * [4] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, and Tat-Seng Chua.  SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning.  CoRR, abs/1611.05594, 2016.
  * [5] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO captions: Data collection and evaluation server.  CoRR, abs/1504.00325, 2015.
  * [6] Xinpeng Chen, Lin Ma, Wenhao Jiang, Jian Yao, and Wei Liu.  Regularizing rnns for caption generation by reconstructing the past with the present.  CoRR, abs/1803.11439, 2018.
  * [7] Xinlei Chen and C. Lawrence Zitnick.  Learning a recurrent visual representation for image caption generation.  CoRR, abs/1411.5654, 2014.
  * [8] Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal.  Fine-grained image captioning with clip reward, 2023.
  * [9] Michael J. Denkowski and Alon Lavie.  Meteor universal: Language specific translation evaluation for any target language.  In WMT@ACL, 2014.
  * [10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  CoRR, abs/1810.04805, 2018.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  CoRR, abs/2010.11929, 2020.
  * [12] Hao Fang, Saurabh Gupta, Forrest N. Iandola, Rupesh Kumar Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick, and Geoffrey Zweig.  From captions to visual concepts and back.  CoRR, abs/1411.4952, 2014.
  * [13] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Zero-shot detection via vision and language knowledge distillation.  CoRR, abs/2104.13921, 2021.
  * [14] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares.  Image captioning: Transforming objects into words.  CoRR, abs/1906.05963, 2019.
  * [15] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  CoRR, abs/2104.08718, 2021.
  * [16] Eric Jang, Shixiang Shane Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  ArXiv, abs/1611.01144, 2016.
  * [17] Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, and Jianfeng Gao.  Tiger: Text-to-image grounding for image caption evaluation.  CoRR, abs/1909.02050, 2019.
  * [18] Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Trung Bui, and Kyomin Jung.  UMIC: an unreferenced metric for image captioning via contrastive learning.  CoRR, abs/2106.14019, 2021.
  * [19] Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung.  Vilbertscore: Evaluating image caption using vision-and-language bert.  Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, 2020.
  * [20] Boyi Li, Kilian Q. Weinberger, Serge J. Belongie, Vladlen Koltun, and René Ranftl.  Language-driven semantic segmentation.  CoRR, abs/2201.03546, 2022.
  * [21] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao.  Grounded language-image pre-training.  CoRR, abs/2112.03857, 2021.
  * [22] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  CoRR, abs/2004.06165, 2020.
  * [23] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  CoRR, abs/1405.0312, 2014.
  * [24] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  CoRR, abs/1908.02265, 2019.
  * [25] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  Clip4clip: An empirical study of CLIP for end to end video clip retrieval.  CoRR, abs/2104.08860, 2021.
  * [26] Yunpeng Luo, Jiayi Ji, Xiaoshuai Sun, Liujuan Cao, Yongjian Wu, Feiyue Huang, Chia-Wen Lin, and Rongrong Ji.  Dual-level collaborative transformer for image captioning.  CoRR, abs/2101.06462, 2021.
  * [27] Ron Mokady, Amir Hertz, and Amit H. Bermano.  Clipcap: Clip prefix for image captioning, 2021.
  * [28] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Annual Meeting of the Association for Computational Linguistics, 2002.
  * [29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  CoRR, abs/2103.00020, 2021.
  * [30] Alec Radford and Karthik Narasimhan.  Improving language understanding by generative pre-training.  2018\.
  * [31] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [32] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara.  From show to tell: A survey on image captioning.  CoRR, abs/2107.06912, 2021.
  * [33] Hao Tan and Mohit Bansal.  LXMERT: learning cross-modality encoder representations from transformers.  CoRR, abs/1908.07490, 2019.
  * [34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  CoRR, abs/1706.03762, 2017.
  * [35] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  CoRR, abs/1411.5726, 2014.
  * [36] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.  Show and tell: A neural image caption generator.  CoRR, abs/1411.4555, 2014.
  * [37] Mengmeng Wang, Jiazheng Xing, and Yong Liu.  Actionclip: A new paradigm for video action recognition.  CoRR, abs/2109.08472, 2021.
  * [38] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-art natural language processing.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, Oct. 2020. Association for Computational Linguistics.
  * [39] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.  Groupvit: Semantic segmentation emerges from text supervision.  arXiv preprint arXiv:2202.11094, 2022.
  * [40] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  CoRR, abs/1502.03044, 2015.
  * [41] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Making visual representations matter in vision-language models.  CoRR, abs/2101.00529, 2021.
  * [42] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with BERT.  CoRR, abs/1904.09675, 2019.
  * [43] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao.  Unified vision-language pre-training for image captioning and VQA.  CoRR, abs/1909.11059, 2019.
  * [44] Xiaojin Zhu and Andrew B. Goldberg.  Introduction to semi-supervised learning.  In Introduction to Semi-Supervised Learning, 2009.

##  Appendix A Comparison with Similar Approaches

Lee Wolf et al. [8] introduces an image captioning approach that shares
certain similarities with ours, particularly the use of a CLIP-based
objective. However, there are several crucial distinctions between their
approach and ours. First, while their method maintains a strong reliance on
captioned images, our approach significantly reduces the dependence on such
data. Second, their approach focuses on the unsupervised finetuning of a well-
trained model, whereas our strategy incorporates both supervised and
unsupervised training within a unified, inter-switchable framework. Lastly,
although we have experimented with their REINFORCE method using a self-
critical baseline for optimization, we ultimately select our Gumbel-Softmax
method due to its superior performance.

##  Appendix B Implementation Details

#### CLIP.

The pre-trained CLIP model we use is the ViT-B/32 version. The vision encoder
is a Vision Transformer with 12 layers, a width of 768, and 12 attention
heads. The text is similarly represented by a transformer with 12 layers, a
width of 512, and 8 attention heads. Both encoding networks produce a feature
vector of length 512.

#### GPT.

We employ a pre-trained GPT-2 as our language model to autoregressively
generate captions, following the implementation of Wolf et al. [38].

#### Mapping Networks.

The MLP mapping network contains a single hidden layer, while the Transformer
mapping network utilizes 8 layers of multi-head self-attention with 8 heads
each. These networks convert a 512-length feature vector into a language model
input prefix of length 10.