# Adaptively Clustering Neighbor Elements for Image Captioning

Zihua Wang1,2 Xu Yang1 Haiyang Xu2111Corresponding authors. Hanwang Zhang3
Chenliang Li2
Songfang Huang2 Fei Huang2 Yu Zhang1111Corresponding authors.
1 School of Computer Science & Engineering, Key Lab of Computer Network
& Information Integration (Ministry of Education), Southeast Univ., Nanjing,
China
2Alibaba Group
3 School of Computer Science & Engineering, Nanyang Technological Univ.,
Singapore.
{zihua, 101013120, zhang_yu}@seu.edu.cn,{shuofeng.xhy, lcl193798,
songfang.hsf<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We design a novel global-local Transformer named Ada-ClustFormer (ACF) to
generate captions. We use this name since each layer of ACF can adaptively
cluster input elements to carry self-attention (Self-ATT) for learning local
context. Compared with other global-local Transformers which carry Self-ATT in
fixed-size windows, ACF can capture varying graininess, _e.g_., an object may
cover different numbers of grids or a phrase may contain diverse numbers of
words. To build ACF, we insert a probabilistic matrix $\bm{C}$ into the Self-
ATT layer. For an input sequence $\\{\bm{s}_{1},...,\bm{s}_{N}\\}$,
$\bm{C}_{i,j}$ softly determines whether the sub-sequence
$\\{\bm{s}_{i},...,\bm{s}_{j}\\}$ should be clustered for carrying Self-ATT.
For implementation, $\bm{C}_{i,j}$ is calculated from the contexts of
$\\{\bm{s}_{i},...,\bm{s}_{j}\\}$, thus ACF can exploit the input itself to
decide which local contexts should be learned. By using ACF to build the
vision encoder and language decoder, the captioning model can automatically
discover the hidden structures in both vision and language, which encourages
the model to learn a unified structural space for transferring more structural
commonalities. The experiment results demonstrate the effectiveness of ACF
that we achieve CIDEr of 137.8, which outperforms most SOTA captioning models
and achieve comparable scores compared with some BERT-based models. The code
will be available in the supplementary material.

##  1 Introduction

Image Captioning (IC) aims to learn a shared vision-language representation
space for facilitating the transfer of multimodal knowledge to generate
visually grounded sentence [22]. Two prevailing deep learning techniques help
the IC model learn such space. The first one is the vision encoder-language
decoder pipeline [41] which back-propagates the language semantic to the
visual encoder and another one is the attention mechanism [46] which directly
bridges between vision and language domains for transferring multimodal
knowledge. Transformers [39], which build the encoder and decoder based on
dense attention operations, have both of the above-mentioned advantages.
Transformers have two types of attention operations which are self-attention
(Self-ATT) and cross-modal attention (Cross-ATT). From the perspective of
structure learning, Self-ATT applies the fully connected (FC) graph prior to
the data sequence. By using Self-ATT in both encoder and decoder, the graph
structures of both vision and language data can be discovered and Cross-ATT
helps transfer these structural commonalities for narrowing the modality gaps.
Therefore, Transformer prevails in IC tasks [12, 13, 28, 10].

Figure 1: (a) Transformer with fixed-size windows (size = $2$); (b) ACF which
adjusts the window size according to the input. (c) ACF-based IC. The
left/right part shows how the vision/language ACFs cluster image
grids/language words for transferring structural commonalities.

Interestingly, structure learning is one of the most significant research
directions of IC since the paired vision-language data usually share a unified
internal semantic structure although they have diverse external appearances.
Thus, if this unified semantic structure is captured, more structural
commonalities can be transferred for generating better captions. Motivated by
this, various IC models are proposed to exploit scene graphs [5, 49, 21] or
hierarchy trees [43, 51] to narrow the domain gap. However, such structures
need additional well-trained parsers. Moreover, vision and language parsers
usually have domain gaps that the parsed structures of the paired image-
sentence may not match, which may even weaken the effectiveness of these IC
models. We prefer an IC model that can adaptively discover the unified
semantic structures to remove the costs of the additional structure
annotations and more importantly, to learn a unified structure space for
transferring structural commonalities.

Transformer seems to be a good starting point since it can implicitly build
graphs by Self-ATT. However, it exploits the FC graph prior, while the useful
semantic structure is usually sparse and hierarchical like the scene graphs or
trees. To discover more sparse structures, researchers design various global-
local Transformers [33, 20, 29]. As sketched in Figure 1(a), these
Transformers gradually merge the neighbor elements in fixed-size windows into
bigger clusters and carry Self-ATT in each cluster. For example, the 1-st
layer clusters 2 neighboring elements like $\\{\bm{s}_{1},\bm{s}_{2}\\}$ to
carry Self-ATT for local contexts and the 2-nd layer merges
$\\{\bm{s}_{1},\bm{s}_{2}\\}$ and $\\{\bm{s}_{3},\bm{s}_{4}\\}$ into a bigger
one to learn more global context. Then a hierarchical structure is built from
lower to higher layers where local and global contexts are respectively
captured. However, these Transformers still do not satisfy our requirement
since vision and language data have diverse graininess, _e.g_., objects may
cover varying grids and phrases may compose different numbers of words, while
fixed-size windows cannot effectively capture such varying graininess.

To capture the varying graininess, we propose to Adaptively Cluster the
neighbor elements to carry Self-ATT and named the novel Transformer as Ada-
ClustFormer (ACF). As shown in Figure 1(b), in each layer, the window size is
not fixed but can be adjusted to each specific input sequence, _e.g_., in the
1-st layer, $\\{\bm{s}_{1},\bm{s}_{2},\bm{s}_{3}\\}$, $\\{\bm{s}_{4}$},
$\\{\bm{s}_{5},\bm{s}_{6}$}, $\\{\bm{s}_{7}$}, $\\{\bm{s}_{8}\\}$ are
respectively clustered. The higher layers merge small clusters into bigger
ones for global contexts, _e.g_., the 2-nd layer respectively merges
$\\{\bm{s}_{1},\bm{s}_{2},\bm{s}_{3},\bm{s}_{4},\bm{s}_{5},\bm{s}_{6}\\}$,
$\\{\bm{s}_{7},\bm{s}_{8}\\}$ into two clusters to carry Self-ATT. To achieve
this adaptive clustering, we insert a probabilistic clustering matrix $\bm{C}$
into the Self-ATT layer, where the probability $\bm{C}_{ij}$ softly determines
whether the sub-sequence $\\{\bm{s}_{i},...,\bm{s}_{j}\\}$ should be clustered
or not. To calculate $\bm{C}_{ij}$, we consider whether the next element
$\bm{s}_{j}$ is similar to the mean-pooling of
$\\{\bm{s}_{i},...,\bm{s}_{j-1}\\}$. Thus ACF can adjust the window of Self-
ATT based on each specific data sample.

To construct an IC model based on ACF, besides building 1-D ACF for the
language decoder, we also extend it to the 2-D ACF as the vision encoder. In
this way, both the visual encoder and language decoder can automatically
discover the hidden structures of the image and language data. This means that
the ACF model does not need any additional structure annotations as some
previous IC models [5, 2] but still exploits the sparse structures implied in
both vision and language data. For example, as shown in Figure 1(c), a visual
ACF can merge the smaller grids into bigger regions to capture both grid-level
[15] and region-level [4] contexts. And the language one gradually clusters
the single words into phrases to generate the captions in an imaginary phrase-
by-phrase manner [38, 48]. More importantly, compared with certain global-
local Transformers which are exclusively developed in vision and language
domains [24, 47], the visual and language ACF exploit the same way to discover
hidden structures. So, our ACF model is a homogeneous structure that helps
transfer more structural commonalities between vision and language domains,
_e.g_., as shown in Figure 1(c), the patches of the object “snow board” is
clustered in the image and correspondingly, the phrase “a snow board” is also
clustered in the language domain.

In summary, our contributions can be listed as follows:

  * •

We propose ACF that can adaptively capture varying graininess.

  * •

We extend ACF to the 2-D case for building a homogeneous IC model that learns
unified structural space for transferring more structural commonalities.

  * •

The experimental results show that our ACF model outperforms the classic
Transformers in IC.

##  2 Related Work

Image Captioning (IC). IC aims to generate descriptions according to the given
images. Typically, an encoder-decoder paradigm is used to convert visual
inputs to sequence outputs. In the early stage, image features are extracted
by CNN-based encoders, as the input of the RNN-based decoders [4, 35, 41, 16].
For example, Up-Down [4] employs a Faster R-CNN [34] to extract image region
features and LSTM networks to generate sentences.

Nowadays, Transformer-based models have shown their might in Neural Language
Process (NLP) and replace RNN-based decoders in IC [19, 12, 14]. Subsequently,
more advanced Transformer-based decoders are proposed, _e.g_.,
$\mathcal{M}^{2}$ Transformer [8] proposes a meshed-memory Transformer to
interact with the low-level and high-level features; X-Linear Transformer [31]
selectively capitalizes the visual information from image regions by bilinear
pooling.

However, these models still use CNN-based feature extractors. More recently,
witnessing the boom of Vision Transformers (ViT) [9, 24], researchers use ViT-
based visual encoders for captioning. For instance, CPTR [23] introduces grid-
based features that are extracted by ViT [9] instead of using the ROI-based
features; DLCT [25] fuses the ROI-based features with the grid-based features
to overcome the shortcoming of both features. Besides that, some models
exploit the knowledge distilled from Vision-Language BERTs for better captions
[18]. VinVL [52] and GRIT [28] propose the object detection model in IC.
ClipCAP [27] and LEMON [13] introduce large-scale pretraining on IC.
Noteworthy, the methods above employ the ViT [9] or Swin Transformer [24] as
their backbone. Thus, our ACF adopts the Swin Transformer as our encoder
backbone.

Among the previous IC models, Auto-Parsing Network (APN) [48] has a similar
motivation as ours, which also inserts a clustering matrix into the Self-ATT
layer. However, Ada-ClustFormer (ACF) calculates this matrix differently. APN
only considers whether pairwise neighboring elements should be clustered or
not, while we calculate this probability from a more global scope.
Specifically, we consider whether the next element is similar to the previous
clustered elements. More importantly, we extend our ACF into the 2-D case,
which can adaptively cluster the visual patches into regions, while APN only
treats a sequence of ROI features as the visual input and still applies a 1-D
clustering matrix to address it. More comparisons will be given in the
supplementary material.

Global-Local Transformer. To alleviate the fully connected graph prior in
Transformer, researchers propose various global-local Transformers to learn
sparse structures of the language [26, 6]. For example, Global-local [26]
introduces a fixed-size of the global and local attention model in neural
machine translation. Longformer [6] proposes global and local window
attentions, which can provide inductive bias and long sequence representation,
respectively. Hi-Transformer [44] learns sentence-level and document-level
semantics through the hierarchical structure.

The global-local Transformer mechanism is also effective in vision area [7,
53, 25]. Pairwise and patchwise self-attention are proposed in image
recognition [53]. Furthermore, GLiT [7] proposes to adaptively trade off the
global and local information of the images. DLCT [25] explores the global and
local information by the combination of grid-based features and ROI-based
features.

However, these models are exclusively developed in a single domain (either NLP
or CV), while our ACF provides a general approach in both the vision and
language domains. Thus, using ACF to build the IC model encourages learning a
unified structure space for transferring more structure commonalities.

##  3 Ada-ClustFormer IC model

Compared with the classic Transformer, Ada-ClustFormer (ACF) inserts an
adaptively clustering matrix $\bm{C}$ into each self-attention (Self-ATT)
layer to adaptively control the scope of Self-ATT. The calculation of $\bm{C}$
is detailed in Section 3.1 where we first show the 1-D language case and then
extend it to the 2-D vision case. By stacking these revised Self-ATT layers,
ACF can be built for constructing the vision encoder and language decoder for
captioning (cf. Section 3.2).

###  3.1 Ada-ClustFormer

Multi-Head Attention (MHA). ACF is built based on Transformer, whose most
elemental building block is the Multi-Head Attention (MHA). Given the query
$\mathcal{Q}\in\mathbb{R}^{N_{Q}\times d}$, key
$\mathcal{K}\in\mathbb{R}^{N_{K}\times d}$, and value
$\mathcal{V}\in\mathbb{R}^{N_{V}\times d}$, MHA calculates the output
$\mathcal{Z}=\textbf{MHA}(\mathcal{Q},\mathcal{K},\mathcal{V})$ as:

Input: $\displaystyle\mathcal{Q},\mathcal{K},\mathcal{V}$ (1) ATT:
$\displaystyle\mathcal{A}_{l}=\text{Softmax}(\frac{\mathcal{Q}\mathcal{W}_{l}^{Q}(\mathcal{K}\mathcal{W}_{l}^{K})^{T}}{\sqrt{d}})$
$\displaystyle\textbf{Head}:$
$\displaystyle\mathcal{H}_{l}=\mathcal{A}_{l}\mathcal{V}\mathcal{W}_{l}^{V},$
Multi-Head:
$\displaystyle\mathcal{H}=[\mathcal{H}_{1},\mathcal{H}_{2},...,\mathcal{H}_{h}]\mathcal{W}^{H},$
Output: $\displaystyle\mathcal{Z}=\text{LN}(\mathcal{H}+\mathcal{Q}),$

where
$\mathcal{W}_{l}^{Q},\mathcal{W}_{l}^{K},\mathcal{W}_{l}^{V}\in\mathbb{R}^{d\times
d_{h}}$, $\mathcal{W}_{l}^{H}\in\mathbb{R}^{d\times d}$ are all learnable
parameters; $h$ denotes the head number and $d_{h}=d/h$; $\mathcal{A}_{l}$ is
the $l$-th attention matrix corresponding to the $l$-th head
$\mathcal{H}_{l}$; $[\cdot]$ is the concatenation operation; and LN denotes to
the Layer Normalization.

Given an input sequence $\bm{S}=\\{\bm{s}_{1},...,\bm{s}_{N}\\}$, if
$\mathcal{Q}=\mathcal{K}=\mathcal{V}=\bm{S}$, Eq. (1) is also called self-
attention (Self-ATT). Self-ATT captures the global contexts between any two
elements $\bm{s}_{i}$ and $\bm{s}_{j}$ by calculating the pairwise attention
weight in the “ATT” operation. From the perspective of structure learning [5],
single-head Self-ATT constructs a fully-connected (FC) graph where the nodes
are the elements of $\bm{S}$ and the pairwise edges are weighted by the
pairwise attention weight. Correspondingly, a $h$-head Self-ATT constructs $h$
FC graphs with different edge weights.

Adaptive Clustering Matrix $\bm{C}$. To sparsify this FC-graph, researchers
[9, 24] propose to carry Self-ATT in fixed-size windows, which is achieved by
revising “Head” in Eq. (1):

$\displaystyle\bm{C}\textbf{-based Head}:$
$\displaystyle\mathcal{H}=\text{Softmax}(\mathcal{A}\otimes\bm{C})\mathcal{V}\mathcal{W}^{V},$
(2)

where “$\otimes$” denotes the element-wise production; $\bm{C}$ is a $N\times
N$ binary clustering matrix that only the elements in the window can attend to
each other, _i.e_., if the window size is $w$, $\bm{C}_{i,j}=1$ if $|i-j|\leq
w$ and $\bm{C}_{i,j}=0$ if $|i-j|>w$. However, language or vision data usually
have diverse graininess, _e.g_., a phrase may contain different numbers of
words or an object may cover diverse spatial regions, while the fixed-size
windows can not capture the varying graininess.

To amend this, we revise the binary $\bm{C}$ to a probabilistic one where
$\bm{C}_{i,j}$ softly determines whether to cluster the embeddings from
$\bm{s}_{i}$ to $\bm{s}_{j}$ for carrying Self-ATT. Then if $\bm{C}_{i,j}$ is
small, the pairwise attention in $\mathcal{A}$ between $\bm{s}_{i}$ and
$\bm{s}_{j}$ is weakened in Eq. (2), which means $\bm{s}_{i}$ and $\bm{s}_{j}$
are less likely to stay in the same cluster. To adaptively decide the window
size according to each specific input for capturing the varying graininess, we
use the input itself to calculate $\bm{C}_{i,j}$:

$\bm{C}_{i,j}=P(\bm{s}_{i},...,\bm{s}_{j})=\prod_{k=i}^{j}P(\bm{s}_{k}|\bm{s}_{i},...,\bm{s}_{k-1}),$
(3)

where the joint distribution is decomposed to the productions of conditional
distributions $P(\bm{s}_{k}|\bm{s}_{i},...,\bm{s}_{k-1})$, which softly
decides whether to merge a new element $\bm{s}_{k}$ into the sub-sequence
$\\{\bm{s}_{i},...,\bm{s}_{k-1}\\}$. In the implementation,
$P(\bm{s}_{k}|\bm{s}_{i},...,\bm{s}_{k-1})$ is calculated as:

$P(\bm{s}_{k}|\bm{s}_{i},...,\bm{s}_{k-1})=\text{Sigmoid}(\text{FC}([\bm{s}_{k},\bm{s}_{i:k-1}])),$
(4)

where $\bm{s}_{i:k-1}$ is the mean pooling of the embeddings from $\bm{s}_{i}$
to $\bm{s}_{k-1}$. Intuitively, Eq. (4) exploits the context of the whole sub-
sequence $\\{\bm{s}_{i},...,\bm{s}_{k-1}\\}$ to decide whether to merge a new
element $\\{\bm{s}_{k}\\}$ into this sub-sequence. Note that Eq. (3) and Eq.
(4) only make sense when $i<k$. Since clustering the embeddings from
$\bm{s}_{i}$ to $\bm{s}_{k}$ equals to clustering from $\bm{s}_{k}$ to
$\bm{s}_{i}$, we set $\bm{C}_{i,k}=\bm{C}_{k,i}$ if $i>k$ and since a single
element $\bm{s}_{i}$ is itself a cluster, we set $\bm{C}_{i,i}=1$.

From Eq. (3), we can also find that:

$\displaystyle\bm{C}_{i,j}=$ $\displaystyle
P(\bm{s}_{j}|\bm{s}_{i},...,\bm{s}_{j-1})\times
P(\bm{s}_{i},...,\bm{s}_{j-1})$ (5) $\displaystyle=$ $\displaystyle
P(\bm{s}_{j}|\bm{s}_{i},...,\bm{s}_{j-1})\times\bm{C}_{i,j-1}.$

Since $P(\bm{s}_{j}|\bm{s}_{i},...,\bm{s}_{j-1})\leq 1$, we have
$\bm{C}_{i,j}\leq\bm{C}_{i,j-1}$, which means that two elements in the shorter
distance are more likely to be clustered for carrying Self-ATT. In this way,
local contexts are encouraged to be captured, as is shown in Figure 2(a).

Figure 2: (a) shows how to calculate $C_{1,4}$, where the shade denotes the
probability value, the darker the color, the larger the probability value. (b)
shows that the clustered elements in the lower layer will be further clustered
in a higher layer, _e.g_., the color of
$\\{\bm{s}_{1},\bm{s}_{2},\bm{s}_{3}\\}$ in the 2-nd layer is darker than the
1-st layer.

Stacking Revised Self-ATT. To learn global contexts, we can stack these
revised Self-ATT layers. When stacking, we hope that the higher layers will
carry Self-ATT in bigger windows than the lower layers to capture the global
contexts [43, 48]. To achieve this, for the $m$-th layer, we re-calculate
$\bm{C}^{(m)}$ as $\tilde{\bm{C}}^{(m)}$:

$\tilde{\bm{C}}^{(m)}=(1-\bm{C}^{(m)})\tilde{\bm{C}}^{(m-1)}+\bm{C}^{(m)}.$
(6)

Then $\tilde{\bm{C}}^{(m)}$ is used in Eq. (2) when $m>1$ and
$\tilde{\bm{C}}^{(1)}={\bm{C}}^{(1)}$. Since $0\leq\bm{C}^{(m)}_{i,j}\leq 1$,
$\tilde{\bm{C}}^{(m)}_{i,j}$ is a convex combination of
$\tilde{\bm{C}}^{(m-1)}_{i,j}$ and 1, which means that
$\tilde{\bm{C}}^{(m-1)}_{i,j}\leq\tilde{\bm{C}}^{(m)}_{i,j}\leq 1$. If
$\tilde{\bm{C}}^{(m-1)}_{i,j}$ is large, _i.e_., the sub-sequence
$\\{\bm{s}_{i},...,\bm{s}_{j}\\}$ should be clustered in the ${(m-1)}$-th
layer, then $\tilde{\bm{C}}^{(m)}_{i,j}$ must be larger, _i.e_.,
$\\{\bm{s}_{i},...,\bm{s}_{j}\\}$ is also clustered in the ${m}$-th layer. For
example, Figure 2(b) shows that the 2-nd layer will further cluster
$\\{\bm{s}_{1},\bm{s}_{2},\bm{s}_{3}\\}$ since
$\tilde{\bm{C}}^{(1)}_{1,3}\leq\tilde{\bm{C}}^{(2)}_{1,3}$. Thus, the higher
layers will carry Self-ATT in a bigger window than the lower layers to learn
more global contexts.

Figure 3: (a) The example of 2-D $\bm{C}$, where $C_{1,4;1,3}$ is used as the
example, which is decomposed into vertical and horizontal directions
probabilities. (b) Overview of the Down-Up Sampling Strategy.

2-D Clustering Matrix. Eq. (3) shows how to calculate $\bm{C}$ when the input
is a 1-D language sequence, next we extend it to the 2-D vision surface. Given
a 2-D feature map $\bm{V}=\\{\bm{v}_{1,1},...,\bm{v}_{H,W}\\}$, we use
$\bm{C}_{i,j;x,y}$ to denote the probability that softly decides whether a
sub-region $\\{\bm{v}_{i,x},...,\bm{v}_{j,y}\\}$ should be clustered or not,
which is:

$\displaystyle\bm{C}_{i,j;x,y}=P(\bm{v}_{i;x},...,\bm{v}_{j;y})$ (7)
$\displaystyle=$
$\displaystyle\prod_{k=i}^{j}\prod_{u=x}^{y}P(\bm{v}_{k;u}|\bm{v}_{i;x},\bm{v}_{i+1;x},...,\bm{v}_{k-1;u-1})$

where $i,j$ and $x,y$ respectively denote the horizontal and vertical
dimensions. To cover all the sub-regions in a $H\times W$ map, it requires
applying $O(H^{2}\times W^{2})$ times for Eq. (4) to get all the
probabilities. To reduce the computation burden, we apply the independence
assumption to decompose the 2-D distribution into two independent ones, which
respectively correspond to the horizontal and vertical dimensions:

$\displaystyle
P(\bm{v}_{i;x},...,\bm{v}_{j;y})=P_{h}(\bm{v}_{i;x},...\bm{v}_{j;x})P_{v}(\bm{v}_{i;x},...,\bm{v}_{i;y})$
(8) $\displaystyle=$
$\displaystyle\prod_{k=i}^{j}P_{h}(\bm{v}_{k;x}|\bm{v}_{i;x},...,\bm{v}_{k-1;x})\prod_{u=x}^{y}P_{v}(\bm{v}_{i;x}|\bm{v}_{i;x},...,\bm{v}_{i;u-1}),$

In this way, we only need to apply $O(H^{2}+W^{2})$ times for Eq. (4) and once
matrix production. Noteworthy, as sketched in Figure 2, for the 2-D region
which spans the horizontal axis from $i$ to $j$ and the vertical axis from $x$
to $y$, we use the left-most vertical and top-most horizontal to calculate two
1-D distributions and then multiply them to get $\bm{C}_{i,j;x,y}$. As Figure
3(a) shows, to calculate $\bm{C}_{1,4;1,3}$, for the vertical distribution
$P_{v}$, the horizontal ordinate is fixed to $1$ and the vertical ordinate
changes. $P_{h}(\bm{v}_{k;1}|\bm{v}_{1;1},...,\bm{v}_{k-1;1})|_{k=1,2,3,4}$
and $P_{v}(\bm{v}_{1;u}|\bm{v}_{1;1},...,\bm{v}_{1;u-1})|_{u=1,2,3}$ are
calculated in the same way as Eq. (4). The above-mentioned symmetric
characteristic is also applied.

Down-Up Sampling Strategy. If the sequence (feature map) is too long (big), we
can apply the Down-Up Sampling Strategy to reduce the computation cost. We use
a 1-D language case as an example to show this strategy. For
$\bm{S}=\\{\bm{s}_{1},...,\bm{s}_{L}\\}$, we can downsample it to
$\bm{\bar{S}}=\\{\bm{\bar{s}}_{1},...,\bm{\bar{s}}_{L/2}\\}$ where
$\bm{\bar{s}}_{i}$ is the mean pooling of $\bm{s}_{2*i-1}$ and $\bm{s}_{2*i}$.
Then $\bm{\bar{S}}$ is used in Eq. (3) and Eq. (4) to get $\bm{\bar{C}}$. To
upsample $\bar{\bm{C}}$ to the original size, we set
$\bm{C}_{i,j}=\bm{\bar{C}}_{\lceil i/2\rceil,\lceil j/2\rceil}$. Figure 3(b)
shows one simple case where $L=4$.

Figure 4: Overview of our ACF-based encoder-decoder IC model. The “Add$\&$LN”
is the Add and Layer Normalization. $m_{e}$/$m_{d}$ represent the number of
the encoder/decoder layers, respectively.

###  3.2 Encoder-Decoder Architecture

As is shown in Figure 4, we apply the ACF to build the vision encoder and
language decoder. Compared to the classic Transformer, our ACF introduces
clustering-restrained attention head. Specifically, in encoder, we calculate a
2-D clustering matrix $\bm{C}$ (cf. Eq. (7)) to softly cluster the elements
for carrying Self-ATT. Similarly, in decoder, the attention head is revised
with the 1-D $\bm{C}$ (cf. Eq. (5)). The output of this encoder-decoder is
used to calculate the word distributions $\bm{Z}$.

To train our IC model, we optimize the model by minimizing the cross-entropy
loss and maximizing the Reinforcement learning (RL) [35] reward. First, we
train the model by minimizing the cross-entropy loss:

$L_{CE}=-\log P(\bm{Z}^{*}),$ (9)

where $\bm{Z}^{*}$ is the ground-truth captions. Then, we further train the
model by minimizing the negative reward:

$L_{rl}=-\mathbb{E}_{\bm{Z}^{s}\sim
P(\bm{Z})}(\mathbb{S}(\bm{Z}^{*},\bm{Z}^{s})),$ (10)

where $\bm{Z}^{s}$ is sampled from $\bm{Z}$, $\mathbb{E}$ represents the
mathematical expectation, and $\mathbb{S}$ represents the evaluation metrics,
_e.g_., CIDEr [40].

##  4 Experiments

###  4.1 Dataset, Metrics, and Settings

MSCOCO. Following [31, 48, 14, 12, 8], we train and evaluate our model on
MSCOCO [22], which contains $123,287$ images, and each one is annotated with 5
captions. In the experiments, we use the Karpathy split (113,287/5,000/5,000
train/val/test images) [16] for offline training and the official split (40775
test images) for online testing.

Metrics. We adopt five widely-used metrics in captioning for evaluation,
including BLEU [32], METOR [1], ROUGE-L [36], CIDEr [40], and SPICE [3].

Settings. In the training process, we convert all the captions into lowercase
and delete all the words that occur less than 6 times. The remaining 9487
words are regarded as our vocabulary. We adopt Swin Transformer [24] as the
visual encoder to extract the visual features. The size of the feature map is
$H\times W=12\times 12$, and we apply the Down-Up Sampling Strategy (cf.
Section 3). We train 20/25 epochs in the cross-entropy/RL stage. In the cross-
entropy stage, the Adam optimizer is used with the learning rate of $5\times
10^{-5}$ and decays by 0.8 per 5 epochs. In the RL stage, the learning rate is
initialized to $5\times 10^{-6}$ and we implement the same decay policy for 10
epochs. Then the “Reduce-On-Plateau” strategy is applied with a decay rate of
0.5 and patience of 3. The batch size is 40 at the whole training stage.

###  4.2 Ablation Studies

We conduct extensive ablations for quantifying the difference between classic
self-attention (Self-ATT) layers and Ada-ClustFormer (ACF) layers (cf. Section
4.2.1), the impact of the depth of the ACF layers (cf. Section 4.2.2), and the
impact of the orders of ACF and the Self-ATT layers (cf. Section 4.2.3).

####  4.2.1 Differences Between ACF and Self-ATT

Comparing Methods. To evaluate the effectiveness of the ACF, we ablate our ACF
with the following baselines: $BASE$: We employ 6 Self-ATT encoder layers and
decoder layers, which is shown in Table 1 as “6S”. $ACF^{1}$ / $ACF^{2}$: We
replace the encoder/decoder with our ACF, which is represented as “$6$C”.
Results. The results of the ablation are listed in Table 1. Compared with
$BASE$, we can find that only using ACF encoder ($ACF^{1}$) or decoder
($ACF^{2}$) has marginal improvements, which is 0.3 or 0.7 on CIDEr. However,
when combining the ACF encoder and decoder to build a homogeneous architecture
ACF, the improvement is substantial, which is 3.4. This comparison suggests
that a homogeneous model transfers more structural commonalities for better
captions.

Table 1: Comparison between with and without Ada-ClustFormer.

Models | $m_{e}$ | $m_{d}$ | B@4 | M | R | C | S
---|---|---|---|---|---|---|---
$BASE$ | 6S | 6S | $40.0$ | $29.7$ | $59.6$ | $134.4$ | $23.4$
$ACF^{1}$ | 6C | 6S | $40.3$ | $29.6$ | $59.6$ | $134.7$ | $23.5$
$ACF^{2}$ | 6S | 6C | $40.2$ | $29.8$ | $59.9$ | $135.1$ | $23.7$
$ACF$ | 6C | 6C | $\bm{41.1}$ | $\bm{30.1}$ | $\bm{60.2}$ | $\bm{137.8}$ | $\bm{24.1}$

####  4.2.2 Impact of the Layer Depth

Comparing Methods. $ACF^{3}$: We reduce the depth of the encoder and decoder
layer to 3. $ACF^{4}$/$ACF^{5}$: The number of the encoder/decoder layers is
set to 3 and the number of the decoder/encoder layer remains 6.
Results. From Table 2, we observe that stacking 6 layers generally outperforms
the 3-layer case. Our method with 6 ACF layers in the encoder and decoder
achieves the best performance among them. We also further explore the
influence of $m_{e}$ by fixing $m_{d}=6$. We present the impact of the number
of the encoder layers $m_{e}$ in Figure 5. It suggests that CIDEr
approximately linearly increases when $m_{e}$ increases.

Table 2: The performances with different layer depth

Models | $m_{e}$ | $m_{d}$ | B@4 | M | R | C | S
---|---|---|---|---|---|---|---
$ACF^{3}$ | 3C | 3C | $38.9$ | $28.4$ | $58.8$ | $132.3$ | $22.0$
$ACF^{4}$ | 6C | 3C | $39.3$ | $28.9$ | $59.1$ | $135.9$ | $23.7$
$ACF^{5}$ | 3C | 6C | $40.2$ | $29.8$ | $59.7$ | $136.0$ | $24.0$
$ACF$ | 6C | 6C | $\bm{41.1}$ | $\bm{30.1}$ | $\bm{60.2}$ | $\bm{137.8}$ | $\bm{24.1}$

####  4.2.3 Impact of the Layer Order

Comparing Methods. We discuss the combination of the ACF layers and the Self-
ATT layers. We freeze the depth of the decoder layer $m_{d}=6$ and quantify
the influence of the order of the encoders: $ACF^{5}$: It stacks 3 ACF layers.
$ACF^{6}$/$ACF^{7}$: Both of them have 3 ACF layers and 3 Self-ATT layers. The
difference between them is that $ACF^{7}$ encodes on 3 Self-ATT layers
firstly.
Results. The results are listed in Table 3, where we can see that the
performances are not sensitive to the orders of ACF and Self-ATT layers,
_i.e_., $ACF^{6}$ and $ACF^{7}$ differ only 0.4. We can also find that
replacing all the Self-ATT layers with our ACF layers will achieve the best
captioning quality.

Figure 5: Impact of the number of encoder layers $m_{e}$. Table 3: The impact
of the layer orders.

Models | $m_{e}$ | $m_{d}$ | B@4 | M | R | C | S
---|---|---|---|---|---|---|---
$ACF^{5}$ |  $3$C |  $6$C | $40.2$ | $29.8$ | $59.7$ | $136.0$ | $24.0$
$ACF^{6}$ |  $3$C+ $3$S |  $6$C | $40.7$ | $29.7$ | $59.9$ | $135.7$ | $23.8$
$ACF^{7}$ |  $3$S+ $3$C |  $6$C | $40.5$ | $29.9$ | $59.9$ | $136.1$ | $23.9$
$ACF^{2}$ |  $6$S |  $6$C | $40.2$ | $29.8$ | $59.9$ | $135.1$ | $23.7$
$ACF$ |  $6$C |  $6$C | $\bm{41.1}$ | $\bm{30.1}$ | $\bm{60.2}$ | $\bm{137.8}$ | $\bm{24.1}$

Qualitative Results.

Figure 6: Examples of the generated captions by $BASE$ and $ACF$ models. We
visualize the 2-D $C$ and 1-D $C$ in the 1-st, 3-rd, 5-th, and 6-th layers as
the clustered patches.

We visualize the hierarchical structures of the image and the generated
captions in Figure 6 according to the 2-D and 1-D clustering matrix calculated
from the 1-st, 3-rd, 5-th, and 6-th layers in encoder and decoder. By
inspecting the images and captions, we can find that the patches and the words
are respectively clustered, _e.g_., in the left part of (b), the patches in
the “motorcycles” region are clustered, and in the right part, the words
“sitting on motorcycles” are clustered into a phrase. More importantly, when
uniting the image and caption, we can find that structural commonalities are
transferred, _e.g_., in (b), the “motorcycle” region helps generate the phrase
“sitting on motorcycles”.

Table 4: The performances of SOTA methods on MSCOCO Karpathy split.

Models | Cross-Entroy Loss |  | CIDEr optimization
---|---|---|---
B@4 | M | R | C | S |  | B@4 | M | R | C | S
${}^{\color[rgb]{1,0.2,0.2}{{\text{{ROI-based feature}}}}}$ |  |  |  |  |  |  |  |  |  |  |
Up-Down [4] | $36.2$ | $27.0$ | $56.4$ | $113.5$ | $20.3$ |  | $36.3$ | $27.7$ | $56.9$ | $120.1$ | $21.4$
ORT [12] | $35.5$ | $28.0$ | $56.6$ | $115.4$ | $21.2$ |  | $38.6$ | $28.7$ | $58.4$ | $128.3$ | $22.6$
AoANet [14] | $37.2$ | $28.4$ | $57.5$ | $119.8$ | $21.4$ |  | $38.9$ | $29.2$ | $58.8$ | $129.8$ | $22.4$
$\mathcal{M}^{2}$ Transformer [8] | - | - | - | - | - |  | $39.1$ | $29.2$ | $58.6$ | $131.2$ | $22.6$
CATT [50] | $37.3$ | $28.5$ | $57.4$ | $119.0$ | $21.5$ |  | $39.4$ | $29.3$ | $58.9$ | $131.7$ | $22.8$
APN [48] | - | - | - | - | - |  | $39.6$ | $29.2$ | $59.1$ | $131.8$ | $23.0$
X-Transformer [31] | $\bm{38.2}$ | $28.8$ | $58.0$ | $122.0$ | $\bm{21.9}$ |  | $39.7$ | $29.5$ | $59.2$ | $132.8$ | $23.2$
${}^{\color[rgb]{1,0.2,0.2}{\text{{Grid-based feature}}}}$ |  |  |  |  |  |  |  |  |  |  |
CPTR [23] | - | - | - | - | - |  | $40.0$ | $29.1$ | $59.4$ | $129.4$ | $-$
APN♯ [48] | - | - | - | - | - |  | $40.1$ | $29.4$ | $59.4$ | $133.2$ | $23.3$
Dual-Global [45] | - | - | - | - | - |  | $40.3$ | $29.2$ | $59.4$ | $132.4$ | $23.3$
DLCT [25] | - | - | - | - | - |  | $40.8$ | $29.9$ | $59.8$ | $137.5$ | $23.3$
${}^{\color[rgb]{0.2,0.2,1}{{\text{{End-to-End training}}}}}$ |  |  |  |  |  |  |  |  |  |  |
PureT-standard [42] | - | - | - | - | - |  | $40.3$ | $29.9$ | $59.9$ | $137.5$ | $23.8$
PureT-Swin [42] | - | - | - | - | - |  | $40.9$ | $\bm{30.2}$ | $60.1$ | $\bm{138.2}$ | $\bm{24.2}$
${}^{\color[rgb]{1,0.2,0.2}{{\text{{Visual-language BERT pretraining}}}}}$ |  |  |  |  |  |  |  |  |  |  |
RSTNet [54] | - | - | - | - | - |  | $40.1$ | $28.9$ | $59.5$ | $135.6$ | $23.3$
ViTCAP-small [10] | $35.7$ | $28.8$ | $57.6$ | $121.8$ | $22.1$ |  | $40.1$ | $29.4$ | $59.4$ | $133.1$ | $23.0$
ViTCAP-large [10] | $36.3$ | $29.3$ | $58.1$ | $125.2$ | $22.6$ |  | $41.2$ | $30.1$ | $60.1$ | $138.1$ | $24.1$
$\bm{ACF}$ | $38.1$ | $\bm{28.8}$ | $\bm{58.4}$ | $\bm{123.8}$ | $21.8$ |  | $\bm{41.1}$ | $30.1$ | $\bm{60.2}$ | $137.8$ | $24.1$

Table 5: The scores on the MSCOCO online test server.

Models | B@4 | M | R | C
---|---|---|---|---
c5 | c40 | c5 | c40 | c5 | c40 | c5 | c40
Up-Down [4] | $36.9$ | $68.5$ | $27.6$ | $36.7$ | $57.1$ | $72.4$ | $117.9$ | $120.5$
SGAE [49] | $37.8$ | $68.7$ | $28.1$ | $37.0$ | $58.2$ | $73.1$ | $122.7$ | $125.5$
ETA [19] | $38.9$ | $70.2$ | $28.6$ | $38.0$ | $58.6$ | $73.9$ | $122.1$ | $124.4$
APN [48] | $38.9$ | $70.2$ | $28.8$ | $38.0$ | $58.7$ | $73.7$ | $126.3$ | $127.6$
NG-SAN [11] | 38.8 | 70.2 | 29.0 | 38.4 | 58.7 | 74.0 | 126.3 | 128.6
Dual-Global [45] | $39.1$ | $71.2$ | $28.9$ | $38.4$ | $58.9$ | $74.4$ | $126.3$ | $129.2$
AoANet [14] | $39.4$ | $71.2$ | $29.1$ | $38.5$ | $58.9$ | $74.5$ | $126.9$ | $129.6$
$\mathcal{M}^{2}$ Transformer [8] | $39.7$ | $72.8$ | $29.4$ | $39.0$ | $59.2$ | $74.8$ | $129.3$ | $132.1$
RSTNet [54] | $39.7$ | $72.5$ | $29.3$ | $38.7$ | $59.2$ | $74.2$ | $130.1$ | $132.4$
$\bm{ACF}$ | $\bm{39.0}$ | $\bm{71.3}$ | $\bm{29.2}$ | $\bm{39.2}$ | $\bm{59.2}$ | $\bm{74.2}$ | $\bm{130.2}$ | $\bm{132.3}$

###  4.3 Comparisons with SOTA

Comparing Methods. Nowadays, the SOTA of image captioning has been updated
quickly and these models can be categorized into 3 groups. The first one is
the methods which use ROI-based features, including Up-Down [4], ORT [12],
AoANet [14], $\mathcal{M}^{2}$ Transformer [8], TreeTransformer [43], APN
[48], and X-Transformer [31]. Among the above methods, Up-Down [4] deploys a
famous architecture with a CNN-based encoder and an LSTM-based decoder. ORT
[12] applies Transformer to language decoder. AoANet [14] and
$\mathcal{M}^{2}$ Transformer [8] further improve the attention mechanism on
the language decoder. Tree-Transformer [43] and APN [48] reveal the validity
of the utilization of the sequence structure. To capture high-order
interaction between sequence and regions, X-Transformer [31] introduces a
bilinear pooling structure. The second group are the methods using grid-based
features: CPTR [23], Dual-Global [45], DLCT [25], and PureT [42]. Among them,
Dual-Global [45] and DLCT [25] combine the grid-based features with the ROI-
based features. PureT [42] end-to-end trains the whole model and PureT-
standard/PureT-Swin respectively use Transformer [9]/Swin Transformer [24] as
the vision encoder to deal with the visual features, which is also extracted
from a Swin Transformer. The third group distills the knowledge from large-
scale pretraining models: RSTNet [54], and ViTCAP [10]. Accordingly, we
segment the performances into 3 parts in Table 4, where the top/middle/bottom
parts are the ROI-based, grid-based, and BERT-based models. Note that for APN,
besides reporting the results in their paper [48], which is got by using ROI-
based features, we also report the performances using the same visual features
as ours, which is denoted as “APN♯”.
Results. From Table 4, we can see that ACF is comparable to most of state-of-
the-art performance when compared with ROI and grid-based models. Moreover,
ACF achieves comparable performances with ViTCAP-large [10] that distills
knowledge from Google-CC [37], SBU Caption dataset [30], MSCOCO [22], and
Visual Genome dataset [17], which uses 9.9M image-text pairs and 4.1M
independent images to pretrain a detector-free IC model. However, we only use
the captions from MSCOCO to train our ACF. Moreover, compared with APN♯ [48]
which inserts an additional clustering matrix into the Self-ATT layers into
the decoder, ACF achieves higher performance since it inserts the clustering
matrix in both vision encoder and language decoder to build a homogeneous
model.

Also, we submit the single-model results to the online server for testing,
which is shown in Table 5. We can see that ACF achieves the best performance
than the other models, even we do not ensemble the results as AoANet [14],
$\mathcal{M}^{2}$ Transformer [8], and RSTNet [54].

Limitations and Potential Solutions. From Table 4, we can find that PureT-Swin
[42] achieves higher CIDEr than ours. There are two major reasons cause this.
Firstly, PureT-Swin extracts visual features from Swin Transformer [24] and
then still uses Swin Transformer as the visual encoder to deal with the
extracted features. For ACF, the used vision encoder is quite different from
Swin Transformer that they apply shifted fixed-size windows, while we insert
an adaptive clustering matrix into the Transformer. In this way, the whole
captioning model (including the vision extractor) is not a strictly
homogeneous structure. Also, it can be seen that ACF outperforms PureT-
standard which applies a standard Transformer as the vision encoder, which
means that once PureT is not homogeneous, their performance will be worse.

Secondly, they end-to-end train the whole architecture by captioning data
since Swin Transformer [24] provides well-trained parameters that PureT does
not need to train their visual extractor from scratch. However, this requires
heavy computation resources to end-to-end train the visual extractor by image
annotations while we now cannot afford such computation burdens. However, even
with these two limitations, it can be found that ACF still achieves comparable
performances compared with PureT.

To solve these limitations, we prepare to extend the computation resources
like the GPU servers to build a novel pure vision global-local Transformer
where ACF prior is used to learn hierarchical structure. And then using this
model to extract visual features for solving more vision-language tasks,
_e.g_., by building a homogeneous ACF-based vision-language model.

##  5 Conclusion

We propose a novel global-local Transformer named as Ada-ClustFormer (ACF)
that can adaptively cluster the input elements for carrying self-attention
(Self-ATT) to learn global-local contexts. Specifically, this is achieved by
inserting a clustering matrix into the Self-ATT layer, where the probability
terms are calculated from the input data and thus ACF can adaptively cluster
the elements. Moreover, we use ACF to build an image captioning model to
transfer more structural commonalities for better captions. The experiment
results confirm the effectiveness of the proposed model.

## References

  * [1] Abhaya Agarwal and Alon Lavie.  Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments.  Proceedings of WMT-08, 2007.
  * [2] Mahtab Ahmed, Muhammad Rifayat Samee, and Robert E Mercer.  You only need attention to traverse trees.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 316–322, 2019.
  * [3] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.  Spice: Semantic propositional image caption evaluation.  In European conference on computer vision, pages 382–398. Springer, 2016.
  * [4] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086, 2018.
  * [5] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al.  Relational inductive biases, deep learning, and graph networks.  arXiv preprint arXiv:1806.01261, 2018.
  * [6] Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document transformer.  arXiv preprint arXiv:2004.05150, 2020.
  * [7] Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang.  Glit: Neural architecture search for global and local image transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12–21, 2021.
  * [8] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara.  Meshed-memory transformer for image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578–10587, 2020.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2021.
  * [10] Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lin Liang, Zhe Gan, Lijuan Wang, Yezhou Yang, and Zicheng Liu.  Injecting semantic concepts into end-to-end image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18009–18019, 2022.
  * [11] Longteng Guo, Jing Liu, Xinxin Zhu, Peng Yao, Shichen Lu, and Hanqing Lu.  Normalized and geometry-aware self-attention network for image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10327–10336, 2020.
  * [12] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares.  Image captioning: Transforming objects into words.  In Advances in Neural Information Processing Systems, pages 11137–11147, 2019.
  * [13] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang.  Scaling up vision-language pre-training for image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17980–17989, 2022.
  * [14] Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei.  Attention on attention for image captioning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4634–4643, 2019.
  * [15] Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, and Xinlei Chen.  In defense of grid features for visual question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10267–10276, 2020.
  * [16] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015.
  * [17] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International Journal of Computer Vision, 123(1):32–73, 2017.
  * [18] Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung.  Vilbertscore: Evaluating image caption using vision-and-language bert.  In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 34–39, 2020.
  * [19] Guang Li, Linchao Zhu, Ping Liu, and Yi Yang.  Entangled transformer for image captioning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [20] Jinpeng Li, Yichao Yan, Shengcai Liao, Xiaokang Yang, and Ling Shao.  Local-to-global self-attention in vision transformers.  arXiv preprint arXiv:2107.04735, 2021.
  * [21] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In European Conference on Computer Vision, pages 121–137. Springer, 2020.
  * [22] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [23] Wei Liu, Sihan Chen, Longteng Guo, Xinxin Zhu, and Jing Liu.  Cptr: Full transformer network for image captioning.  arXiv preprint arXiv:2101.10804, 2021.
  * [24] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [25] Yunpeng Luo, Jiayi Ji, Xiaoshuai Sun, Liujuan Cao, Yongjian Wu, Feiyue Huang, Chia-Wen Lin, and Rongrong Ji.  Dual-level collaborative transformer for image captioning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2286–2293, 2021.
  * [26] Minh-Thang Luong, Hieu Pham, and Christopher D Manning.  Effective approaches to attention-based neural machine translation.  arXiv preprint arXiv:1508.04025, 2015.
  * [27] Ron Mokady, Amir Hertz, and Amit H Bermano.  Clipcap: Clip prefix for image captioning.  arXiv preprint arXiv:2111.09734, 2021.
  * [28] Van-Quang Nguyen, Masanori Suganuma, and Takayuki Okatani.  Grit: Faster and better image captioning transformer using dual visual features.  arXiv preprint arXiv:2207.09666, 2022.
  * [29] Xuan-Phi Nguyen, Shafiq Joty, Steven Hoi, and Richard Socher.  Tree-structured attention with hierarchical accumulation.  In International Conference on Learning Representations, 2020.
  * [30] Vicente Ordonez, Girish Kulkarni, and Tamara Berg.  Im2text: Describing images using 1 million captioned photographs.  Advances in neural information processing systems, 24, 2011.
  * [31] Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei.  X-linear attention networks for image captioning.  In CVPR, pages 10971–10980, 2020.
  * [32] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [33] Samrudhdhi B Rangrej, Kevin J Liang, Tal Hassner, and James J Clark.  Glitr: Glimpse transformers with spatiotemporal consistency for online action prediction.  arXiv preprint arXiv:2210.13605, 2022.
  * [34] S Ren, K He, R Girshick, and J Sun.  Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 2015.
  * [35] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel.  Self-critical sequence training for image captioning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7008–7024, 2017.
  * [36] Lin CY ROUGE.  A package for automatic evaluation of summaries.  In Proceedings of Workshop on Text Summarization of ACL, Spain, 2004\.
  * [37] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [38] Ying Hua Tan and Chee Seng Chan.  Phrase-based image caption generator with hierarchical lstm network.  Neurocomputing, 333:86–100, 2019.
  * [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [40] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  * [41] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.  Show and tell: A neural image caption generator.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2015.
  * [42] Yiyu Wang, Jungang Xu, and Yingfei Sun.  End-to-end transformer based model for image captioning.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2585–2594, Jun. 2022.
  * [43] Yau-Shian Wang, Hung-Yi Lee, and Yun-Nung Chen.  Tree transformer: Integrating tree structures into self-attention.  arXiv preprint arXiv:1909.06639, 2019.
  * [44] Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang.  Hi-transformer: hierarchical interactive transformer for efficient and effective long document modeling.  arXiv preprint arXiv:2106.01040, 2021.
  * [45] Tiantao Xian, Zhixin Li, Canlong Zhang, and Huifang Ma.  Dual global enhanced transformer for image captioning.  Neural Networks, 148:129–141, 2022.
  * [46] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In International conference on machine learning, pages 2048–2057. PMLR, 2015.
  * [47] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao.  Focal attention for long-range interactions in vision transformers.  Advances in Neural Information Processing Systems, 34:30008–30022, 2021.
  * [48] Xu Yang, Chongyang Gao, Hanwang Zhang, and Jianfei Cai.  Auto-parsing network for image captioning and visual question answering.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2197–2207, 2021.
  * [49] Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai.  Auto-encoding scene graphs for image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10685–10694, 2019.
  * [50] Xu Yang, Hanwang Zhang, Guojun Qi, and Jianfei Cai.  Causal attention for vision-language tasks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9847–9857, 2021.
  * [51] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei.  Hierarchy parsing for image captioning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2621–2629, 2019.
  * [52] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579–5588, 2021.
  * [53] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun.  Exploring self-attention for image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10076–10085, 2020.
  * [54] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao.  Unified vision-language pre-training for image captioning and vqa.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13041–13049, 2020.