Triangular Numbers With a Single Repeated Digit
Christian Hercher
Institut für Mathematik
Europa-Universität Flensburg
Auf dem Campus 1c
24943 Flensburg
Germany
<EMAIL_ADDRESS>
Karl Fegert
Neu-Ulm
Germany
<EMAIL_ADDRESS>

###### Abstract

The question of which triangular numbers have a decimal representation
containing a single repeated digit seamed to be settled since at least the
1970s: Ballew and Weger provided a complete list and a proof that these are
the only numbers of this kind. This assertion is referenced by other authors
in the field. However, their proof is flawed. We present a new and elementary
proof of the statement, which corrects the error.

##  1 Introduction

The study of triangular numbers is a long-standing and well-researched area
within the field of number theory. For example, Gauß proved in his
Disquisitiones Arithmeticae [4] that every positive integer can be expressed
as the sum of three triangular numbers.

A further question that arises is that of the decimal representation of
triangular numbers, in particular: Which triangular numbers can be expressed
using a single digit? That is to say, are there triangular numbers of the form
$d\cdot\frac{10^{i}-1}{9}$ with some digit $d\in\\{1,\dots,9\\}$ and postivie
integer $i$? One instance for this question is Problem 15648 [2], which was
proposed by Youngman and subsequently answered by Escott in the same journal.
However, Escott demonstrated only that triangular numbers with repeated digits
exceeding $666$, comprising a maximum of $30$ digits, do not exist.

Subsequently, Ballew and Weger [1] presented an erroneous proof for the
conjecture that the only triangular numbers whose decimal representations
consist of a single repeated digit are $1,3,6,55,66,$ and $666$.

In a recent contribution, Kafle, Luca, and Togbé [5] generalized the problem
to include repeating blocks with two digits: The only triangular numbers whose
decimal representation can be written using only one block of two digits are
$10,15,21,28,36,45,55,66,78,91,5050,$ and $5151$. However, for the original
problem, they just cite the work of Ballew and Weger.

Recently, on the internet platform MathOverflow [6] a discussion regarding
this topic was held. In this discussion, the error in Ballew and Weger’s proof
was identified, and a proof using the methods of elliptic curves was
presented. In this paper, we provide an elementary proof for this statement
using only Pell equations.

### Outline

In accordance with the methodology proposed by Escott, Ballew, and Weger, we
initially reformulate the problem. Subsequently, we provide a concise account
of the erroneous assertion present within the proof of Ballew and Weger. In
the following section, a proof is provided for each remaining digit,
demonstrating that there are no further triangular numbers consisting of a
single repeated digit.

##  2 Reduction and reformulation of the problem

For a positive integer $k$ let $T_{k}$ denote the $k$’th triangular number. In
accordance with Escott’s approach and the valid part of Ballew’s and Weger’s
paper, we arrive at the following conclusion: If the decimal expression of
$T_{k}$ consists solely of the single repeated digit $d$, we have

$\displaystyle T_{k}=\frac{k(k+1)}{2}$
$\displaystyle=d\cdot\frac{10^{i}-1}{9},i\geq 1,d\in\\{1,\dots,9\\}$
$\displaystyle\iff(2k+1)^{2}=4k^{2}+4k+1$
$\displaystyle=8d\cdot\frac{10^{i}-1}{9}+1.$

A solution $(k,d,i)$ in positive integers for the original problem exists, if
and only if the quantity $D:=1+8d\cdot\frac{10^{i}-1}{9}$ is a perfect square.

It is $D\equiv 1+8d\ \mbox{\rm(mod}\ 10\mbox{\rm)}$. Thus, immediately, we can
rule out $d\in\\{2,4,7,9\\}$, because in those cases $D$ would be congruent
$7$ or $3$ modulo $10$, which are not quadratic residues. Therefore, the value
of $d$ must be $1,3,5,6,$ or $8$. Now we look at these cases one by one:

  * $d=1$:

If $i=1$ it follows $D=8\cdot\frac{10^{i}-1}{9}+1=9=3^{2}$, which yields
$T_{1}=1$, and for $i>1$ we have $D=88\ldots 89$.

  * $d=3$:

If $i=1$ then $D=24\cdot\frac{10^{i}-1}{9}+1=25=5^{2}$, which yields
$T_{2}=3$. Otherwise it is $D=26\ldots 65$.

  * $d=5$:

If $i=1$ we have $D=40\cdot\frac{10^{i}-1}{9}+1=41$. For $i=2$ we get
$D=441=21^{2}$ ,which yields $T_{10}=55$, and for $i>2$ it is $D=44\ldots 41$.

  * $d=6$:

If $i=1$, $2$, or $3$ then $D=48\cdot\frac{10^{i}-1}{9}+1=49=7^{2}$,
$D=529=23^{2}$, or $D=5329=73^{2}$, which yields $T_{3}=6,T_{11}=66$, and
$T_{36}=666$. If $i>3$ it is $D=53\ldots 329$.

  * $d=8$:

If $i=1$ or $2$ then $D=64\cdot\frac{10^{i}-1}{9}+1=65$ or $D=705$. Otherwise
$D=71\ldots 105$.

Since $\ldots 05$ and $\ldots 65$ cannot be the last two digits of a square
number, it can be concluded that $d=8$ can be excluded and $d=3$ as well,
except for $T_{3}$.

### The false statement

Ballew and Weger have incorrectly asserted that there is no number $z=\ldots
88889$ whose square has more than four final digits $8$. This assertion is,
however, erroneous. For example, $8072917^{2}=\ldots 88888889$. In fact, since
there are solutions to the congruence equations $p^{2}\equiv
8\cdot\frac{-1}{9}+1\ \mbox{\rm(mod}\ 2^{3}\mbox{\rm)}$ and $p^{2}\equiv
8\cdot\frac{-1}{9}+1\ \mbox{\rm(mod}\ 5^{1}\mbox{\rm)}$, Hensel’s Lemma
ensures solutions modulo every power of $2$ and $5$, and thus, modulo every
power of $10$. Conequently, there are square numbers whose last digits are
$\ldots 8889$ with an arbitrary number of digits $8$.

##  3 A proof using Pell equations

We therefore introduce a new line of reasoning which rules out the remaining
cases, namely those with $d\in\\{1,5,6\\}$.

### The case $d=1$

$p^{2}=88\ldots 89=8\cdot\frac{10^{i}-1}{9}+1\iff(3p)^{2}-(8\cdot 10^{i})=1$

Assume there is a solution $(p,i)$, $i\geq 2$ (the case $i=1$ leads to
$T_{1}$): Then we can make a case distinction by the parity of $i$.

  * Case A:

$i$ even, i.e., $i=2r$, $r\in\mathbb{N}$. Subsequently, we have that

$\displaystyle(3p)^{2}-2\cdot(2\cdot 10^{r})^{2}$ $\displaystyle=1,$ and we
seek for solutions to the Pell equation $\displaystyle x^{2}-2y^{2}$
$\displaystyle=1$ (1)

with the requirements $x=3p$ and $y=2\cdot 10^{r},r\geq 1$. Thus, $y$ must
satisfy $y\equiv 0\ \mbox{\rm(mod}\ 5\mbox{\rm)}$ and $y\not\equiv 0\
\mbox{\rm(mod}\ 7\mbox{\rm)}$.

But the solutions to (1) are given by $(x_{0},y_{0})=(3,2)$,
$x_{n+1}=3x_{n}+4y_{n}$, $y_{n+1}=2x_{n}+3y_{n}$. A calculation modulo 5 and
modulo 7 as in Table 1 yields $y\equiv 0\ \mbox{\rm(mod}\ 5\mbox{\rm)}\iff
y\equiv 0\ \mbox{\rm(mod}\ 7\mbox{\rm)}$, hence $y\neq 2\cdot 10^{r}$.

$n$ | $x_{n}\bmod{5}$ | $y_{n}\bmod{5}$ | $x_{n}\bmod{7}$ | $y_{n}\bmod{7}$
---|---|---|---|---
0 | 3 | 2 | 3 | 2
1 | 2 | 2 | 3 | 5
2 | 4 | 0 | 1 | 0
3 | 2 | 3 | 3 | 2
4 | 3 | 3 | 3 | 5
5 | 1 | 0 | 1 | 0
6 | 3 | 2 | 3 | 2
7 | $\dots$
Table 1: Solutions of equation (1) for $d=1$, Case A, mod $5$ and mod $7$.

  * Case B:

$i$ odd, i.e., $i=2r+1$, $r\in\mathbb{N}$. Then we have
$(3p)^{2}-20\cdot(2\cdot 10^{r})^{2}=1$, and we seek for solutions to

$\displaystyle x^{2}-20y^{2}=1$ (2)

with $x=3p$ and $y=2\cdot 10^{r}$, $r\geq 1$. Hence, it is $y\equiv 0\
\mbox{\rm(mod}\ 5\mbox{\rm)}$ and $y\not\equiv 0\ \mbox{\rm(mod}\
11\mbox{\rm)}$.

But the solutions to (2) are $(x_{0},y_{0})=(9,2)$, $x_{n+1}=9x_{n}+40y_{n}$,
$y_{n+1}=2x_{n}+9y_{n}$, and a calculation modulo $5$ and $11$ similar to the
one in Case A yields $y\equiv 0\ \mbox{\rm(mod}\ 5\mbox{\rm)}\iff y\equiv 0\
\mbox{\rm(mod}\ 11\mbox{\rm)}$, so $y\neq 2\cdot 10^{r}$ with $r\geq 1$.

### The case $d=5$

$p^{2}=44\ldots 41=40\cdot\frac{10^{i}-1}{9}+1.$

Assume, there is a solution $(p,i)$.

  * Case A:

$i$ even, i.e., $i=2r$, $r\in\mathbb{N}$. Then we have
$(3p)^{2}-10\cdot(2\cdot 10^{r})^{2}=-31$, and we seek solutions to the
equation

$\displaystyle x^{2}-10y^{2}$ $\displaystyle=-31$ (3)

with $x=3p$ und $y=2\cdot 10^{r}$, $r\geq 1$. Hence, $y$ has to satisfy the
congruences $y\equiv 0\ \mbox{\rm(mod}\ 8\mbox{\rm)}$ and $y\not\equiv 0\
\mbox{\rm(mod}\ 7\mbox{\rm)}$.

It is evident, that $(x_{0},y_{0})\in\\{(3,2);(63,20)\\}$ are solutions to
(3). All other solutions of (3) can be obtained from these by
$x_{n+1}=19x_{n}+60y_{n}$, $y_{n+1}=6x_{n}+19y_{n}$. A calculation modulo $8$
and $7$ as in the case $d=1$ yields $y\equiv 0\ \mbox{\rm(mod}\
8\mbox{\rm)}\iff y\equiv 0\ \mbox{\rm(mod}\ 7\mbox{\rm)}$, so $y\neq 2\cdot
10^{r}$.

  * Case B:

$i$ odd, i.e., the number of digits four in the decimal representation of
$p^{2}$ is odd. Then according to a basic rule

$44\ldots 41\equiv 1-4+(4-4)+\dots+(4-4)\equiv-3\equiv 8\ \mbox{\rm(mod}\
11\mbox{\rm)},$

which is not a quadratic residue modulo $11$.

(This is Problem 2 from the first round of the 2024 Bundeswettbewerb
Mathematik in Germany [3]. When we tried to solve the related open question
for Case A, i.e., whether 1 and 441 are the only squares with an even number
of digits four, we came across Ballews and Wegers flawed proof and ended up
with this paper.)

### The case $d=6$

Assume, there is a solution $(p,i)$ with $i\geq 4$. (The cases with $i\leq 3$
yield the solutions $T_{3}$, $T_{11}$, and $T_{36}$.) Then we have

$p^{2}=53\ldots 329=48\cdot\frac{10^{i}-1}{9}+1\iff 9p^{2}-3\cdot(4^{2}\cdot
10^{i})=-39.$

  * Case A:

$i$ even, i.e., $i=2r$. Thus, $3p^{2}-(4\cdot 10^{r})^{2}=-13$ and we seek
solutions to the Pell equation

$\displaystyle x^{2}-3y^{2}$ $\displaystyle=13$ (4)

with the additional requirement of $x=4\cdot 10^{r}$, $y=3p$, and $r\geq 2$.
Hence, it has to be $x\equiv 0\ \mbox{\rm(mod}\ 50\mbox{\rm)}$. We get all
solutions of (4) starting from $(x_{0},y_{0})=(4,1)$ or $(x_{0},y_{0})=(5,2)$
through the recursion $x_{n+1}=2x_{n}+3y_{n}$ and $y_{n+1}=x_{n}+2y_{n}$. A
calculation modulo $50$ and $241$ as in the case $d=1$ yields $x_{i}\equiv 0\
\mbox{\rm(mod}\ 50\mbox{\rm)}\iff x_{i}\equiv 94\ \mbox{\rm(mod}\
241\mbox{\rm)}$. Thus, in every solution we have $94\equiv 4\cdot 10^{r}\
\mbox{\rm(mod}\ 241\mbox{\rm)}$ and therefore, $10^{r}\equiv 144\
\mbox{\rm(mod}\ 241\mbox{\rm)}$. However, this is not the case for any
positive integer $r$.

  * Case B:

$i$ odd, i.e., $i=2r+1$. This leads to $(3p)^{2}-30\cdot(4\cdot
10^{r})^{2}=-39$ and therefore, the Pell equation

$\displaystyle x^{2}-30y^{2}$ $\displaystyle=-39$ (5)

with $x=3p$, $y=4\cdot 10^{r}$, and $r\geq 2$. All solutions of (5) can be
obtained from the starting solutions $(x_{0},y_{0})=(9,2)$ or
$(x_{0},y_{0})=(21,4)$ through the recursion $x_{n+1}=11x_{n}+60y_{n}$,
$y_{n+1}=2x_{n}+11y_{n}$.

Now assume there is a solution with $r\geq 4$. Then it follows $y=4\cdot
10^{r}\equiv 0\ \mbox{\rm(mod}\ 64\mbox{\rm)}$. A calculation modulo $64$ and
$31$ as in the case $d=1$ yields $y_{i}\equiv 0\ \mbox{\rm(mod}\
64\mbox{\rm)}\iff y_{i}\equiv 3\ \mbox{\rm(mod}\ 31\mbox{\rm)}$ and, therefore
$10^{r}\equiv 24\ \mbox{\rm(mod}\ 31\mbox{\rm)}$, which is not the case for
any positive integer $r$.

In the cases with $r\leq 3$, it can be verified that in both families of
solutions of (5) we have $y_{3}>4000$. Consequently, the only remaining
candidates for additional solutions are $(x_{1},y_{1})$ and $(x_{2},y_{2})$ in
both families. However, a brief calculation reveals that they are no solutions
to the original problem.

This demonstrates that, in each case, there are no additional solutions beyond
those initially presented in the paper, thereby establishing the proof’s
conclusion.

######  Remark 1.

The method can be applied in the generalized version with repeated blocks of
digits as well. In consideration of blocks of length $2$, as presented in [5],
we have to solve the equation

$\displaystyle\frac{k(k+1)}{2}$ $\displaystyle=$ $\displaystyle
c\cdot\frac{100^{i}-1}{99},\text{ $c\in\\{10,\dots,99\\}$}$
$\displaystyle\iff$ $\displaystyle(2k+1)^{2}$ $\displaystyle=$ $\displaystyle
8c\cdot\frac{100^{i}-1}{99}+1$ $\displaystyle\iff$
$\displaystyle(33\cdot(2k+1))^{2}$ $\displaystyle=$ $\displaystyle 88c\cdot
100^{i}+1089-88c$ $\displaystyle\iff$ $\displaystyle x^{2}-22c\cdot y^{2}$
$\displaystyle=$ $\displaystyle 1089-88c,\text{ with
}x:=33\cdot(2k+1),y:=2\cdot 10^{i}$

in positive integers. Now we can compute the set of all integer solutions
$(x,y)$ to the Pell equation given in the last line and find moduli that
disprove that $y$ can be of the form $y=2\cdot 10^{i}$.

## References

  * [1] D. W. Ballew and R. C. Weger, Triangular numbers with repeated digits, Proc. S.D. ACAD. Sci. 51 (1972), 52–55, https://sdaos.org/wp-content/uploads/pdfs/Vol%2051%201972/72p52.pdf.
  * [2] E. B. Escott, and C. E. Youngman, Problem 15648, Educational Times 8 (1905), 33–34, https://archive.org/details/educationaltimes58educ/page/87/mode/1up.
  * [3] K. Fegert, Bundeswettbewerb Mathematik – Aufgaben und Lösungen 1. Runde 2024, https://www.mathe-wettbewerbe.de/fileadmin/Mathe-Wettbewerbe/Bundeswettbewerb_Mathematik/Dokumente/Aufgaben_und_Loesungen_BWM/2024/loes_24_1_e.pdf, 2024.
  * [4] C. F. Gauß, Disquisitiones Arithmeticae, In commiss. apud Gerh. Fleischer, 1801.
  * [5] B. Kafle, F. Luca, and A. Togbé, Triangular repblocks, Fibonacci Quart. 56 (2018), 325–328, https://www.fq.math.ca/Papers1/56-4/kafle1022018.pdf.
  * [6] Internet discussion on mathoverflow, triangular repdigits, https://mathoverflow.net/questions/467405/triangular-repdigits, 2024.


#  COPA : Efficient Vision-Language Pre-training Through Collaborative Object-
and Patch-Text Alignment

Chaoya Jiang<EMAIL_ADDRESS>National Engineering Research Center for
Software Engineering, Peking University, Peking UniversityBeijingChina ,
Haiyang Xu<EMAIL_ADDRESS>DAMO Academy, Alibaba
GroupHangzhouChina ,  Wei Ye<EMAIL_ADDRESS>National Engineering Research
Center for Software Engineering, Peking University, Peking
UniversityBeijingChina ,  Qinghao Ye<EMAIL_ADDRESS>DAMO
Academy, Alibaba GroupHangzhouChina ,  Chenliang Li<EMAIL_ADDRESS>DAMO Academy, Alibaba GroupHangzhouChina ,  Ming Yan<EMAIL_ADDRESS>DAMO Academy, Alibaba GroupHangzhouChina ,  Bin Bi<EMAIL_ADDRESS>DAMO
Academy, Alibaba GroupHangzhouChina ,  Shikun Zhang<EMAIL_ADDRESS>National Engineering Research Center for Software Engineering, Peking
University, Peking UniversityBeijingChina ,  Ji Zhang  zj122146@alibaba-
inc.com DAMO Academy, Alibaba GroupHangzhouChina and  Fei Huang
<EMAIL_ADDRESS>DAMO Academy, Alibaba GroupHangzhouChina

(2023)

###### Abstract.

Vision-Language Pre-training (VLP) methods based on object detection enjoy the
rich knowledge of fine-grained object-text alignment but at the cost of
computationally expensive inference. Recent Visual-Transformer (ViT)-based
approaches circumvent this issue while struggling with long visual sequences
without detailed cross-modal alignment information. This paper introduces a
ViT-based VLP technique that efficiently incorporates object information
through a novel patch-text alignment mechanism. Specifically, we convert
object-level signals into patch-level ones and devise a Patch-Text Alignment
pre-training task (PTA) to learn a text-aware patch detector. By using off-
the-shelf delicate object annotations in 5% training images, we jointly train
PTA with other conventional VLP objectives in an end-to-end manner, bypassing
the high computational cost of object detection and yielding an effective
patch detector that accurately detects text-relevant patches, thus
considerably reducing patch sequences and accelerating computation within the
ViT backbone. Our experiments on a variety of widely-used benchmarks reveal
that our method achieves a speedup of nearly 88% compared to prior VLP models
while maintaining competitive or superior performance on downstream tasks with
similar model size and data scale.

Vision-Language Pretraining; Efficiency; Patch-Text Alignment; Detection

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
31st ACM International Conference on Multimedia; October 29-November 3, 2023;
Ottawa, ON, Canada††booktitle: Proceedings of the 31st ACM International
Conference on Multimedia (MM ’23), October 29-November 3, 2023, Ottawa, ON,
Canada††price: 15.00††doi: 10.1145/3581783.3611826††isbn:
979-8-4007-0108-5/23/10††ccs: Information systems Multimedia and multimodal
retrieval††ccs: Computing methodologies Image representations††ccs: Computing
methodologies Object detection

##  1\. Introduction

Figure 1. Subfigure (a) illustrates the impact of Text-aware Patch Detector
(TPD) in the VQA scenario on various keeping ratios, which is a hyperparameter
determining the proportion of retained visual tokens to all tokens. Subfigure
(b) demonstrates how Patch-Text Alignment converts object-level annotations to
patch-level annotations and optimizes TPD based on the obtained supervision
signals. Subfigure (c) presents the VQA accuracy and throughput results for
our VLP model and the baseline.

Recently, Vision-Language Pre-training (VLP) (Tan and Bansal, 2019; Chen et
al., 2020; Lu et al., 2019; Huang et al., 2020; Su et al., 2020; Li et al.,
2020; Chen et al., 2020; Zhou et al., 2020; Li et al., 2021; Yu et al., 2021;
Li et al., 2022b) has achieved remarkable success across a wide range of
Vision-Language (VL) tasks, establishing itself as a dominant paradigm.
Existing VLP methods can be broadly classified into two categories based on
their approach to image feature extraction.

Detection-based Models (Tan and Bansal, 2019; Chen et al., 2020; Lu et al.,
2019; Li et al., 2020) identify objects/regions within images by employing
pre-trained object detectors (Ren et al., 2015; Redmon et al., 2016; He et
al., 2017) and learn to align detected objects/regions with text. Although
these methods benefit from the valuable fine-grained cross-modal alignment
knowledge, they are burdened by the considerable computational cost of
object/region detection and the error propagation issues arising from the two-
step pre-training strategy.

ViT-based Models refer to more recent efforts (Li et al., 2021; Radford et
al., 2021; Kim et al., 2021; Wang et al., 2021a; Singh et al., 2021; Li et
al., 2022b) built on Vision Transformer (ViT) (Dosovitskiy et al., 2021). They
utilize ViT as the visual encoder or cross-modal fusion encoder due to its
capacity to handle lengthy visual sequences derived from image grids/patches.
Although these approaches sidestep the high computational cost and error
propagation associated with object detection, they still face two challenges.
First, they can only learn coarse-grained alignments between the entire image
and text, owing to the absence of fine-grained alignment annotations (e.g.,
between patches and text), while acquiring fine-grained alignment in VLP is
essential for numerous cross-modal understanding and reasoning tasks (e.g.,
Visual Question Answering and Visual Grounding). Second, many redundant
patches in the image do not align with the input text, resulting in extended
visual sequences, particularly for high-resolution images. For instance, as
illustrated in Figure 1(a), during the inference of the VQA task with an image
size of 386$\times$386 for the question ”What is the weather like?”,
approximately 70% of the patches in the image depict a bird and sand, which do
not correspond to the question. Eliminating these text-irrelevant patches will
not impact the model output but can significantly reduce inference time.

To harness the strengths of both model types while mitigating their
shortcomings, we propose a novel VLP method named COPA, which effectively
incorporates fine-grained object-text alignment knowledge into a ViT-based
model in a lightweight and efficient manner. COPA stands for Collaboration
between Object- and Patch-Text Alignment. Specifically, by transforming
object/region annotations to patch-level annotations, we design an innovative
auxiliary pre-training task— Patch-Text Alignment (PTA)—to capture fine-
grained cross-modal alignment knowledge and train an accurate Text-aware Patch
Detector (TPD). Our method offers two key benefits. First, PTA is jointly
trained with other traditional VLP pre-training tasks using ready-made object
annotations from just 5% of training images, resulting in an end-to-end
architecture without the need for additional computation, e.g., of object
detection in prior detection-based models. Second, guided by PTA, the TPD,
which is integrated into the ViT-based visual backbone, can accurately
identify text-related patches, thereby reducing the length of the visual
sequence and further decreasing the overall computational cost. Impressively,
we employ a relatively small number of object-level labels from the COCO (Lin
et al., 2014)(0.11M) and VG (Krishna et al., 2016)(0.10M) datasets, yet
succeed in training a robust and generalizable patch detector that can
effectively identify text-relevant patches in large-scale and possibly out-of-
domain pre-training data in CC (Sharma et al., 2018)(3M) dataset (see Figure
5).

We evaluate COPA on various representative VL understanding and generation
tasks, including visual question answering, cross-modal retrieval, and image
captioning. Our findings show that by retaining only 50% text-relevant image
patches in the visual backbone, we can achieve competitive or even superior
downstream task performance. Simultaneously, the efficiency (more
specifically, the throughput) of COPA increases by 88% compared to previous
similar VLP models. For example, as illustrated in Figure 1 (c) and Table 4,
COPA boosts the throughput of the baseline from 186.42 to 349.71 and even
improves by about 0.3 on the VQA test-dev under the same experimental
settings. Moreover, by increasing the input image resolution, COPA attains
well-designed state-of-the-art downstream task performance (e.g., 78.25 on VQA
test-dev, see Table 5) as it benefits from incorporating more image tokens
without raising computational costs compared to other baselines.

Figure 2. (a) Overview of our VLP model ( COPA ). By incorporating the PTA
task, we can learn the fine-grained patch-text alignment end-to-end through
joint optimization with other pre-training tasks. (b) The architecture of the
Text-aware Patch Detector (TPD) is plugged into the ViT-base visual backbone
(ViT-TPD). In this sub-figure, we give a simplified example to show how to
detect text-relevant patches and calculate the PTA Loss.

##  2\. Related Work

###  2.1. Vision-Language Pre-training

The existing work on vision language pre-training typically falls into two
categories: Detector-based VLP model and CNN/ViT-Based VLP models. Previous
Detector-based VLP methods (Lu et al., 2019; Li et al., 2019; Tan and Bansal,
2019; Li et al., 2020; Chen et al., 2020; Yu et al., 2021) mainly take a two-
step training pipeline approach, which first extracts visual features by a
pre-trained object detector and then trains the cross-modal pre-training model
to align text and visual features. Even though there are some region-based
methods that reduce the computation cost with the lightweight model
architecture (Wang et al., 2020a), those methods still suffer from the
following weaknesses: (1) the expensive computational cost and time
consumption for detecting objects/regions. (2) the error propagation problems
caused by the two-step pre-training strategy. More recently, ViTs-based (Li et
al., 2021; Kim et al., 2021; Radford et al., 2021; Wang et al., 2021a; Li et
al., 2022a, b; Wang et al., 2021c; Kim et al., 2021) methods (especially the
patch-based ViT) removes the complicated object detector in feature extraction
to conduct end-to-end VL learning. These methods avoid the drawbacks of object
detectors but face excessively long visual sequences without fine-grained
cross-modal alignment information. Such long visual sequences also bring
expensive computation costs but there is no work that focuses on decreasing
the high computational cost. In this work, we propose a novel method to detect
the text-relevant patches and reduce the redundant undetected patches to a
single one in the backbone, thus can decrease the computation cost of VLP
models.

###  2.2. ViTs Acceleration

To accelerate the computation of the transformer(Vaswani et al., 2017) based
model, many studies focus on proposing more efficient attention mechanisms
(Wang et al., 2020b; Kitaev et al., 2020; Choromanski et al., 2021) or
compress Transformer structures (Liu et al., 2021; Heo et al., 2021; Wang et
al., 2021b). Recently, some approaches have focused on accelerating ViTs by
reducing the number of tokens involved in the inference of ViTs. For example,
to expedite ViTs, Ryoo et al. (2021) proposed TokenLearner, in which a
relatively small amount of tokens are learned by aggregating the entire
feature map weighted by dynamic attention. Rao et al. (2021) introduces a
method to reduce tokens for a fully trained ViT, where an extra learnable
neural network is added to ViT to select a subset of tokens. Liang et al.
(2022) proposes to reduce the computational overhead of inference by proposing
a token reorganization method to reduce and reorganize image tokens
progressively. However, those methods are unsuitable for VLP as they reduce
the image tokens without considering the text context. There is recent VLP
work (Jiang et al., 2022) has noticed this issue and attempted to reduce the
redundant visual tokens based on coarse-grained cross-modal semantic
alignment. However, due to the absence of fine-grained visual-text
supervision, they struggle to accurately select text-relevant patches, thereby
leading to a degradation in the model performance.

##  3\. Method

In this section, we will first give an overview of our model architecture and
then introduce the Text-aware Patch Detector (TPD) in the Vision Transformer
(ViT) backbone. Finally, we give the details about the pre-training task of
Patch-Text Alignment (PTA).

###  3.1. Model Architecture

As shown in Figure 2(a), COPA contains a visual encoder, a text encoder, a
multimodal fusion encoder for performing cross-modal interaction, and a
multimodal decoder for text generation (Note that we implement our method
based on Li et al. (2022b) which provides details about the model
architecture). The visual encoder named ViT-TPD is a ViT-based network that
consists of multiple Transformer layers and a Text-aware Patch Detector (TPD)
utilized to detect text-relevant patches.

Formally, suppose we have an input image-text pair denoted as
$\left(I,T\right)$. For the input text, we feed it to the text encoder and get
the text representation $T=\\{t_{cls},t_{1},t_{2},\cdots,t_{m}\\}$ where
$t_{cls}$ is the embedding of the text [CLS] token which is used to summarize
the global semantic information of the text. For the input image, we divide
the input image into $n$ non-overlapping patches
$P=\\{p_{cls},p_{1},p_{2},\cdots,p_{n}\\}$. Then we feed the patch sequence to
the visual encoder ViT-TPD and get the patch sequence representation
$V=\\{v_{cls},v_{1},v_{2},\cdots,v_{u}\\},u\textless n$. In the ViT-TPD, we
apply the text-aware patch detector to detect the text-relevant image patches
and fuse other undetected redundant patches to a single token, which can
reduce the visual sequence length for training and inference efficiency. After
that, the image and text representations are fed into the cross-modal encoder
and we get the cross-modal representations
$\\{c_{cls},c_{1},c_{2},\dots,c_{l}\\}$ where $l=u+m$. The cross-modal
representations can be used to finetune downstream multi-modal understanding
tasks. Besides, the output cross-modal representations
$\\{c_{cls},c_{1},c_{2},\dots,c_{l}\\}$ of the multi-modal encoder are fed
into a Transformer decoder for sequence-to-sequence learning.

###  3.2. Text-aware Patch Detection

As shown in Figure 2, our ViT-based visual backbone contains $N$ standard
Transformer layers and a plug-and-play Text-aware Patch Detector (TPD), which
is the only difference from the standard ViT. The TPD dynamically detects the
image patches with the guidance of textual input. Specially, suppose the TPD
is plugged between the $k_{th}$ ( $1\leq k\textless N$ ) Transformer layer and
$(k+1)_{th}$ Transformer layer. Suppose the output patch sequence features of
$k_{th}$ Transformer layer is
$v^{k}=\\{v^{k}_{cls},v^{k}_{1},\cdots,v^{k}_{n}\\}$. We exclude the image
[CLS] tokens $v^{k}_{cls}$ and feed the left patch tokens
$\\{v^{k}_{1},\cdots,v^{k}_{n}\\}$ and the text [CLS] feature $t_{cls}$
together to the TPD. The text [CLS] feature is output by the text encoder and
represents global information of the input text $T$.

In the TPD, we first concatenate the text [CLS] feature with each image patch
token as follows:

$\dot{v}^{k}_{i}=concat(v^{k}_{i},t_{cls})$

where $v^{k}_{i}\in R^{d},t_{cls}\in R^{d},\dot{v}^{k}_{i}\in
R^{2d},i\in\\{1,2,\dots,n\\}$. Then the concatenated patch features
$\\{\dot{v}^{k}_{i}\\}$ are fed to the patch detector. The patch detector is
an MLP that contains three linear layers and is used to predict the alignment
score between patches and the input text T. The first two linear layers will
linearly project the concatenated patch features $\\{\dot{v}^{k}_{i}\\}$ to
the hidden representations $\\{h^{k}_{i}\\}$ and then the hidden
representations $\\{h^{k}_{i}\\}$ is fed to the last linear layer denoted as
$\mathbf{F}_{\theta}$ which can be seen as a classifier to predict whether the
patches are relevant to the input text. The output of the last linear layer
has only one dimension and will be fed to a Sigmoid activation function.
Formally, the alignment score $a_{i}$ between the $i_{th}$ image patch and
input text T can be calculated as follow:

$a_{i}=Sigmoid(\mathbf{F}_{\theta}(h^{k}_{i})),i\in\\{1,2,\dots,n\\}$

Then, we identify and preserve the image tokens with high alignment scores
with the text which corresponds to the $K$ largest elements in the alignment
score sequence $\\{a_{1},..a_{n}\\}$, where $K=n\times\alpha$, and $\alpha$ is
a hyper-parameter and named Keeping Ratio which is used to control the
proportion of detected patches to total patches. The detected top-K image
patch tokens are kept and the undetected patch tokens
$\\{v_{z_{1}},v_{z_{2}},\cdots,v_{z_{n-K}}\\}$ which generally have lower
alignment scores with the text will be treated as text-irrelevant tokens and
further fused by a token fusion operation. We fuse undetected tokens to one
token $v_{f}$ by a weighted sum operation to supplement one as follows:

(1)
$\left[\hat{a}_{z_{1}},\cdot,\hat{a}_{z_{n-k}}\right]=Softmax(\left[{a}_{z_{1}},\cdot,{a}_{z_{n-k}}\right])$
(2) $v_{f}=\sum\limits^{n-k}\limits_{i=1}\hat{a}_{z_{i}}\cdot v_{z_{i}}$

After fusing the undetected patch tokens to single one token $v_{f}$, we
reconstruct the $k_{th}$ visual sequence as
$v^{k}=\left[v^{k}_{cls},v^{k}_{1},\cdots,v^{k}_{u},{v}^{k}_{f}\right]$, which
consists of the image [CLS] token embedding, the detected text-relevant image
patch embeddings, and the fused patch embedding. Then the reduced visual
sequence is fed to the next $(k+1)_{th}$ transformer layer. Such Text-aware
Patch Detector (TPD) works during both pre-training and finetuning, which can
be optimized in the pre-training by the Patch-Text Alignment objectives which
will be introduced in the next.

###  3.3.  Patch-Text Alignment

The key component of COPA is the text-aware Patch Detector which needs to
detect text-relevant patches according to the fine-grained alignment scores
between the image patches and input text. However, such fine-grained patch-
text alignment capabilities of traditional ViT-based models are weak as the
lack of fine-grained patch-text labels. To address the above difficulties, in
this sub-section, we introduce a novel pre-training task named Patch-Text
Alignment which facilitates the patch detector training and drives our model
to learn the fine-grained patch-text alignment.

We find that in most object objection and visual grounding datasets, the
object and region generally be paired with a class label or text description.
Therefore, we can transfer every object class label to a text description
based on a text template such as ”This is a [class label].”. Thus, for each
(object/region) bounding box in an image, we can find a text description.
Then, we transform the bounding box annotations to the patch-level labels by
following this rule: Given an image and a bounding box annotation, if there is
an overlap between an image patch and a bounding box, it will be assigned with
label 1, otherwise, it will be assigned with label 0. For different text
descriptions and bounding boxes, the labels of the patch are different. In
this way, we can generate fine-grained patch-text labels which can be served
as the supervisory signal to pre-train our model.

After that, in each step of pre-training, we randomly sample a mini-batch of
images from the object detection/visual grounding datasets (e.g., COCO (Lin et
al., 2014) or VG (Krishna et al., 2016)). For each image, we randomly select
an object/region bounding box and translate the bounding box annotation to the
image patch label sequence following the transformation rule we mentioned
before. Then, we feed the batch of text descriptions of the bounding boxes and
the images together to our VLP model. In the ViT-TPD backbone, we hope the
text-aware Patch Detector can detect all patches which have overlap with the
region in the bounding box with the guidance of the bounding box text
description. Supposing the Text-aware Patch Detector has predicted the
alignment scores between image patches and text, we will calculate the binary
cross entropy loss between the alignment scores and patch labels as follows:

(3)
$\mathbf{L}_{PTA}=\frac{1}{e}\sum_{i=1}^{e}Y_{i}log\left(a_{i}\right)+\left(1-Y_{i}\right)log\left(1-a_{i}\right)$

where $a_{i}$ is the alignment score between $i_{th}$ patch in the image and
the input text, $Y_{i}$ is the patch label of $i_{th}$ patch. After
calculating the Patch-Text Alignment loss $\mathbf{L}_{PTA}$, we then randomly
sample a mini-batch of normal image-text pairs from the dataset of 4M images
(refer to subsection 4.1) and calculate the Image-Text Contrastive (ITC) loss
$\mathbf{L}_{ITC}$, Image-Text Matching (ITM) loss $\mathbf{L}_{ITM}$, Masked
Language Modeling (MLM) loss $\mathbf{L}_{MLM}$ and Prefix Language Modeling
(PrefixLM) loss $\mathbf{L}_{Prefix}$ based on other four pre-training
objectives (for more details about other pre-training objectives, please refer
to Appendix B). We assign equal loss weights to each pre-training loss, and
thus the full pre-training loss is:

(4)
$\mathbf{L}=\mathbf{L}_{ITC}+\mathbf{L}_{ITM}+\mathbf{L}_{MLM}+\mathbf{L}_{Prefix}+\mathbf{L}_{PTA}$

We also provide the pseudo algorithm 1 to further elaborate on our pre-
training schedule. Besides, at the beginning of pre-training, as the PTA loss
has not yet converged, thus the performance of the patch detector is not
ideal, we detect the image patches directly based on the attention weights of
the image [CLS] token to other patch tokens. As the PTA loss gradually
converges, we will use the patch detector to detect the text-relevant patches
in the TPD module.

1

Input: Large scale pre-training dataset $\mathcal{D}$, Object/Region Dataset
$\mathcal{O}$, the number of pre-training epochs $T$, the pre-training
learning rate $\alpha$, the batch size $B_{D}$ of dataset $\mathcal{D}$, the
batch size $B_{O}$ of dataset $\mathcal{O}$.

2

3Initialize the parameters $\theta$ of our model $M$ ;

4 for  _$t=1$ to $T$_ do

5 Randomly sample a mini-batch of $B_{O}$ Images
$\\{\hat{v}_{1},\hat{v}_{2},\dots,\hat{v}_{B_{O}}\\}$ from $\mathcal{D}$ ;

6 for  _$i=1$ to $B_{O}$_ do

7 Select a object or region $r_{i}$ from image $\hat{v}_{i}$ ;

8 Convert the object class label $\hat{y}_{i}$ to text description
$\hat{t}_{i}$;

9 Convert the bounding box annotation of $r_{i}$ to patch annotations
$Y^{i}=\\{y^{i}_{1},y^{i}_{2},\dots,y^{i}_{n}\\}$ ;

10

11 Run forward of $M$ on the mini-batch of image-text pairs
$\\{\\{\hat{v}_{1},\hat{t}_{1}\\},\\{\hat{v}_{2},\hat{t}_{2}\\},\dots,\\{\hat{v}_{B_{O}},\hat{t}_{B_{O}}\\}\\}$
and $\\{Y^{1},Y^{2},\dots,Y^{B_{O}}\\}$ to obtain the loss $\mathcal{L}_{PTA}$
;

12

13 Randomly sample a mini-batch of $B$ Image-Text Pairs
$\\{\\{v_{1},t_{1}\\},\\{v_{2},t_{3}\\},\ldots,\\{v_{B_{D}},t_{B_{D}}\\}\\}$
from $\mathcal{D}$ ;

14

15 Run forward of $M$ on the mini-batch of image-text pairs
$\\{\\{v_{1},t_{1}\\},\\{v_{2},t_{3}\\},\ldots,\\{v_{B_{D}},t_{B_{D}}\\}\\}$
to obtain the losses $\mathcal{L}_{ITC}$, $\mathcal{L}_{ITM}$,
$\mathcal{L}_{MLM}$, $\mathcal{L}_{Prefix}$ ;

16

17 Calculate the overall loss:

18
$\mathbf{L}=\mathbf{L}_{ITC}+\mathbf{L}_{ITM}+\mathbf{L}_{MLM}+\mathbf{L}_{Prefix}+\mathbf{L}_{PTA}$;

19

20 Backward the overall loss $\mathbf{L}$ and update the parameters of $M$
using gradient descent with learning rate $\alpha$ and the average loss
$\mathbf{L}$ over the mini-batch:

21
$\theta\leftarrow\theta-\alpha\frac{1}{B}\sum_{i=1}^{B}\nabla_{\theta}\mathcal{L}(\theta;s_{i})$
;

22

23

24return $M$ with pre-trained parameters $\theta$ ;

25

Algorithm 1 Pre-training algorithm of COPA .

##  4\. Experiments

###  4.1. Data & Setup

Following the previous work (Li et al., 2021), we use the same pre-training
dataset with 4M images with texts, which includes two in-domain datasets (MS
COCO (Lin et al., 2014) and Visual Genome (Krishna et al., 2016)), and three
web out-of-domain datasets (Conceptual Captions (Sharma et al., 2018), SBU
Captions (Ordonez et al., 2011). See Appendix A for more details on the pre-
training datasets.

We pre-train the model for 30 epochs with a total batch size of 1024 on 16
NVIDIA A100 GPUs. We use a 6-layer Transformer for both the text encoder and
the cross-modal skip-connected network, and a 12-layer Transformer for the
decoder. The text encoder is initialized using the first 6 layers of the
BERTbase (Devlin et al., 2019) model and the skip-connected network is
initialized using the last 6 layers of the BERTbase. Please see Appendix D for
more details of the pre-training setting of our model.

Models | # Pretrain | VQA | COCO Caption | NoCaps
---|---|---|---|---
Data |  | Cross-entropy Optimization | CIDEr Optimization |
Test-dev | Test-std | B@4 | M | C | S | B@4 | M | C | S | C | S
E2E-VLP (Xu et al., 2021) | 4M | 73.25 | 73.67 | 36.2 | - | 117.3 | - | - | - | - | - | - | -
OSCAR (Li et al., 2020) | 6.5M | 73.16 | 73.44 | - | - | - | - | 41.7 | 30.6 | 140.0 | 24.5 | 83.4 | 11.4
VinVL (Zhang et al., 2021) | 5.65M | 76.52 | 76.60 | 38.5 | 30.4 | 130.8 | 23.4 | 41.0 | 31.1 | 140.9 | 25.2 | 97.3 | 13.8
METER (Dou et al., 2021) | 4M | 77.68 | 77.64 | - | - | - | - | - | - | - | - | - | -
BLIP (Li et al., 2022a) | 14M | 77.54 | 77.62 | 38.6 | - | 129.7 | - | - | - | - | - | 105.1 | 14.4
VLMo (Wang et al., 2021a) | 4M | 76.64 | 76.89 | - | - | - | - | - | - | - | - | - | -
ViLBERT (Lu et al., 2019) | 3.3M | 70.63 | 70.92 | - | - | - | - | - | - | - | - | - | -
VisualBERT (Li et al., 2019) | 180K | 70.80 | 71.00 | - | - | - | - | - | - | - | - | - | -
SimVLM (Wang et al., 2021c) | 1.8B | 77.87 | 78.14 | 39.0 | 32.9 | 134.8 | 24.0 | - | - | - | - | - | -
ALBEF (Li et al., 2021) | 14M | 75.84 | 76.04 | - | - | - | - | - | - | - | - | - | -
TRIPS (Jiang et al., 2022) | 4M | 76.23 | 76.48 | - | - | - | - | - | - | - | - | - | -
mPLUG (Li et al., 2022b) | 4M | 77.55 | 77.73 | 39.3 | 30.1 | 132.4 | 23.34 | 41.2 | 30.8 | 140.2 | 25.2 | 98.3 | 12.9
COPA | 4M | 77.84 | 77.91 | 39.5 | 32.8 | 133.8 | 24.12 | 41.5 | 31.0 | 140.4 | 25.1 | 98.9 | 13.1
Table 1. Evaluation Results on VQA, COCO Caption ”Karpathy” test split and NoCaps validation set. B@4: BLEU@4, M: METEOR, C: CIDEr, S: SPICE. More details about comparison models in Appendix LABEL:sup:comparison_models .  Models | # Pretrain | MSCOCO (5K test set) | Flickr30K (1K test set)
---|---|---|---
data | TR | IR | TR | IR
|  | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
ALIGN (Jia et al., 2021) | 1.8B | 77.0 | 93.5 | 96.9 | 59.9 | 83.3 | 89.8 | 95.3 | 99.8 | 100.0 | 84.9 | 97.4 | 98.6
OSCAR (Li et al., 2020) | 4M | 70.0 | 91.1 | 95.5 | 54.0 | 80.8 | 88.5 | - | - | - | - | - | -
E2E-VLP (Xu et al., 2021) | 4M | - | - | - | - | - | - | 86.2 | 97.5 | 98.92 | 73.6 | 92.4 | 96.0
UNITER (Chen et al., 2020) | 4M | 65.7 | 88.6 | 93.8 | 52.9 | 79.9 | 88.0 | 87.3 | 98.0 | 99.2 | 75.6 | 94.1 | 96.8
VLMo (Wang et al., 2021a) | 4M | 78.2 | 94.4 | 97.4 | 60.6 | 84.4 | 91.0 | 95.3 | 99.9 | 100.0 | 84.5 | 97.3 | 98.6
ALBEF (Li et al., 2021) | 14M | 77.6 | 94.3 | 97.2 | 60.7 | 84.3 | 90.5 | 95.9 | 99.8 | 100.0 | 85.6 | 97.5 | 98.9
BLIP (Li et al., 2022a) | 14M | 80.6 | 95.2 | 97.6 | 63.1 | 85.3 | 91.1 | 96.6 | 99.8 | 100.0 | 87.2 | 97.5 | 98.8
TRIPS (Jiang et al., 2022) | 14M | 78.1 | 94.8 | 97.6 | 61.3 | 84.3 | 91.4 | 96.3 | 99.8 | 100.0 | 85.8 | 98.1 | 99.0
mPLUG (Li et al., 2022b) | 4M | 80.2 | 95.1 | 97.7 | 62.5 | 84.8 | 90.9 | 96.4 | 99.8 | 100.0 | 86.5 | 97.5 | 98.8
COPA | 4M | 80.8 | 95.6 | 98.1 | 63.6 | 85.6 | 91.6 | 96.8 | 99.8 | 100.0 | 87.3 | 97.9 | 98.9

Table 2. Image-text retrieval results on Flickr30K and COCO datasets.

###  4.2. Main Result

We evaluate our model COPA on four widely explored vision-language downstream
tasks: Visual Question Answering (VQA), Cross-modal Retrieval, Image Caption,
and Visual Grounding (VG). We plug the Text-aware Patch Detector before the
6-th Transformer layer in the ViT encoder and set the keeping ratio to 50%,
achieving the desired trade-off between the downstream task performance and
the model inference speed. The fine-tuning hyperparameters are described in
Appendix E. Details of the comparison methods are in Appendix
LABEL:sup:comparison_models.

####  4.2.1. Visual Question Answering

The VQA task (Agrawal et al., 2015) requires the model to answer natural
language questions given an image. During fine-tuning and inference of VQA, we
feed the [CLS] token of the question to the TPD to detect the question-
relevant patch tokens (We also give the visualization of the detected tokens
in Figure 4 which indicates the effectiveness and generalization of TPD). We
follow (Li et al., 2021) and consider VQA as an answer-generation problem. We
report test-dev and test-std scores by submitting our results to the
evaluation server111https://eval.ai/web/challenges/challenge-page/830/overview
in Table 1. Compared with the VLP baselines, our COPA can get the better
performance (e.g. 77.84 on VQA test-dev) with SOTAs under the same image
resolution (384 $\times$ 384) and even speed up about 88% of model
inference(see the report results in Table 4 and Table 5). Furthermore, when we
increase the image resolution to 512 $\times$ 512 (as shown in Table 5), we
can achieve better performance (e.g. 78.25 on VQA test-dev. ) while keeping a
similar inference computation cost with the baselines mPLUG (Li et al., 2022b)
(e.g. 65.23 of COPA 512×512 VS 63.57 of mPLUG (Li et al., 2022b) on
Throughput.). The results demonstrate the effectiveness and efficiency of COPA
.

####  4.2.2. Image Captioning

As there is no textual input in the image caption task, we directly detect the
patches based on the vision information where we use the attention weight of
the image [CLS] token to other image tokens as the detection scores and fusion
the image tokens with low attention weight of image [CLS] token. Following (Li
et al., 2020), we first fine-tune COPA with cross-entropy loss and then with
CIDEr optimization (Rennie et al., 2017) for extra 5 epochs. As shown in Table
1, COPA can get comparable or better results with SOTA models on both COCO
Caption (Lin et al., 2014) and Nocaps (Agrawal et al., 2018) datasets.

Model |  RefCOCO+
---|---
val |  testA |  testB
UNITER (Chen et al., 2020) |  75.90 |  81.45 |  66.70
VL-BERT (Su et al., 2020) |  72.59 |  78.57 |  62.30
ViLBERT(Lu et al., 2019) |  72.34 |  78.52 |  62.61
VILLA (Gan et al., 2020) |  76.17 |  81.54 |  66.84
MDETR (Kamath et al., 2021) |  79.52 |  84.09 |  70.62
UNICORN (Yang et al., 2021) |  80.30 |  85.05 |  71.88
mPLUG (Li et al., 2022b) |  80.07 |  85.21 |  71.03
COPA |  80.37 |  86.03 |  71.81
Table 3. Evaluation results of Visual grounding on ReferCOCO+. We use the accuracy of IOU 0.5 on visual grounding (a prediction is right if the IoU between the grounding-truth box and the predicted bounding box is larger than 0.5) Models | VQA | FLOPs | Throughput | Latency
---|---|---|---|---
UNITER (Chen et al., 2020) | 72.70 | 949.90 | 6.42 |  870ms
OSCAR (Li et al., 2020) | 73.16 | 956.40 | 6.35 |  860ms
VinVL (Zhang et al., 2021) | 76.52 | 1023.30 | 7.32 |  640ms
E2E-VLP (Xu et al., 2021) | 73.25 | 144.3 | 80.23 |  70ms
ViLT (Kim et al., 2021) | 71.26 | 55.40 | 247.530 |  19ms
ALBEF (Li et al., 2021) | 74.54 | 33.42 | 197.52 |  22ms
TRIPS (Jiang et al., 2022) | 76.23 | 20.89 | 343.05 | 11ms
mPLUG (Li et al., 2022b) | 77.55 | 36.63 | 186.42 |  24ms
COPA | 77.84 | 19.84 | 349.71 |  10ms
Table 4. The comparison of the efficiency of different models. Here, we report
the VQA test-dev result and FLOPs, throughput, and latency. The FLOPs results
of the baselines come from (Kim et al., 2021). Since FLOPs are proportional to
input size, for a fair comparison, we keep same the input size with (Kim et
al., 2021), which is 197 for image patches (the image resolution is $224\times
224$ ) length and 40 for text tokens length. We keep the same setting when
calculating throughput and latency.

####  4.2.3. Image-Text Retrieval

We conduct experiments for both image-to-text retrieval (TR) and text-to-image
retrieval (IR) on MSCOCO (Lin et al., 2014) and Flickr30K (Plummer et al.,
2015) datasets. We jointly optimize the ITC loss and the ITM loss during fine-
tuning. The results are reported in Table 2. As shown in Table 2, the
experimental results show that our model gets comparable performance with
other VLP baselines. For more details, please refer to the Appendix E

####  4.2.4. Visual Grounding

Following the setting of mPLUG (Li et al., 2022b), we also evaluate COPA on
the visual grounding task, which requires models to localize the referred
object in the image based on a given text query. In this task, we feed the
text query to the TPD to detect the query-relevant patch tokens, then, instead
of directly regressing the bounding boxes, we concatenate detected patch
features and attended textual features and feed them into the multi-modal
decoder to predict the coordinates. Table 3 demonstrates the performance of
COPA in the visual grounding task. COPA achieves comparable results with
competitive baseline methods.

Detection Location |  Keeping ratio |  image size |  VQA test-dev |  FLOPs(G) |  Throughput
---|---|---|---|---|---
- |  - |  $384\times 384$ |  77.55 |  84.872 |  63.57
[6] |  50% |  $224\times 224$ |  76.83 |  19.84 |  349.71
[6] |  50% |  $256\times 256$ |  77.11 |  23.04 |  303.03
[6] |  50% |  $304\times 304$ |  77.32 |  32.97 |  247.62
[6] |  50% |  $384\times 384$ |  77.84 |  47.56 |  144.38
[6] |  50% |  $464\times 464$ |  78.03 |  75.99 |  81.02
[6] |  50% |  $512\times 512$ |  78.25 |  83.24 |  65.23
Table 5. Results of COPA finetuning on VQA task with different resolution
images. The Settings for calculating FLOPs and throughput are the same as
Table 4 except for the image resolution. The first row in the table reports
the result of our baseline mPLUG

###  4.3. Efficiency of Text-aware Patch Detection

(a) Keeping Ratio - GPU MEM (b) Detection Location - GPU MEM

Figure 3. The single GPU Memory cost of pre-training of COPA with different
keeping ratios and detection locations of TPD. We set the batch size to 512,
the image size to 256 and the text length to 25. The red line in sub-figure
(a) is the GPU memory cost of the baseline model mPLUG (Li et al., 2022b)

To assess the efficiency of the Text-aware Patch Detection mechanism, we first
compare the computational complexity of various models. We report the Floating
Point Operations Per Second (FLOPs), a widely used evaluation metric for model
computational complexity. Additionally, to evaluate the computational speed of
our model, we compare the throughput and latency of different models. We use a
Xeon Platinum 8163 CPU and an NVIDIA V100 GPU to measure latency and
throughput.

As illustrated in Table 4, COPA exhibits not only the lowest computational
complexity (e.g., 19.84 FLOPs) but also the fastest computational speed (e.g.,
349.71 throughput and 13ms latency). Moreover, as shown in Figure 3, we
evaluate the single GPU memory cost under varying keeping ratios and detection
location, which refers to the position where TPD is integrated into the ViT
backbone. For example, Detection Location=6 indicates that TPD is inserted
before the 6th transformer layer in the ViT backbone. The results demonstrate
that Text-aware Patch Detection significantly reduces GPU memory usage
compared to the baseline model. We also observe that the keeping ratio
influences GPU memory consumption, while the detection location of TPD does
not impact it.

###  4.4.  The Impact of Detection Location and Keeping Ratio

To investigate the influence of detection location in the ViT backbone and
keeping ratio on the efficiency and effectiveness of COPA , we train COPA
using different detection locations and keeping ratios. Note that when
calculating FLOPs and throughput, we set the input image size to $224\times
224$ and input text length to 40. As depicted in Table 9, two main conclusions
can be drawn:

First, incorporating the Text-aware Patch Detector (TPD) before shallower
layers can reduce computational complexity but at the cost of accuracy. For
instance, when TPD is placed before the 4th layer, there is a significant
increase in throughput, but the accuracy drops considerably. A possible
explanation is that the patch embedding in shallow layers may not adequately
represent visual semantics, making it challenging to learn the fine-grained
patch-text alignment and subsequently leading to a decline in accuracy.

Second, introducing too many undetected image tokens into the TPD module can
significantly impair downstream task performance. For example, when
positioning the TPD module before the 4th layer in ViT and setting the keeping
ratio to 10%, the performance on the VQA task decreases to 76.48, compared to
77.84 achieved by the model with a 50% keeping ratio in the 6th layer.

###  4.5. Finetuning on Higher Resolution Images

We can regulate the computational cost by fusing different numbers of
inattentive tokens. To this end, we fine-tune COPA on the VQA task, using
images with varying resolutions as input. The results are reported in Table 5.
The experimental findings indicate that by increasing the input image
resolution, the model benefits from processing more image tokens, resulting in
improved performance. For instance, by fine-tuning COPA with 512$\times$512
resolution images, we can achieve a score of 78.25 on VQA, surpassing the
baseline fine-tuned with 384$\times$384 images while maintaining a similar
computational complexity.

###  4.6. Ablation Study

####  4.6.1. Effectiveness of Text-aware Patch Detector

We also perform ablation studies to investigate the effects of our proposed
pre-training task Patch-Text Alignment (PTA) and Text-aware Patch Detector
(TPD). In Table 10, w/o PTA indicates we remove the PTA task but keep the TPD
in the visual backbone. However, without the PTA task, the TPD can not be
optimized and thus is ineffective, therefore, we replace the TPD with a simple
strategy in which we directly detect the patch in a transformer layer based on
the self-attention weights of image [CLS] token to other patch tokens. As
shown in Table 10, we find that without the text guidance, w/o PTA, which
directly detects patches based on visual information, will get a significant
accuracy drop on VQA compared with the baseline model mPLUG. On the contrary,
COPA detects the text-relevant patches based on the TPD and can even get a
slight improvement compared with the baseline model mPLUG, which indicates the
effectiveness of TPD and PTA.

####  4.6.2. Effectiveness of Patch-Text Alignment

w/o TPD means we remove the TPD but keep the PTA pre-training task, compared
with the baseline model mPLUG (Li et al., 2022b) (w/o TPD & w/o TPA), we find
that even though the efficiency of the model can not be improved compared with
the baseline, we can get a remarkable improvement on VQA. This experiment
results not only shows the efficiency of the text-aware Patch Detection
mechanism but also indicate the effectiveness of PTA, which enables our model
to learn fine-grained cross-modal alignment, thus leading to improvement in
VQA task.

###  4.7. Extension to Single-stream Model

Models | EL | KR | FLOPs(G) | Throughput | VQA
---|---|---|---|---|---
Test-dev
COPA -S | 6 | 50% | 26.23 | 436.74 | 71.51
ViLT | - | - | 55.40 | 247.53 | 71.26
COPA | 6 | 50% | 19.84 | 349.71 | 77.84
mPLUG | - | - | 36.63 | 186.42 | 77.55
Table 6. The evaluate results of COPA , COPA -S and their baselines ViLT and
mPLUG on VQA test-dev. The setting for calculating FLOPs and throughput is the
same as Table 4. KR refers to Keeping Ratio, EL refers to Detection Location.

The proposed Text-aware Patch Detection mechanism can also be extended to
single-stream models by incorporating the TPD into the multimodal encoder and
pre-training the model with the Patch-Text Alignment task. To verify the
effectiveness of TPD and Patch-Text Alignment task in single-stream models, we
first implement a single-stream model ( COPA -S) based on the ViLT (Kim et
al., 2021) framework, which employs a visual transformer as the cross-modal
encoder. Next, we insert the TPD before the 6th layer of the cross-modal
encoder and pre-train it with the PTA task (For the single-stream model, the
TPD can be directly integrated into the cross-modal encoder and detect patches
based on the overall [CLS] token’s attention value.). We then evaluate the
downstream task performance, computational complexity, and inference speed of
COPA and COPA -S (both with and without Text-aware Patch Detection). The
results are shown in Table 6, and we observe consistent improvements in
inference speed and downstream task performance for both COPA and COPA -S when
incorporating Text-aware Patch Detection and Patch-Text Alignment. These
results indicate that the proposed image patch selection mechanism is not only
efficient but also effective. Notably, compared to the dual-stream model COPA
, COPA -S has faster inference due to the parameter efficiency of the single-
stream model. However, its performance falls short of state-of-the-art
performance on downstream VL tasks.

###  4.8. Case Study

Figure 4. The visualization of the VQA case and the detected text-relevant
image patches. We set the detection location to 6.

The proposed Text-aware Patch Detector (TPD) identifies text-consistent image
tokens in the vision backbone and retains the detected image patches. To
further investigate the effectiveness of the TPD, we visualize the VQA case
and the detected text-relevant image patches in Figure 4. It is evident that
based on the text questions, the TPD module can effectively detect relevant
patches, and even when preserving only 10% of the detected patches, our model
can still produce correct answers. For instance, in the first case, the
question is ”Are they having breakfast?”, and the TPD effectively detects
patches of food and the girl’s mouth in the image, which are highly relevant
to the question. In the second case, the question is ”What color is the
suitcase?”, indicating that the red suitcase in the image is text-relevant,
while other visual objects like the black cat are text-irrelevant. As
illustrated in Figure 4, the TPD effectively detects text-relevant patches,
which help our model predict the correct answer. It is worth noting that these
examples are not cherry-picked, and this phenomenon is commonly observed in
other samples.

##  5\. Conclusion

We have presented COPA , an efficient and effective VLP model that
successfully learns fine-grained patch-text alignment and reduces lengthy
visual sequences for streamlined training. Specifically, we devise a Patch-
Text Alignment pre-training task (PTA) based on a Text-aware Patch Detector
(TPD). The TPD is incorporated into the ViT backbone to identify text-relevant
patches and eliminate redundant ones. PTA allows our model to learn fine-
grained patch-text alignment end-to-end by jointly optimizing with other pre-
training tasks. Experiments demonstrate that our method enhances efficiency
through the reduction of visual sequences while maintaining or even improving
the performance of downstream tasks.

## References

  * (1)
  * Agrawal et al. (2015) Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. 2015.  VQA: Visual Question Answering.  _International Journal of Computer Vision_ 123 (2015), 4–31.
  * Agrawal et al. (2018) Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2018\.  nocaps: novel object captioning at scale.  _CoRR_ abs/1812.08658 (2018).  arXiv:1812.08658 http://arxiv.org/abs/1812.08658
  * Bi et al. (2020) Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2020.  Palm: Pre-training an autoencoding&autoregressive language model for context-conditioned generation.  _arXiv preprint arXiv:2004.07159_ (2020).
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020\.  UNITER: UNiversal Image-TExt Representation Learning. In _ECCV_.
  * Choromanski et al. (2021) Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy J. Colwell, and Adrian Weller. 2021\.  Rethinking Attention with Performers.  _ArXiv_ abs/2009.14794 (2021).
  * Cubuk et al. (2020) Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. 2020\.  Randaugment: Practical automated data augmentation with a reduced search space. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. 702–703.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _ArXiv_ abs/1810.04805 (2019).
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  _ArXiv_ abs/2010.11929 (2021).
  * Dou et al. (2021) Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Zicheng Liu, Michael Zeng, et al. 2021\.  An Empirical Study of Training End-to-End Vision-and-Language Transformers.  _arXiv preprint arXiv:2111.02387_ (2021).
  * Gan et al. (2020) Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. 2020\.  Large-Scale Adversarial Training for Vision-and-Language Representation Learning. In _NeurIPS_.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6904–6913.
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. 2017.  Mask R-CNN.  _2017 IEEE International Conference on Computer Vision (ICCV)_ (2017), 2980–2988.
  * Heo et al. (2021) Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. 2021\.  Rethinking Spatial Dimensions of Vision Transformers.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ (2021), 11916–11925.
  * Huang et al. (2020) Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. 2020.  Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers.  _ArXiv_ abs/2004.00849 (2020).
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. 2021\.  Scaling up visual and vision-language representation learning with noisy text supervision.  _arXiv preprint arXiv:2102.05918_ (2021).
  * Jiang et al. (2022) Chaoya Jiang, Haiyang Xu, Chenliang Li, Ming Yan, Wei Ye, Shikun Zhang, Bin Bi, and Songfang Huang. 2022\.  TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection. In _Conference on Empirical Methods in Natural Language Processing_.
  * Kamath et al. (2021) Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. 2021\.  MDETR-modulated detection for end-to-end multi-modal understanding. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 1780–1790.
  * Karpathy and Fei-Fei (2015) Andrej Karpathy and Li Fei-Fei. 2015.  Deep visual-semantic alignments for generating image descriptions. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3128–3137.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision. In _ICML_.
  * Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020\.  Reformer: The Efficient Transformer.  _ArXiv_ abs/2001.04451 (2020).
  * Krishna et al. (2016) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2016\.  Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations.  _International Journal of Computer Vision_ 123 (2016), 32–73.
  * Li et al. (2022b) Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, and Luo Si. 2022b.  mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections.
  * Li et al. (2022a) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022a.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  _arXiv preprint arXiv:2201.12086_ (2022).
  * Li et al. (2021) Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. 2021.  Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. In _NeurIPS_.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019.  VisualBERT: A Simple and Performant Baseline for Vision and Language.  _ArXiv_ abs/1908.03557 (2019).
  * Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020\.  Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. In _ECCV_.
  * Liang et al. (2022) Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. 2022\.  Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations.  _ArXiv_ abs/2202.07800 (2022).
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft COCO: Common Objects in Context. In _ECCV_.
  * Liu et al. (2023) Siyi Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chun yue Li, Jianwei Yang, Hang Su, Jun-Juan Zhu, and Lei Zhang. 2023\.  Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.  _ArXiv_ abs/2303.05499 (2023).
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ (2021), 9992–10002.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled Weight Decay Regularization. In _ICLR_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019\.  ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks.  _NeurIPS_ (2019).
  * Ordonez et al. (2011) Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 2011\.  Im2Text: Describing Images Using 1 Million Captioned Photographs. In _NIPS_.
  * Plummer et al. (2015) Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, J. Hockenmaier, and Svetlana Lazebnik. 2015.  Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models.  _International Journal of Computer Vision_ 123 (2015), 74–93.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021\.  Learning Transferable Visual Models From Natural Language Supervision. In _ICML_.
  * Rao et al. (2021) Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021\.  DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification. In _NeurIPS_.
  * Redmon et al. (2016) Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. 2016.  You Only Look Once: Unified, Real-Time Object Detection.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2016), 779–788.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015\.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 39 (2015), 1137–1149.
  * Rennie et al. (2017) Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017.  Self-Critical Sequence Training for Image Captioning. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 1179–1195.  https://doi.org/10.1109/CVPR.2017.131
  * Ryoo et al. (2021) Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021\.  TokenLearner: Adaptive Space-Time Tokenization for Videos. In _NeurIPS_.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018\.  Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In _Annual Meeting of the Association for Computational Linguistics_.
  * Singh et al. (2021) Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2021.  FLAVA: A Foundational Language And Vision Alignment Model.  _ArXiv_ abs/2112.04482 (2021).
  * Su et al. (2020) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020\.  VL-BERT: Pre-training of Generic Visual-Linguistic Representations.  _ArXiv_ abs/1908.08530 (2020).
  * Tan and Bansal (2019) Hao Hao Tan and Mohit Bansal. 2019.  LXMERT: Learning Cross-Modality Encoder Representations from Transformers.  _arXiv preprint arXiv:1908.07490_ (2019).
  * Vaswani et al. (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need.  _ArXiv_ abs/1706.03762 (2017).
  * Wang et al. (2020a) Jianfeng Wang, Xiaowei Hu, Pengchuan Zhang, Xiujun Li, Lijuan Wang, L. Zhang, Jianfeng Gao, and Zicheng Liu. 2020a.  MiniVLM: A Smaller and Faster Vision-Language Model.  _ArXiv_ abs/2012.06946 (2020).
  * Wang et al. (2020b) Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020b.  Linformer: Self-Attention with Linear Complexity.  _ArXiv_ abs/2006.04768 (2020).
  * Wang et al. (2021a) Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. 2021a.  VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts.  _ArXiv_ abs/2111.02358 (2021).
  * Wang et al. (2021b) Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. 2021b.  Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ (2021), 548–558.
  * Wang et al. (2021c) Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021c.  SimVLM: Simple Visual Language Model Pretraining with Weak Supervision.  _ArXiv_ abs/2108.10904 (2021).
  * Xu et al. (2021) Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang, Wenming Xiao, and Fei Huang. 2021.  E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning.  _ArXiv_ abs/2106.01804 (2021).
  * Yang et al. (2021) Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2021\.  Crossing the Format Boundary of Text and Boxes: Towards Unified Vision-Language Modeling.  _CoRR_ abs/2111.12085 (2021).  arXiv:2111.12085 https://arxiv.org/abs/2111.12085
  * Yu et al. (2021) Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021.  ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph. In _AAAI_.
  * Yu et al. (2016) Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016.  Modeling context in referring expressions. In _European Conference on Computer Vision_. Springer, 69–85.
  * Zeng et al. (2021) Yan Zeng, Xinsong Zhang, and Hang Li. 2021.  Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts.  _ArXiv_ abs/2111.08276 (2021).
  * Zhang et al. (2021) P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao. 2021.  VinVL: Making Visual Representations Matter in Vision-Language Models.  (2021).
  * Zhou et al. (2020) Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. 2020\.  Unified Vision-Language Pre-Training for Image Captioning and VQA.  _ArXiv_ abs/1909.11059 (2020).

##  Appendix A Datasets

Following the previous work (Li et al., 2021), we use the same pre-training
dataset with 4M images with texts, which includes two in-domain datasets (MS
COCO (Lin et al., 2014) and Visual Genome (Krishna et al., 2016)), and three
web out-domain datasets (Conceptual Captions (Sharma et al., 2018), SBU
Captions (Ordonez et al., 2011).

| COCO | VG | SBU | CC3M
---|---|---|---|---
image | 113K | 100K | 860K | 3M
text | 567K | 769K | 860K | 3M
Table 7. Statistics of the pre-training datasets. | image | Captions | Objects | Regions
---|---|---|---|---
COCO | 0.11M | 0.55M | 0.45M | -
VG | 0.10M | - | 2.0M | 3.7M
Table 8. Statistics of objects/regions annotations used in the pre-training.

Table 7 shows the statistics of the 4M images with texts used in the pre-
training stage. Besides, As shown in table 8 we use also use the
objects/regions annotations from COCO(Lin et al., 2014) and VG (Krishna et
al., 2016) datasets, and we give statistics of object and region annotations
of each dataset. Note that we use the object/region annotations provided by
Zeng et al. (2021) thus we follow their setting which filtered out some
samples because of: 1) invalid annotations (e.g. negative values for bounding
boxes or boxes being outside of the images); 2) boxes being too small (¡ 1%);
3) highly overlapped textual descriptions of regions (¿75%), etc. After pre-
processing, we keep COCO objects 446,873 (from 859,999), VG objects 2,043,927
(from 3,802,349), VG regions 3,699,598 (from 5,402,953).

##  Appendix B Pre-training Objectives

We pre-train our model with five standard objectives: Patch-Text Alignment
(PTA), Image-Text Contrastive learning (ITC), Image-Text Matching (ITM),
Masked Language Modeling (MLM), Prefix Language Modeling (PrefixLM). These
pre-training tasks are optimized jointly. As we have talked about the Patch-
Text Alignment before, in this subsection, we only introduce the last four
pre-training tasks.

Image-text Contrastive (ITC) For COPA , We follow the (Li et al., 2021) and
apply ITC to align the image representation and text representation from the
unimodal encoders. For the image, the image feature corresponding to the image
[CLS] token is chosen as the image representation. For the text, the text
token feature corresponding to the text [CLS] token is the text
representation.

Image-Text Matching (ITM) The goal of image-text matching is to predict
whether the input image and text are matched. We follow the design of (Li et
al., 2021) and select hard negative image-text pairs based on the contrastive
text-image similarity. We take the text [CLS] embedding of the multimodal
encoder’s output as the joint representation, followed by a Multi-Layer
Perceptron (MLP) layer for prediction.

Masked Language Modeling (MLM) The task setup is basically the same as in BERT
(Devlin et al., 2019), where we randomly mask 15$\%$ of tokens in text, and
the model is asked to predict these masked words with the cross-modal
representations.

Prefix Language Modeling (PrefixLM). This task aims to generate the caption
given an image and predict the text segment subsequent to the cross-modal
context as (Bi et al., 2020). It optimizes a cross-entropy loss by maximizing
the likelihood of text in an autoregressive manner.

##  Appendix C More Experiment Results.

###  C.1. Generalization of Text-aware Patch Detector

(a) ACC of TPD (b) Recall of TPD

Figure 5. The visualization of Accuracy and Recall of TPD on the 10K test
dataset randomly sampled from CC (Sharma et al., 2018).

The Text-aware Patch Detector (TPD) is pre-trained based on the Patch-Text
Alignment task, utilizing object/region annotations from the COCO (Lin et al.,
2014) and VG (Krishna et al., 2016) datasets as supervised signals. Our
subsequent experiments demonstrate that using these two relatively small-scale
datasets alone is sufficient to train a robust and generalizable TPD without
requiring additional object/region annotations while maintaining the
scalability of web-based pre-training data.

In detail, we sample 10K image-text pairs from CC (Sharma et al., 2018)
dataset, which is crawled from the web and potentially contains out-of-domain
data. We then employ the state-of-the-art open-set object detection model,
Grounding DINO (Liu et al., 2023), to detect regions in each image
corresponding to the text. Following the approach described in subsection 3.3,
we convert these regions into patch-level labels. We subsequently evaluate the
accuracy and recall of TPD in detecting patches on this test dataset.

As depicted in Figure 5, the accuracy and recall of TPD’s patch predictions
gradually improve with each training epoch. By the 30th epoch, TPD’s accuracy
reaches 0.87, and its recall approaches 0.74. These results suggest that TPD
is effective and robust in detecting text-related image patches, regardless of
whether the image-text pairs originate from manual annotations or are crawled
from the internet. This showcases TPD’s remarkable generalizability and
effectiveness, allowing us to confidently scale up the size of the web-based
pre-training dataset without compromising TPD’s ability to handle out-of-
domain data from the web.

Location | Keeping ratio | VQA | FLOPs (G) | Throughput
---|---|---|---|---
[4] | 10% | 76.48 | 10.15 | 514.26
[4] | 30% | 77.02 | 14.84 | 443.48
[4] | 50% | 77.33 | 16.62 | 363.65
[4] | 70% | 77.52 | 23.32 | 337.45
[6] | 10% | 76.89 | 16.51 | 418.95
[6] | 30% | 77.53 | 17.84 | 361.05
[6] | 50% | 77.84 | 19.84 | 349.71
[6] | 70% | 77.89 | 26.77 | 306.11
[8] | 10% | 76.93 | 22.45 | 275.13
[8] | 30% | 77.62 | 23.36 | 310.56
[8] | 50% | 77.89 | 25.66 | 273.06
[8] | 70% | 77.93 | 30.22 | 221.88
- | 100% | 77.98 | 36.63 | 186.42
Table 9. Results of pre-training and finetuning COPA with different locations and keeping ratios. We report the text-dev score results of VQA, FLOPs, and Throughput. The throughput (image-text/s) is measured on an NVIDIA V100 GPU using the largest possible batch size for our model.  model | VQA | FLOPs(G) | Throughput
---|---|---|---
COPA | 77.84 | 19.84 | 349.71
-w/o PTA | 77.12 | 19.43 | 352.56
-w/o TPD | 77.98 | 36.63 | 186.42
mPLUG | 77.55 | 36.63 | 186.42
Table 10. The result of ablations. We finetune COPA on VQA and report test-dev
results. The setting for calculating FLOPs and throughput is the same as Table
4. For -w/o PTA, we keep the same setting with COPA (detection location = 6
and keeping ratio = 50%).

##  Appendix D Pre-training Details

We use the AdamW (Loshchilov and Hutter, 2019) optimizer with a weight decay
of 0.02. The learning rate is warmed-up to 1e-5 (ViT-B/16) and 1e-4 (BERTbase)
in the first 1000 iterations, and decayed to 1e-6 following a cosine schedule.
COPA is pre-trained for about 20 epochs with 8*A100-80G GPUs on the 4M pre-
training dataset for 41 hours. During pre-training, the batch size on a single
GPU is 512, and the overall batch size is 1024.

During pre-training, we take random image crops of resolution 256 $\times$ 256
as input and also apply RandAugment (Cubuk et al., 2020) to improve the
generalization of vision encoders. For VQA and image captioning tasks, we
increase the image resolution during finetuning. For image-text contrastive
learning, the queue size is set as 65,536, and the momentum coefficient is set
as 0.995.

##  Appendix E Downstream Task Details

We evaluate COPA on the four downstream vision-language tasks. The
hyperparameters that we use for finetuning on the downstream tasks are listed
in Table 11. Following (Li et al., 2021), all tasks adopt RandAugment, AdamW
optimizer with a weight decay of 0.05 and a cosine learning rate schedule.
Next, we introduce the dataset settings in detail.

##### VQA.

The VQA task (Agrawal et al., 2015) requires the model to answer natural
language questions given an image. Most methods (Tan and Bansal, 2019; Wang et
al., 2021a; Li et al., 2020; Wang et al., 2021c) deal with visual question-
answering tasks as multi-label classification on pre-defined answer sets. This
strategy achieves strong performance, but it is not suitable for real-world
open scenarios. We conduct an experiment on the VQA2.0 dataset (Goyal et al.,
2017), which contains 83k/41k/81k images for training/validation/test.
Following (Li et al., 2021), we use both training and validation splits for
training, and incorporate additional training data from Visual Genome (Krishna
et al., 2016). Besides, we concatenate the question with the object labels and
OCR tokens extracted from the image.

Task | LR (ViT-L/BERTbase) | batch size | epochs
---|---|---|---
VQA | 2e-5/5e-6 | 1024 | 8
Captioning$\dagger$ | 1e-5&8e-7 | 256 | 5
Retrieval | 1e-5/2e-6 | 256 | 5
Visual Grounding | 2e-5/2e-6 | 512 | 120
Table 11. Finetuning hyperparameters for downstream tasks. $\dagger$ denotes
two-stage fine-tuning.

##### Image Captioning.

The image captioning task requires a model to generate an appropriate and
fluent caption for a given image. We evaluate image captioning on two datasets
COCO Caption (Lin et al., 2014) and NoCaps (Agrawal et al., 2018). COPA
finetuned with training data of COCO Caption is tested on both of the
datasets. We train COPA on the MS COCO Caption and test on the same Karpathy
split (Li et al., 2020; Wang et al., 2021c) and NoCaps validation set.
Following (Li et al., 2020), we first fine-tune COPA with the cross-entropy
loss for 5 epochs with a learning rate of 1e-5 and a batch size of 256. Based
on the fine-tuned model, we then fine-tune it with CIDEr optimization (Rennie
et al., 2017) for extra 5 epochs with a smaller learning rate of 8e-7. We use
the best checkpoint on COCO Caption and predict on the Nocaps validation set
directly. During inference, we use beam search with a beam size of 10 and set
the maximum generation length as 20.

##### Image-Text Retrieval.

We conduct experiments for both image-to-text retrieval (TR) and text-to-image
retrieval (IR) on COCO (Lin et al., 2014) and Flickr30K (Plummer et al., 2015)
datasets. We adopt the widely-used Karpathy split (Karpathy and Fei-Fei, 2015)
for both COCO and Flickr30K. COCO contains 113k/5k/5k images for
train/validation/test, and Flickr30K contains 29k/1k/1k images for
train/validation/test. Following (Li et al., 2021, 2022a), we jointly optimize
the ITC loss and the ITM loss during fine-tuning. During inference, we first
select top-k candidates by computing the dot-product similarity between the
image and text encoder features (When extracting the image encoder feature,
for efficiency of coarse-grained ranking, we replace the TPD with a simple
strategy in which we directly detect the patch in a transformer layer based on
the self-attention weights of image [CLS] token to other patch tokens ), and
then rerank the selected candidates based on their ITM scores (In the fine-
grained reranking stage, for the same image, we re-extracting multiple image
encoder features based on TPD with the guidance of multiple text candidates.).
We set $k=256$ for COCO and $k=128$ for Flickr30K.

##### Visual Grounding.

The task of visual grounding involves localizing the referred object in an
image given a plain text query. Instead of directly regressing bounding boxes,
our approach concatenates visual features with textual features, which are
then fed into the multi-modal decoder to predict the object’s coordinates. We
evaluate our method on the referring expression grounding dataset: RefCOCO+(Yu
et al., 2016). The RefCOCO+ dataset contains 19K images and 141K queries.