# Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product
Retrieval

Xiao Dong*, Xunlin Zhan*, Yunchao Wei, Xiaoyong Wei, Yaowei Wang, Minlong Lu,
Xiaochun Cao, and Xiaodan Liang, X. Dong is with the School of Artificial
Intelligence, Zhuhai Campus, Sun Yat-Sen University, Zhuhai, P.R. China,
519082. X.L. Zhan and X.D. Liang are with the School of Intelligent Systems
Engineering, Shenzhen Campus, Sun Yat-Sen University, Shenzhen, P.R. China,
528406. Y.C. Wei is with the School of Computer and Information Technology,
Beijing Jiaotong University, Beijing, P.R. China, 100044. X.Y. Wei and Y.W.
Wang are with PengCheng Laboratory, Shenzhen, P.R. China, 322099. M.L. Lu is
with Alibaba Group, Hangzhou, P.R. China, 322099. X.C. Cao is with School of
Cyber Science and Technology, Shenzhen Campus, Sun Yat-sen University,
Shenzhen, P.R. China, 518107.
E-mail<EMAIL_ADDRESS>dongx55@mails2.sysu.edu.cn);
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>wangyw<EMAIL_ADDRESS>caoxch5<EMAIL_ADDRESS>* indicates equal first author.
Xiaodan Liang is the Corresponding Author.

###### Abstract

Our goal in this research is to study a more realistic environment in which we
can conduct weakly-supervised multi-modal instance-level product retrieval for
fine-grained product categories. We first contribute the Product1M datasets,
and define two real practical instance-level retrieval tasks to enable the
evaluations on the price comparison and personalized recommendations. For both
instance-level tasks, how to accurately pinpoint the product target mentioned
in the visual-linguistic data and effectively decrease the influence of
irrelevant contents is quite challenging. To address this, we exploit to train
a more effective cross-modal pertaining model which is adaptively capable of
incorporating key concept information from the multi-modal data, by using an
entity graph whose node and edge respectively denote the entity and the
similarity relation between entities. Specifically, a novel Entity-Graph
Enhanced Cross-Modal Pretraining (EGE-CMP) model is proposed for instance-
level commodity retrieval, that explicitly injects entity knowledge in both
node-based and subgraph-based ways into the multi-modal networks via a self-
supervised hybrid-stream transformer, which could reduce the confusion between
different object contents, thereby effectively guiding the network to focus on
entities with real semantic. Experimental results well verify the efficacy and
generalizability of our EGE-CMP, outperforming several SOTA cross-modal
baselines like CLIP [1], UNITER [2] and CAPTURE [3].

###### Index Terms:

Multi-modal pre-training, instance-level retrieval, knowledge injection,
product dataset.

##  1 Introduction

E-commerce application as one of the largest multi-modal downstream scenarios
has facilitated person’s lives. And there are many product-based tasks in the
application scenario, such as similar item retrieval [4], online commodity
recommendation [5] and identical product match for cross-price comparison [6].
With a wide variety of downstream E-commerce applications in the real world,
we focus solely on the pretraining of multi-modality data of products. In the
general scenarios, multi-modal vision-language pre-training model [7, 8, 9, 3,
2, 1, 10, 11, 12] such as CLIP [1] and VilBert [7] have shown superior
performances on the diverse downstream tasks, such as zero-shot classification
[13], cross-modal retrieval [14] and open-world detection [15]. They learn
visual and textual embedding features from a huge number of image-text pairs
acquired from different sources and have quite generalization capability and
robustness. The fundamental methodology behind these models is either
contrastive semantic alignment of images and texts or specific network
architectures designed for better common space learning. However, these models
mentioned above focus more on the attention regions adaptively learned by
themselves and are easily influenced by the contents that are not related to
the visual objects from each modality, leading to ignoring the importance of
concept knowledge, i.e., words or phrases with a certain meaning. As an
illustration in Figure 1, we expect the trained multi-modal network to pay
more attention to four typical objects (Fair Water, Facial Cream, Clearner and
Clear Lotion) and to ignore extra content (Four piece Set) in a more fine-
grained application setting. If the multi-modal network is trained directly by
visual-language contrastive learning, the learned attention region may be SK-
II, Cream Cleaner, Lotion Four-piece, which is unsuitable and incorrect to
achieve the expected objects of attention. In the past few years, there has
been increasing attention in designing the knowledge injection [16, 17, 18,
19] for natural language processing (NLP) tasks to demonstrate and enchance
the attention effectiveness of NLP models. These models are more based on a
well fully annotated knowledge graph, which is time-cost and laborious for a
new task. In this paper, we balance the trade-off between knowledge base
construction and knowledge extraction, and propose a simple but effective
entity-graph enhanced visual-language pretraining model, which explicitly
inject the concept knowledge generated from the caption data into the multi-
modal model via a special knowledge graph (entity-graph).

Figure 1:  The task we propose aims to retrieve instance-level products in
multi-modal data.

Next, We first investigate a realistic problem: how to perform instance-level
111Instance-level product retrieval refers to the retrieval of all single
products that existed in a product portfolio image. fine-grained product
retrieval given the weakly annotated multi-modality data on a vast scale? In
Figure 1, we give a comparison of different retrieval patterns to better show
the challenge. From the figure, we could find that image-level retrieval tends
to match retrieval objects from all aspects since it is not capable of
discriminating between distinct instances, but multi-modal instance-level
retrieval is much more capable of searching for different types of commodities
among multi multimodality data. Although the challenge has generality and
practical importance, it is under-examined enough owing to a lack of high-
quality datasets fitting the real scenarios and a clear description of the
issue. According to relevant studies of product retrieval, uni-modal [20, 21,
22, 23] and cross-modal retrieval [24, 25, 26, 27, 28, 29] take as input
unimodal information, e.g., a visual feature or a textual description, and
performs a matching search between individual samples. Unfortunately, such
retrieval strategies severely limit their applicability in many scenarios
where both the queries and the targets include multi-modal information. More
importantly, past researches concentrate on comparatively straightforward
scenario, i.e., image-level 222Image-level product retrieval refers to
recognizing a specific product instance in a single-product image. retrieval
for single-product images [30, 31], leaving the instance-level aspect of
retrieval untouched.

Figure 2: Five categories of visual-language pretraining models. The two left
are the typical network structures. The three right are the knowledge
injecting versions. Compared with (c) and (d), our structure (e) builds the
entity graph directly from the caption without additional annotation
information. In the figure, CTL, LPM, MRP, ENL, ESL denote the ContrasTive
Learning, Link Prediction Modeling, Masked Relation Prediction, Entity Node
Loss and Entity Subgraph Loss, respectively.

In the paper, we propose a large-scale Entity-Graph Enhanced Cross-Modal
Pretraining framework (EGE-CMP) to address these challenges. Inspired by the
effectiveness of knowledge injection [16, 17, 32, 33, 34] into large-scale
pretrained models for single modality, we model the cross-modal semantic
alignment under the supervision of an entity-graph instead of directly using
image and text interaction, which makes the trained model more focused on
meaningful entity objects and thus reduce the influence of irrelevant content.
Specifically, we extract the noun entity directly from the caption data via
named entity recognition (NER) tools and obtain the image proposals from the
image data via the bottom-up-attention feature representation model [35].
Then, EGE-CMP is presented to capture the potential synergy of images,
entities and title descriptions via entity-graph learning and some task-
agnostic masking training. We showcase that some widely used cross-modal
pretraining methods [7, 9, 2, 10] might be ineffective under the multi-
instance setting without external knowledge guidance. In contrast, EGE-CMP
utilizes a hybrid-stream architecture that encodes data of different
modalities separately and fuses them in a unified manner, which has been shown
to be advantageous for our suggested purposed tasks. Additionally, we
introduce the cross-modal contrastive loss to compel EGE-CMP to align images,
texts, and entities, so avoid the mismatch problem introduced by the
unsuitable pretext task. Besides that, an effective entity-graph enhanced
module is adopted to enhance the learning capacity for the whole transformer
model by enforcing the model to pay more attention to some entities with real
semantic information. In our entity graph, the node is each entity data and
the edge is generated via the relation between each entity. To encourage the
text-encoder to be aware of actual key entities and improve the understanding
ability of the transformer model, node-based and subgraph-based ranking losses
are also introduced for the model training, requiring it to accurately
distinguish the actual positive samples of entities according to the global
entity ranking list. As a result, our EGE-CMP model can focus more on the
phrasesor words with real semantic meaning, and the text-image correlation can
be aligned better. Please note that compared with existing knowledge-enhanced
pretraining methods based on the knowledge graph, our EGE-CMP model directly
utilizes the knowledge directly extracted from the original caption data
without any time-consuming and labor-intensive annotated data.

Furthermore, to bridge this gap and enhance the relevant research, we
construct a large-scale dataset from the popular shopping website, named
Product1M, proposed for the performance verification of multi-modal instance-
level retrieval in the real-world scenarios. Product1M has about one million
image-caption pairings and is divided into two sorts of samples, i.e., single-
and multi-product samples. The dataset is one of the biggest multimodal
datasets available and is the first to be created exclusively for multimodal
instance-level retrieval in real-world settings. Two real-world tasks, multi-
product retrieval and identical-product retrieval, are defined in section 3.1
and used for the model comparisons. In summary, as a large benchmark dataset,
our Product1M with ground-truth annotations generated by several human workers
such that the performance of the evaluated multi-product and identical-product
retrieval algorithms can be better analyzed.

The results of experiments on both multi-product retrieval and identical-
product retrieval tasks show the superiority of our EGE-CMP over the SOTA
cross-modal baselines, such as ViLBERT [7], CLIP [1], UNITER [2], CAPTURE [3]
and so on, on all major criteria by a large margin. Moreover, extensive
ablation experiments are conducted to demonstrate the generalizability of EGE-
CMP and investigate various essential factors of our proposed task. Finally,
the quantitative and qualitative results intuitively prove the entity-graph
constructed by caption itself can enhance the representation capability of the
transformer models.

The main contributions of our paper are threefold.

  * •

We conduct Product1M, one of the largest multi-modal product datasets for the
real-world instance-level retrieval task, and contribute a subset extracted
from Product1M for identical product retrieval for price comparison. Different
from the existing E-commerce datasets, our Product1M dataset has real-world
testing circumstances and abundant categories, which can help advance future
research in the high-level difficulties.

  * •

We devise a novel Entity-Graph Enhanced Cross-Modal Pretraining (EGE-CMP)
333https://github.com/Xiaodongsuper/Entity-Graph-Enhanced-Cross-Modal-
Pretraining-for-Instance-level-Product-Retrieval framework to learn the
instance-level feature representations via injecting the entity knowledge with
real semantic information to the visual-text alignment. We find our entity-
graph significantly benefits the text-image cross-modal alignment.

  * •

Experimental results on both our proposed instance-level retrieval tasks,
i.e., multi-product retrieval and same product search demonstrate that the
model representation can be effectively augmented by the proposed entity
graph. Furthermore, both the quantitative and visualized research results
intuitively prove the efficacy of our EGE-CMP framework.

The study builds on our prior conference presentation, CAPTURE [3]. In the
version, we add the following features to the conference version: (1) To be
suggested, a new goal, identical-product retrieval, must be adequately
specified. (2) Additionally, a novel Cross-Modal Pretraining framework called
EGE-CMP is presented to integrate entity information into visual-linguistic
alignment. We demonstrate the superiority of the suggested model EGE-CMP over
many state-of-the-art approaches. Additionally, we show feature embeddings, do
ablation investigations, and perform generalization analysis.

The rest of the paper is divided into the following sections: Section 2
discusses related works. Section 3 describes the definition of multi-product
and identical product retrieval tasks and then introduces our model, including
its network architectures and formulation. Section 4 exhibits and explains the
experimental findings. Finally, we make a conclusion in Section 5.

Figure 3: Characteristics and statistics of Product1M: (1a) Complex
combinations of single products; (1b) Weak supervision and fuzzy
correspondence; (1c) Difficulties in real-world settings; (2) Long-tailed
category distribution of Product1M. The line represents the sample size of
each category in decreasing order. Product1M comprises a diverse set of
categories and the long-tailed class distribution corresponds to real-world
cases.

##  2 Related Work

###  2.1 Visual-Language Datasets.

According to the collection source, visual-language pre-training datasets can
be categorized into at least two main groups: 1) General Sources.: There are
several visual-language datasets [36, 37, 38, 39, 40, 41] collected from the
social media platforms (e.g. Twitter and Facebook). Twitter100k [36] is
presented for cross-media retrieval with a low degree of supervision. It is a
collection of 100k image-text pairings crawled from the Twitter website. There
are no limitations on the categories of images, and users supply the textual
description in a colloquial language. Similarly, this dataset emphasizes
image-level retrieval and employs single-modal input as the query, which is
incompatible with instance-level multi-modal retrieval. INRIA-Websearch [41]
is a cross-modal retrieval dataset that contains 71,478 image-text pairs
connected with 353 unique search queries, such as actors, films, and so on.
The images are gathered via internet searches, and the textual description is
derived from the text that surrounds the images on websites. In this dataset,
cross-modal retrieval methods are used to tackle the text-to-image search
issue. 2) Product Sources. For the E-commerce datasets, RPC checkout [42] and
Dress Retrieval [43] are crawled from the shopping website as two widely used
typical datasets. RPC [42] comprises 53,739 photos of individual products and
30,000 photographs of checkout. Three independent levels of annotation are
available: category information, point-level annotations, and bounding boxes.
There is, however, a gap between RPC and real-world settings, since RPC was
gathered in a controlled environment and this dataset is devoid of textual
information. Dress Retrieval proposed in [43] collects obtains noisy labeled
data scraped from catalogs of E-commerce websites.

As indicated before, our Product1M dataset is an instance-level dataset.
Unlike the previous datasets, each picture in the Product1M dataset is
connected with a subset of relevant attributes that have been filtered using
specified terminology. Typically, each picture comprises a single instance,
such as a model wearing a garment, with a relatively uncluttered backdrop. In
summary, our Product1M is significantly different from the aforementioned
datasets in three ways: (1) Product1M is the first dataset designed
specifically for multi-product and identical product instance-level retrieval
tasks, both of which have significant promise in the e-commerce business. (2)
Our Product1M enables multi-modal retrieval in addition to conventional intra-
and cross-modal retrieval, when both the query and the target include pictures
and text. This is a more realistic environment, given the pervasiveness of
multimodal information in real-world scenarios. (3) The weakly annotated
examples of Product1M compel the model to uncover relevant features in the
absence of clear labels, endowing the model with a high capacity for
generalization to large-scale and noisy data.

###  2.2 Intra- and Cross-Modal Retrieval.

Intra-modal retrieval [20, 21] has been intensively investigated in the
contexts of keyword-based web document retrieval [44] and content-based image
retrieval [45]. On the other hand, cross-modal retrieval[24, 25, 26, 27, 28,
29] emerges as a prospective route for efficiently indexing and searching vast
amounts of data with multiple modalities, and is extensively utilized in
search engines [46, 47] and E-commerce [48, 43], to mention a few. However,
these techniques [23, 49, 43, 42, 50] are often limited to unimodal inputs,
making them inapplicable to a large number of real-world circumstances non
which both the queries and targets include multi-modal information.

###  2.3 WSOD: Weakly Supervised Object Detection.

By learning from cheaper or publicly accessible data, WSOD [51, 52, 53]
decreases its over-reliance on fine-grained labeling. PCL [51] builds proposal
clusters iteratively to aid in the construction of instance classifiers.
Pseudo labels elicited from picture labels [52] and unstructured textual
descriptions such as captions [53] are also useful for improving WSOD
performance. WSOD, on the other hand, often relies on a fixed-size collection
of predefined classes and is thus unsuitable for our proposed task, since
class labels are not available and categories may be changed dynamically.

###  2.4 Visual-Linguistic Pretraining.

The development of the vision-language pretraining model [54, 55, 56, 57, 58,
59, 60, 61, 62, 63, 64, 65, 66, 67] has advanced tremendously, with joint
representations often constructed using a multimodal fusion transformer
network topology. Existing pre-trained models for vision-language often learn
image-text semantic alignment using a multi-layer Transformer architecture,
such as Bert [68], on multi-modal input in a shared cross-modal space. They
may be loosely divided into two categories based on their network structural
differences: 1) Single-stream models [9, 8, 2, 56, 54, 55, 60] in a unified
architecture encode the integrated multi-modal characteristics. 2) Two-stream
models [7, 10, 57, 58, 59, 61, 67, 66, 65], on the other hand, use distinct
encoders for inputs with varying modalities. These approaches are not
optimized for instance-level retrieval, and we demonstrate how they may be
defective as a result of network architecture design flaws and incorrect
pretext tasks. Additionally, since real-world semantic information cannot be
effectively learned during the network training phase, performance increases
are restricted.

###  2.5 Knowledge-Enhanced Modeling.

In the language models, the knowledge-based model aims to incorporate
knowledge, concepts and relation into the pretrained models [68, 16], which
proved to be beneficial to language understanding [17]. The existing methods
can be coarsely broadly classified into two types: implicit knowledge modeling
and explicit knowledge injection. The former attempts on implicit knowledge
modeling usually consist of entity-level masked modeling [32, 18, 19], entity-
based replacement prediction [69], knowledge embedding loss as regularization
[34] and universal knowledge-text prediction [33, 70]. In contrast to implicit
knowledge modeling, the latter separately maintains a group of parameters for
representing structural knowledge. Such methods [32, 18, 69, 33, 70, 19]
usually require a heterogeneous information fusion component to fuse multiple
types of features obtained from the text and knowledge graphs. In our setting,
it is very hard to build a knowledge graph with good annotations from scratch
to fit the multi-modal data. In the paper, with the purpose of knowledge
modeling and visual-language pretraining, and improving the representation
capability of visual-text models, we explore the solutions for knowledge
enhanced cross-modal pretraining, where knowledge is represented as an entity
parsed from the original caption data and the relationship between different
pieces of knowledge are learned via node-based and subgraph-based ranking
losses.

Figure 2 shows five typical vision-and-language models. According to whether
external knowledge is utilized, these models can be generally fall into the
following categories: 1) models without knowledge supervision that fall under
Figure 2 (a) and (b), 2) knowledge-based models that fall under Figure 2 (c),
(d) and (e). Our proposed model EGE-CMP belongs to the second type. Meanwhile,
different from the existing models as seen in Figure 2 (c) and (d), our model
directly use the entity knowledge extracted from the caption rather than human
labeling, (e.g. property and knowledge graph) which extremely save the labor
and calculation cost. Our proposed EGE-CMP is the first model of type Figure 2
(e) where the entity knowledge drawn from the caption data is explicitly
injected into the model via graph learning.

##  3 Instance-Level Retrieval on Product1M

In this section, we begin by describing the two instance-level tasks, followed
by our proposed EGE-CMP model. The task definition is given in Section 3.1.
The dataset analysis is presented in Section 3.2. The details of our proposed
Entity-Graph Enhanced Cross-Modal Pretraining (EGE-CMP), including model
architecture, masked relation prediction, cross-modal contrastive learning,
and the model inference, are elaborated in Section 3.3.

###  3.1 Task Definition

####  3.1.1 Multi-Product Retrieval

A product sample $(I,C)$ is an image-text pair where $I$ is the product image
and $C$ is the caption. Given the gallery set of single-product samples
$\mathcal{S}=\\{\mathcal{S}_{i}|\mathcal{S}_{i}=(I_{\mathcal{S}}^{i},C_{\mathcal{S}}^{i})\\}$
and the set of multi-product samples
$\mathcal{P}=\\{\mathcal{P}_{i}|\mathcal{P}_{i}=(I_{\mathcal{P}}^{i},C_{\mathcal{P}}^{i})\\}$,
the task is to retrieve and rank the single-products that appear in the query
sample $\mathcal{P}_{i}$, i.e., to predict a list
$RETR^{i}=[id^{i}_{1},id^{i}_{2},\cdots,id^{i}_{k},\cdots,id^{i}_{N}]~{}~{}\forall\mathcal{P}_{i}\in\mathcal{P}$,
where $id^{i}_{k}$ corresponds to a specific single-product sample in
$\mathcal{S}$.

####  3.1.2 Identical-Product Retrieval

In the task, we do not distinguish between single-products and multi-product.
Similar to the multi-product retrieval task, given the gallery set of product
samples
$\mathcal{S}=\\{\mathcal{S}_{i}|\mathcal{S}_{i}=(I_{\mathcal{S}}^{i},C_{\mathcal{S}}^{i})\\}$
and the set of product samples
$\mathcal{P}=\\{\mathcal{P}_{i}|\mathcal{P}_{i}=(I_{\mathcal{P}}^{i},C_{\mathcal{P}}^{i})\\}$,
the task is to retrieve and rank the products that appear in the query sample
$\mathcal{P}_{i}$, i.e., to predict a list
$RETR^{i}=[id^{i}_{1},id^{i}_{2},\cdots,id^{i}_{k},\cdots,id^{i}_{N}]~{}~{}\forall\mathcal{P}_{i}\in\mathcal{P}$,
where $id^{i}_{k}$ corresponds to a same product sample with differnet angles,
views, colors and backgrounds in $\mathcal{S}$. The task focuses on the higher
level of fine-grained match compared with traditional fine-grained retrieval
tasks.

###  3.2 Dataset Analysis

####  3.2.1 Dataset Statistics

We gather several product samples from E-commerce websites for 49 different
brands. Following that, these image-text samples are sorted manually into
single- and multi-product categories based on the product information
associated with them.

Dataset | #samples | #categories | #instances | #obj/img | weak supervision | multi-modal | instance-level retrieval | multi-task
---|---|---|---|---|---|---|---|---
Twitter100k [36] | 100,000 | - | - | - | ✓ | ✓ | - | -
Visual Genome [37] | 108,000 | - | - | - | ✓ | ✓ | - | -
Flickr30K [38] | 31,000 | - | - | - | ✓ | ✓ | - | -
SBU [39] | 890,000 | - | - | - | ✓ | ✓ | - | -
NUS-WIDE [40] | 269,648 | 81 | - | - | ✓ | ✓ | - | -
INRIA-Websearch [41] | 71,478 | 353 | - | - | ✓ | ✓ | - | -
RPC checkout [42] | 30,000 | 200 | 367,935 | 12.26 | - | - | - | -
Dress Retrieval [43] | 20,200 | - |  $\sim$20,200 |  $\sim$1.0 | ✓ | ✓ | - | -
Product1M(Ours) | 1,182,083 | 458 | 92,200 | 2.83 | ✓ | ✓ | ✓ | ✓
TABLE I: Comparisons between different datasets. ‘-’ indicates inapplicable.
The #instances and #obj/img of Product1M are in italics since there are no
instance labels for the train set and we only count the instances in the val
and test set. Product1M is one of the largest multi-modal datasets and the
first to be specifically designed for instance-level retrieval in real-world
applications.

For the multi-product retrieval problem, Product1M is divided into the train,
val, test, and gallery sets. The train set has 1,132,830 samples, including
both single- and multi-product samples, while the val and test sets contain
only multi-product examples, totaling 2,673 and 6,547 samples, respectively.
The gallery set contains 40,033 single-product samples in 458 categories, 392
of which occur in the val and test sets, while the others serve as
interference objects for validating the robustness of a retrieval technique.
The gallery, val, and test sets are annotated with class labels for evaluation
purposes only; they are not used in the training process, and the train set
samples are not annotated. Table I and Figure 3. summarize statistics of the
data for Product1M.

For the identical-product retrieval task, we extract a subset from the our
Product1M to construct train, test and gallery sets. The wholse subset totally
contains 208,228 samples with 819 unique products including mixed single-
product and multi-product samples. The train, gallery and test set
respectively contain 192,164, 14,628 and 1,436 samples. All test samples are
annotated by at least 5 crowdsourced workers, and each worker label a pair of
data, one is query data and another is gallery data, to determine whether both
samples are the same products. As depicted in Figure 4, identical product
retrieval is more fine-grained compared with the traditional similarity
retrieval task.

Figure 4: Several comparison demos of identical-prouduct retrieval and
similarity product retrieval. Accurate/Inaccurate matched results are framed
in green/red boxes.

####  3.2.2 Dataset Characteristics

Multi-product nature and complex combinations: Multi-product photos are
prevalent on E-commerce websites and serve as the query images for product
retrieval at the instance level.

As seen in Figure 3(1a), products may be structured in a variety of ways and
layouts, with a high number of instances. Due to the abundance and variety of
fine-grained single-product samples, complex combinations appear in various
portfolio photos.

Weak supervision and fuzzy correspondence: We investigate retrieving data in
two common modalities, namely pictures and text. In comparison to comparable
datasets with precise class names, the oversight provided by commodity
captions is inadequate and often uninformative. In Figure 3(1b), we illustrate
many sorts of difficult samples. Various samples’ captions include
abbreviations, i.e., a reduced version of several goods. However, an acronym
such as ’eight-piece set’ has no information about the items. The second sort
of sample contains irrelevant data, with the commodities specified in the
title appearing in the picture but not in the title, or vice versa. The vast
dispersion of imprecise connection between pictures and titles complicates
instance-level retrieval even more.

Consistency with real-world scenarios: In Figure 3(1c), we provide several
difficult examples. They may feature a complicated backdrop with irrelevant
elements, amorphous watermarks, or substantial clutter that obscures the
product information. Certain items from various categories may seem
practically identical save for the packaging description, e.g., day cream
versus night cream. As seen in Figure 3(2a,2b), the long-tailed distribution
of Product1M fits nicely with real-world settings.

Figure 5: An overview of our Entity-Graph Enhanced Cross-Modal Pretraining
(EGE-CMP) model. First, three kinds of data are fed into the EGE-CMP model,
where entity data is parsed by NER tools (middle-bottom). Then, we employ a
cross-modal contrastive loss to force modality data to come closer with their
positive samples and away from the negative samples (right). After cross-
transformer and CO-transformer, we utilize the AdaGAE [71] graph network to
construct an entity-graph and propose node-based and subgraph-based ranking
losses for making the multi-modal network pay more attention to the meaningful
entity and reducing the ambiguity between different entities (left).

###  3.3 Methodology

Figure 5 depicts the general architecture of our end-to-end visual-language
model (EGE-CMP). In the network architecture, EGE-CMP contains three kinds of
transformer modules: single-modal transformer (SMT), cross-modal transformer
(CMT) and co-modal transformer (COMT) from bottom to top. Meanwhile, according
to the function modules, our model consists of three modules: Masked Relation
Learning, Cross-modal Contrastive Learning and Entity-graph Enhanced Learning.
We begin this section by elaborating on the architectural design of EGE-CMP in
Section 3.3.1 including detailed functions of three kinds of transformer
modules and the direction of data flow. Then we describe three main function
modules that enables the self-supervised learning of EGE-CMP in Section 3.3.2,
Section 3.3.3 and Section 3.3.4. Finally, in Section 3.3.5, we show the
inference procedure for instance-level retrieval.

####  3.3.1 Architectural Design of EGE-CMP

As mentioned above, our EGE-CMP contains SMT, CMT and COMT three kinds of
transformer structures. EGE-CMP takes an image-text pair $(I,C)$ as input
where $I$ is the product image and $C$ is the caption data. Image $I$ are
first processed by bottom-up-attention model [35] to obtain the proposal
features $p=\\{p_{1},p_{2},..,p_{m}\\}$ where objects are most likely to be
detected and their corresponding position information
$q=\\{q_{1},q_{2},..,q_{m}\\}$. Caption $C$ are parsed by NER analysis tools
to obtain the noun entities $E_{C}=\\{E_{1},E_{2},...\\}$. Based on the above
preparation, SMT extracts the embeddings $v,t,e$ for three different inputs
(image, caption and entity). Then CMT performs cross-modal fusion respectively
on $v$ and $t$ by exchanging key-value pairs to generate the new
representations $h_{v}$ and $h_{t_{1}}$. Similarly, we utilize the same
operation to produce the new representations $h_{e}$ and $h_{t_{2}}$. To keep
the representation consistent, we take the average of $h_{t_{1}}$ and
$h_{t_{2}}$ to obtain the new representations of text $h_{t}$. Finally,
$h_{v}$, $h_{t}$ and $h_{e}$, $h_{t}$ are separately concatenated in the
feature dimension and fed into two COMTs to produce joint image-text-entity
representations $f$ for multi-product retrieval and identical-product
retrieval tasks. We give the detailed explanations of each transformer
structure in below.

Single-Modal Transformer (SMT). For a image input $I$, the position
information $q$ are aligned with the same dimension as the proposed feature
$p$ using a linear projection layer and added with $p$ as the input to obtain
the preliminary image embeddings $u_{v}$:

$u_{v}=\mathrm{SMT}(p+\mathrm{LinearProj}(q))$ (1)

For a caption input $C$ or entity input $E$, it is processed by WordPiece and
tokenized into word tokens ${w_{i}}_{i=1}^{L}$, where $L$ is the length of
tokens in caption $C$ or entity $E$. Our SMT embeds the discrete word token
$w_{i}$ into a continue word embedding $u_{t}$ and $u_{e}$:

$\displaystyle u_{t}=\mathrm{SMT}(w_{t})$ (2) $\displaystyle
u_{e}=\mathrm{SMT}(w_{e})$

where $w_{t}$ and $w_{e}$ are word tokens of caption input $C$ and entity
input $E$.

Cross-Modal Transformer (CMT). By exchanging key-value pairs in the multi-
headed attention mechanism, the Cross-Transformer is utilized to construct the
inter-modal relations between the two distinct modalities. We separately learn
the new embeddings $h_{t_{1}}$ and $h_{v}$ between image and text modalities.
With the same way, the new embeddings $h_{t_{2}}$ and $h_{e}$ are generated by
CMT. Due to the downstream tasks only accessing image and caption data, we
select the $h_{t_{1}}$ as the output $h_{t}$ to serve the COMT module. The
detailed process can be shown as following.

$\displaystyle h_{t_{1}},h_{v}=\mathrm{CMT}(u_{t},u_{v})$ (3) $\displaystyle
h_{t_{2}},h_{e}=\mathrm{CMT}(u_{t},u_{e})$ $\displaystyle
h_{t}=(h_{t_{1}}+h_{t_{2}})/2$

CO-Modal Transformer (COMT). Given image features $h_{v}$, caption featuress
$h_{t}$ and concatenated entity features $h_{e}$, COMT performs cross-modal
fusion over all $\\{h_{v_{i}}\\}_{i=1}^{T}$, $\\{h_{t_{i}}\\}_{i=1}^{L}$ and
$\\{h_{e_{i}}\\}_{i=1}^{L}$ for joint visual-language-entity learning. Because
the caption and entity data could express the semantic information of
different levels, we construct two branches co-transformer to solve the
problem. One branch is to learn the common representation between $h_{v}$ and
$h_{t}$. We add a common learnable position embedding $q_{vt}$ to caption
features $h_{t}$ and image features $h_{v}$, to incorporate sequence ordering
and learn the low-level semantic information. Then, the image and caption
representations after position embedding $q_{vt}$ and following a sepical
$[CLS]$ token are concatenated as the input to the branch of COMT. Similarly,
we also use the same operation to process $h_{t}$ and $h_{e}$. After that, two
group fused presentations $[f_{v},f_{t_{1}}]$ and $[f_{e},f_{t_{2}}]$ are
obtained. To learn the fused text representation, $f_{t_{1}}$ and $f_{t_{2}}$
are concatenated and adopted a linear projection layer to generate the final
output $f_{t}$. The joint features $f$ are learned as:

$\displaystyle
f_{v},f_{t_{1}}=\mathrm{COMT_{1}}([[\mathrm{CLS}],[u_{t},u_{v}]+q_{vt}])$ (4)
$\displaystyle
f_{e},f_{t_{2}}=\mathrm{COMT_{2}}([[\mathrm{CLS}],[u_{t},u_{e}]+q_{et}])$
$\displaystyle f_{t}=\mathrm{LinearProj}([f_{t_{1}},f_{t_{2}}])$
$\displaystyle f=[f_{v},f_{t},f_{e}]$

####  3.3.2 EGE-CMP by Masked Multi-Modal Learning

We utilize several pretext tasks to enable the self-supervised learning of
EGE-CMP and benefit from large-scale multi-modal data. For modality-wise
feature learning, we adopt three masked multi-modal modeling tasks, i.e.,
Masked Language Modeling task (MLM), Masked Region Prediction task (MRP) and
Masked Entity Modeling (MEM) following the standard BERT [68] and VisualBERT
[9]. Masked Language Modeling reconstructs the masked token information to
improve language understanding with the aid of visual and knowledge
representations. Masked Region Prediction recovers the masked visual features
to enhance the visual object perception. Masked Entity Modeling improves the
semantic knowledge understanding, empowering models to make sense of common
concepts with real significance.

Masked Language Modeling (MLM). In the task, approximately 15% of texts and
are masked out and the remaining inputs are used to reconstruct the masked
tokens $x$ from the joint features $h_{t}$ learned from COMT module. The
masked tokens $h_{t}^{m}$ are adopted by a fully-connected (FC) layer
($FC_{MLM}$) to be projected to the discrete pre-defined word space for
classification:

$\displaystyle x_{i}^{\prime}$
$\displaystyle=\mathrm{FC}_{\mathrm{MLM}}\left(h_{t}^{\mathrm{m}}\right),$ (5)
$\displaystyle\mathcal{L}_{\mathrm{MLM}}$
$\displaystyle=-\mathbb{E}\left[\frac{1}{\left|\mathcal{M}^{\mathrm{MLM}}\right|}\sum_{i\in\mathcal{M}^{\mathrm{MLM}}}\log
P\left(x_{i}\mid x_{i}^{\prime}\right)\right],$

where $\mathcal{M}^{\mathrm{MLM}}$ is the index set of masked text tokens.

Masked Region Prediction (MRP). Similar to the MLM, we randomly mask out
proposal inputs $p_{v}^{m}$ with approximately 15%. Different from the
discrete work token, in the task, we directly regress the masked features
$p_{i}$, which is supervised by the features extracted by the bottom-up-
attention model [35] with an MSELoss. The masked proposal inputs $h_{t}^{m}$
are fed into a fully-connected (FC) layer ($FC_{MRM}$) and project to a binary
space for determining whether the input is masked :

$\displaystyle p_{i}^{\prime}$
$\displaystyle=\mathrm{FC}_{\mathrm{MRM}}\left(p_{v}^{\mathrm{m}}\right),$ (6)
$\displaystyle\mathcal{L}_{\mathrm{MRM}}$
$\displaystyle=\frac{1}{2\left|\mathcal{M}^{\mathrm{MRM}}\right|}\sum_{i\in\mathcal{M}^{\mathrm{MRM}}}\left|p_{i}^{\prime}-p_{i}^{t}\right|^{2},$

where $\mathcal{M}^{\mathrm{MRM}}$ is the index set of masked proposal inputs,
and $p_{i}$ and $p_{i}^{t}$ are the predicted results and ground-truth.

Masked Entity Modeling (MEM). In MEM, about 15% of entity tokens are masked
out and the remaining entity inputs are used to reconstruct the masked entity
$x_{e}$ from the joint features $h_{e}$ learned from COMT module. The masked
tokens $h_{e}^{m}$ are adopted by a fully-connected (FC) layer ($FC_{MEM}$) to
be projected to the discrete pre-defined word space for classification:

$\displaystyle x_{e_{i}}^{\prime}$
$\displaystyle=\mathrm{FC}_{\mathrm{MEM}}\left(h_{e}^{\mathrm{m}}\right),$ (7)
$\displaystyle\mathcal{L}_{\mathrm{MEM}}$
$\displaystyle=-\mathbb{E}\left[\frac{1}{\left|\mathcal{M}^{\mathrm{MEM}}\right|}\sum_{i\in\mathcal{M}^{\mathrm{MEM}}}\log
P\left(x_{e_{i}}\mid x_{e}^{\prime}\right)\right],$

where $\mathcal{M}^{\mathrm{MEM}}$ is the index set of masked entity tokens.

####  3.3.3 EGE-CMP by Cross-Modal Contrastive Loss

Apart from intra-modal feature learning, EGE-CMP is enable to construct
coherent representations of multi-modal inputs and learn their alignment. To
do this, we leverage inter-modality contrastive learning [72, 1] to align the
visual and textual modalities. There are $2N$ data points in total for a
minibatch of $N$ image-text samples. We consider the related image-text
pairings to be $N$ positive pairs and the remaining $2(N-1)$ mismatched pairs
to be negative pairs. Formally, given an image-text pair $(x_{i},x_{j})$ and
their encoded features $(\tilde{x}_{i},\tilde{x}_{j})$, the cross-modal
contrastive loss for this positive pair is computed as:

$\mathcal{L}_{\mathrm{VT}}(x_{i},x_{j})=-\log\frac{\exp\left(\operatorname{sim}\left(\tilde{x}_{i},\tilde{x}_{j}\right)/\tau\right)}{\sum_{k=1}^{2N}\mathbbm{1}_{[k\neq
i]}\exp\left(\operatorname{sim}\left(\tilde{x}_{i},\tilde{x}_{k}\right)/\tau\right)},$
(8)

where
$\operatorname{sim}(\boldsymbol{u},\boldsymbol{v})=\boldsymbol{u}^{\top}\boldsymbol{v}/\|\boldsymbol{u}\|\|\boldsymbol{v}\|$
computes the cosine similarity of $(\boldsymbol{u},\boldsymbol{v})$ pairs,
$\tau$ denotes the temperature parameter, $\mathbbm{1}_{[k\neq i]}$ is a
binary indicator function that returns 1 iff $k\neq i$.

This kind of contrastive loss promotes similarity between the encoded
characteristics of positive pairings from various modalities while
differentiating between the encoded properties of negative pairs.

To make the text encoder implicitly injected with abundant semantic
information. Single entity data are stacked as the input to be aligned with
corresponding caption data. The cross-modal constrastive learning between a
pair of entity-caption data $(e_{i},x_{j})$ is represented as:

$\mathcal{L}_{\mathrm{ET}}(e_{i},x_{j})=-\log\frac{\exp\left(\operatorname{sim}\left(\tilde{e}_{i},\tilde{x}_{j}\right)/\tau\right)}{\sum_{k=1}^{2N}\mathbbm{1}_{[k\neq
i]}\exp\left(\operatorname{sim}\left(\tilde{e}_{i},\tilde{x}_{k}\right)/\tau\right)},$
(9)

####  3.3.4 EGE-CMP by Entity-Graph Enhanced Learning

Figure 6: An overview of the EGE module. The module has three steps: 1) Entity
object extraction, 2) Entity dictionary construction, 3) Entity relation
cultivation: Node-based and Subgraph-based.

To improve the generalization capability and entity discrimination of our
framework, we develop an effective entity-graph enhanced (EGE) module, which
can explicitly inject the entity knowledge into the model training process.
Given a caption $C$ with a set of entities $E_{C}=\\{E_{1},E_{2},...\\}$
directly extracted from caption data themselves, EGE aims to learn a semantic
graph to connect each entity data for more accurate instance-level retrieval.
In the EGE module, a unique entity queue $l$ is first initialized. Then each
entity in the queue is encoded by our COMT module to obtain the joint
embeddings $f_{e_{i}}$, and fed to the AdaGAE network [71] that is an
effective graph clustering network for learning better graph semantic
relationships $f_{g_{i}}$ between each other. Finally, we respectively propose
node-based ranking loss and subgraph-based ranking loss to make each entity
pay more attention to its semantically close neighbors. The process of the
module is shown in Figure 6.

Inspired by the memory bank of contrastive learning, we also keep a queue $l$
in our training process for obtaining more accurate negative sample selection.
The queue is initialized with an entity dictionary that is the set of all
entity data. Hence the length of the queue is equal to the number of unique
entities of our dataset.

To update the queue $l$, we feed all entities into the COMT module to obtain
the feature embeddings $l_{f}$. Then based on the embeddings $l_{f}$, we
utilize AdaGAE [71] to learn the graph embedding $l_{g}$ by integrating
relationships with different nodes for better node representations. In our
training process, the cosine similarity $S_{g}$ of $l_{g}$ is to determine
which entity is negative sample. In the netwok training process, for a
minibatch of data, we perform negative sample selection from this small batch
of data based on similarity $S_{g}$. For any entity $h_{e_{i}}$ in a minibatch
of data with $l_{m}$ length, we randomly select one $h_{e_{k}}$ of the last
$k$ entity sample as the negative sample. The node-based ranking loss is
represented as:

$\mathcal{L}_{\mathrm{node}}(h_{e_{i}},h_{e_{k}},y)=\mathcal{\max}(0,-y*(h_{e_{i}}-h_{e_{k}})+\mathrm{margin})$
(10)

where $y$ equal to 1 and margin is to set 0.

Meanwhile, we also propose the sub-graph based loss for keeping the
relationships of nearest neighbor graphs for both positive and negative data
points. In our paper, the feature embedding of any sub-graph is represented
with a global-add pooling via aggregating information of their $k$ nearest
neighbor node. For the positive and negative nearest neighbor subgraph
$h_{g_{i}}$ and $h_{g_{k}}$ for the node $h_{e_{i}}$, we have the following
formulation

$\mathcal{L}_{\mathrm{subgraph}}(h_{g_{i}},h_{g_{k}},y)=\mathcal{\max}(0,-y*(h_{g_{i}}-h_{g_{k}})+\mathrm{margin})$
(11)

where $y$ equal to 1 and margin is to set 0.

####  3.3.5 Inference for Instance-Level Retrieval

For both the single- and multi-product samples, the proposal-level features
are extracted via the bottom-up-attention network [35] pre-trained on the
Visual Genome dataset [37], the captions are used as input and the entity is
to set the same values to EGE-CMP. The COMT layer produces $f_{v}$ and $f_{t}$
as the feature embeddings of the visual and textual inputs, respectively,
during inference. Then, $f_{v}$ and $f_{t}$ are concatenated and normalized to
get the instance’s joint representations. Additionally, since the Text/Visual
and Text/Entity Transformers are supervised using cross-modal contrastive
loss, we could believe that concatenating the feature embeddings of COMTs for
retrieval is useful. Our retrieval model then uses the generated features as
input. After generating the cosine similarity matrix between an instance and
the $gallery$ set samples, the single-product samples with the greatest
similarity are selected and returned for each query for both multi-product and
identical-product retrieval tasks.

##  4 Experiments

###  4.1 Implementation Details

We directly utilize the bottom-up-attention model pre-trained [35] on the VG
dataset [37] to extract proposal-wise features for the visual data and employ
the BERT [68] model to initialize the linguistic transformer of our EGE-CMP.
We limit the number of input areas to between 10 and 36 by picking regions
with predicted confidence greater than a predefined threshold as in [7]. The
region’s features are then flattened and input into EGE-CMP using RoIAlign.
The number of the Text/Visual/Entity Transformer, Text/Visual and Text/entity
Cross-Transformer, and Co-Transformer is set to $L=4,K=4,$ and $H=4$,
respectively, for a total of 12 transformer layers. We set the hidden state
size of EGE-CMP and other baselines to 768 for a fair comparison. The
transformer blocks in EGE-CMP have hidden state size of 768 with 8 attention
heads. Meanwhile, we separately attach a 512 dimensional fully connected layer
after Co-Transformer and Text/Visual/Entity Transformer for task-agnostic
masked learning and cross-modal contrastive learning. By concatenating the
features from the final Co-transformer layer, a 1024 dimensional feature
vector is obtained for retrieval, which is also the same for other baselines.
The maximum sequence length for the sentence is set to 36. Our EGE-CMP model
is trained with a total batch size of 128 for 10 epochs on 8 RTX 3090 GPUs.
And an Adam [73] optimizer is used with an initial learning rate of 1e-4 and a
linear learning rate decay schedule is adopted in the EGE-CMP training
process. Temperature parameter $\tau$ is set to 0.07. The entity numbers are
limited to 12,000 for faster training speed. EGE-CMP generates instance
features from texts and proposal-wise features during inference. Before
retrieval, the instance features of the image/text are all normalized and then
concatenated as the overall representations. During retrieval, we compute
cosine similarities between each box query with single-product samples using
the overall representations mentioned above and score all retrieval single-
product results based on their similarity. The top-$N$ samples are returned as
the retrieval results. Fig 8 shows more retrieval results by EGE-CMP. To
provide a fair comparison with other baselines, we conduct all experiments
using the same training settings and assessment procedures.

Method | mAP@10 | mAP@50 | mAP@100 | mAR@10 | mAR@50 | mAR@100 | Prec@10 | Prec@50 | Prec@100
---|---|---|---|---|---|---|---|---|---
Image-based | 47.46 | 39.79 | 37.06 | 20.81 | 16.12 | 14.63 | 36.28 | 31.20 | 29.20
Text-based | 68.85 | 62.21 | 60.56 | 24.01 | 17.48 | 16.44 | 62.48 | 59.26 | 54.86
ViLBERT [7] | 77.54 | 74.73 | 73.74 | 31.31 | 22.69 | 23.48 | 75.42 | 72.03 | 70.85
LXMERT [10] | 78.62 | 76.57 | 75.29 | 31.63 | 25.98 | 24.92 | 75.57 | 74.29 | 72.81
CLIP* [1] | 76.03 | 75.33 | 73.66 | 30.90 | 25.59 | 24.59 | 73.07 | 72.42 | 70.03
VL-BERT [8] | 76.20 | 75.22 | 72.84 | 30.76 | 25.87 | 24.97 | 73.26 | 72.41 | 70.83
VisualBERT[9] | 77.68 | 76.33 | 75.00 | 31.89 | 26.66 | 25.43 | 75.47 | 73.09 | 69.35
UNITER [2] | 79.00 | 77.37 | 75.73 | 32.05 | 26.40 | 25.22 | 76.39 | 74.42 | 73.99
CAPTURE [3] | 81.49 | 79.38 | 78.81 | 32.99 | 26.86 | 26.46 | 77.27 | 76.54 | 75.00
EGE-CMP (Ours) | 86.91 | 82.68 | 80.57 | 32.99 | 26.92 | 25.68 | 82.59 | 77.89 | 73.72

TABLE II: Comparison of multi-product retrieval performance using various
intra- and cross-modal self-supervised baselines.

Method | mAP@10 | mAP@50 | mAP@100 | Prec@10 | Prec@50 | Prec@100
---|---|---|---|---|---|---
Image-based | 52.90 | 44.65 | 41.51 | 52.90 | 35.22 | 33.75
Text-based | 70.21 | 66.21 | 63.88 | 70.21 | 63.87 | 60.54
ViLBERT [7] | 81.17 | 83.45 | 81.88 | 81.74 | 63.07 | 56.99
LXMERT [10] | 81.09 | 82.02 | 81.48 | 81.09 | 64.76 | 58.78
CLIP* [1] | 81.80 | 82.90 | 81.11 | 81.80 | 62.43 | 57.98
VL-BERT [8] | 81.39 | 83.02 | 81.44 | 81.39 | 64.05 | 58.41
VisualBERT[9] | 79.90 | 81.91 | 80.32 | 79.90 | 62.39 | 55.87
UNITER [2] | 81.74 | 83.45 | 81.88 | 81.74 | 63.07 | 56.99
CAPTURE [3] | 82.03 | 83.60 | 82.04 | 82.03 | 65.22 | 59.03
EGE-CMP (Ours) | 84.66 | 86.37 | 84.79 | 84.66 | 67.76 | 62.00

TABLE III: Comparison of identical product retrieval using different intra-
and cross-modal self-supervised baselines.

###  4.2 Evaluation Metrics

We use two commonly used measures [74, 75] Precision ($\mathrm{Prec}@N$), mean
Average Precision ($\mathrm{mAP}@N$) and a new metric mean Average Recall
($\mathrm{mAR}@N$) to evaluate the instance-level retrieval performances.
Because retrieving each and every product extensively is unnecessary and
impractical in many cases, we report $\mathrm{mAP}@N$, $\mathrm{mAR}@N$, and
$\mathrm{Prec}@N$, where $N=10,50,100$. $\mathrm{Prec}@N$ evaluates the
average accuracy of the top-$N$ predictions per image and is widely used in
the retrieval literature. $\mathrm{Prec}@N(q)$ is defined as follows:

$\mathrm{Prec}@N(q)=\frac{1}{N}\sum_{i=1}^{N}\mathrm{acc}_{q}(i),$ (12)

where $\mathrm{acc}_{q}(i)$ is a binary indicator function that returns $1$
when the $i$-th prediction is correct for the $q$-th query and $0$ otherwise.
$\mathrm{mAP}@N$ is computed as the average $\mathrm{AP@}N$ per image, where
$\mathrm{AP@}N(q)$ is computed as follows:

$\mathrm{AP@}N(q)=\frac{1}{\min\left(m_{q},N\right)}\sum_{k=1}^{N}\mathrm{P}_{q}(k)\mathrm{rel}_{q}(k),$
(13)

where $m_{q}$ is the total number of ground truth images, i.e., corresponding
single-product images that appear in the $q$-th query image, $P_{q}(k)$ is the
precision at rank $k$ for the $q$-th query, and $\mathrm{rel}_{q}(k)$ is a
binary indicator function that returns $1$ when the $k$-th prediction is
correct for the $q$-th query and $0$ otherwise.

To evaluate the recall of instance-level retrieval results, we propose metric
$\mathrm{mAR}@N$, which can be computed as the average $\mathrm{AR@}N$ per
image. $\mathrm{AR@}N(q)$ is re-defined in our paper as follows:

$\mathrm{AR@}N(q)=\frac{1}{C_{q}}\sum_{c=1}^{C}\mathbbm{1}_{c}(q)\mathrm{min}(1,\frac{RETR^{q}_{c}}{\mathrm{min}(\lfloor
r_{q}^{c}\cdot N\rfloor,G_{c})})$ (14)

where $C$ is the total number of single-product categories in the gallery set,
$C_{q}$ equals to the number of existing categories in the $q$-th query,
$\mathbbm{1}_{c}(q)$ is a binary indicator function that returns 1 when class
$c$ exists in the $q$-th query, $RETR^{q}_{c}$ is the number of retrieved
products belonging to class $c$ for the $q$-th query, $G^{c}$ is the number of
ground truths belonging to class $c$ in the gallery set, $r_{q}^{c}$ is the
instance ratio444For instance, for a 2A+3B query image, $r_{q}^{A}=0.4$ and
$r_{q}^{B}=0.6$. of category $c$ in the $q$-th query, and
$\lfloor\cdot\rfloor$ is rounding operation. As per the equation,
$\mathrm{AR@}N(q)$ takes the category distribution into account, i.e., the
inclusion of instance ratio is informative for evaluating both the correctness
and diversity of a retrieval algorithm and guarantees that some trivial
results are not overestimated555For a 2A+3B query image and $N=100$,
$\mathrm{AR@}100(q)$ returns 0.51 and 1.0 for retrieval results 1A+99B and
40A+60B, respectively..

###  4.3 Weakly-Supervised Instance-Level Retrieval

Table II and Table III show the retrieval performances compared with several
intra- and cross-modal models for both multi-product and identical product
retrieval tasks.

Imagebased | VT-Masked | E-Masked | CTR | Node | Subgraph | Concat | mAP/mAR/Prec
---|---|---|---|---|---|---|---
#1 | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | 80.94 / 33.54 / 76.37
#2 | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |  | $\checkmark$ | 83.28 / 32.19 / 78.72
#3 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | 83.86 / 31.40 / 77.71
#4 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  84.74 / 32.91 / 78.76
Textbased | VT-Masked | E-Masked | CTR | Node | Subgraph | Concat | mAP/mAR/Prec
#1 | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | 78.23 / 31.32 / 74.74
#2 | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |  | $\checkmark$ | 83.48 / 32.39 / 78.34
#3 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | 83.50 / 32.03 / 78.41
#4 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  85.25 / 32.83 / 80.52
Image-Textbased | VT-Masked | E-Masked | CTR | Node | Subgraph | Concat | mAP/mAR/Prec
#1 | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | 81.49 / 32.99 / 77.27
#2 | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |  | $\checkmark$ | 85.71 / 33.02 / 81.80
#3 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | 85.76 / 32.60 / 81.91
#4 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  86.91 / 32.99 / 82.59

TABLE IV: The impact of different pretext tasks and cross-modal contrastive
loss. Evaluation for $N=100$. The term ’VT-Masked’ refers to two masked multi-
modal pretext tasks, i.e., MLM and MRP. ’E-Masked’ denotes the MEM task. ’CTR’
refers to cross-modal contrastive loss.

####  4.3.1 Multi-product Retrieval Performance

Intra-Modal Schemes. Our EGE-CMP model is compared to two intra-modal schemes:
Image-based and Text-based schemes. To perform image-based retrieval, the
Image SMT layer are stacked depicted in Section 3.3.1 and the same image input
and pretext tasks are used as same as EGE-CMP, namely MRP. Similarly, the Text
SMT layer is stacked for text-based retrieval and employ just the textual
input and MLM pretext task. Additionally, the depth of these two models is
increased to 24 layers in order to maintain the same number of parameters as
EGE-CMP. According to the experimental results shown in Table II and Table
III, we can observe that these two schemes trail significantly behind since
they are restricted to unimodal data, indicating the importance of modeling
the relationship between multimodality data. In Section 4.5, we present
further experiment findings to substantiate this conclusion.

Cross-Modal Schemes. In Table II, we compare EGE-CMP to many commonly used
self-supervised cross-modal pretraining approaches, including state-of-art
single- and two-stream Vision-language models, as well as a state-of-art zero-
shot classification model, i.e., CLIP [1]. CLIP* refers to a CLIP-like
architecture that encodes images and text separately and is trained with a
contrastive learning. Notably, EGE-CMP exceeds all of these benchmarks on the
majority of three major retrieval criteria. Two-stream models, such as LXMERT
[10], ViLBERT [7], and CLIP*, do not perform better than single-stream models,
indicating that the manner of fusion of cross-modal information is an
important component. Please note that by using entity knowledge, our EGE-CMP
outperforms CAPTURE when employing the hybrid transformer architecture. We
attribute EGE-CMP’s better performance to its entity knowledge injection.
Finally, we examine the influence of various layer types in Section 4.5.

####  4.3.2 Identical product Retrieval Performance

As the same setting with multi-product retrieval task, We also perform the
identical-product retrieval on all the compared methods and our EGE-CMP model.
Table III presents the experimental results. These results indicate that
performances of all methods have a slight improvement due to the limited
number of the gallery set. Meanwhile, compared with multi-product retrieval,
the performances of the multi-modal pretraining models, i.e., ViLBert [7],
UNITER [2], CAPTURE [3] and so on, are much closer, which indicates that
identical product retrieval is a more difficult task. In both mAP and Prec
metrics, our EGE-CMP still obtains the best performance. It can effectively
prove that the commonsense words or phrases existing in caption data are
beneficial to performance improvement.

###  4.4 Impact of Entity Pretext Tasks and Graph Ranking Loss

Table IV provides the experimental results of our EGE-CMP under different
entity loss settings as well as different feature representations, e.g. Masked
Entity Modeling, Entity Node Loss and Entity Subgraph Loss. We apply the
Masked Entity Modeling (E-Masked) at the Entity/Text and Text/Visual COMT
layer to make the representation more focused on the tokens related to the
real contexts, which is beneficial to the modality combination learning. In
mAP metrics, E-Masked Loss helps our model improve the retrieval performance
by 2.8, 7.0 and 5.2 respectively (#1 vs #2) under the different feature
representations, and we find it of certain help when added to the deeper
layers. Moreover, after making Entity Node Loss act on the features from the
Text/Visual Transformer with that from the Co-Transformer for retrieval, it
further improves mAP metrics, respectively (#2 vs #3). We also verify the
effectiveness of Entity Subgraph Loss and observe that it is quite useful to
further improve the model performance, which suggests that the improvement
mainly comes from the Entity learning.

Model | Config | Depth | mAP/mAR/Prec
---|---|---|---
w/o-Cross | (6,0,6) | 12 | 85.9 / 32.2 / 82.6
w/o-Co | (6,6,0) | 12 | 85.8 / 32.6 / 82.4
w/o-T/V/E | (0,6,6) | 12 | 50.0 / 18.7 / 35.9
EGE-CMP-A | (2,5,5) | 12 | 85.2 / 31.4 / 79.7
EGE-CMP-B | (5,2,5) | 12 | 86.9 / 33.2 / 82.4
EGE-CMP-C | (5,5,2) | 12 | 86.7 / 33.3 / 82.2
EGE-CMP-S | (2,2,2) | 6 | 83.7 / 30.1 / 77.7
EGE-CMP-L | (8,8,8) | 24 | 86.9 / 30.9 / 82.5
EGE-CMP | (4,4,4) | 12 | 86.9 / 32.9 / 82.6

TABLE V: Performance of different layer combinations. Evaluation for $N=10$.

| Metric | $N=10$ | $N=50$
---|---|---|---
5 brands |  mAP$@N$ |  63.3 / 64.5 / 70.2 |  60.7 / 62.2 / 67.4
mAR$@N$ |  23.2 / 24.7 / 30.1 |  19.2 / 20.1 / 22.9
Prec$@N$ |  56.5 / 57.1 / 64.7 |  56.5 / 57.7 / 66.3
10 brands |  mAP$@N$ |  56.8 / 58.2 / 64.7 |  54.1 / 55.2 / 60.2
mAR$@N$ |  19.6 / 20.5 / 25.3 |  16.4 /17.4 / 20.4
Prec$@N$ |  50.2 / 51.9 / 56.4 |  51.3 / 52.7 / 57.5
20 brands |  mAP$@N$ |  42.6 / 43.3 / 46.8 |  36.7 / 37.3 / 40.2
mAR$@N$ |  17.5 / 17.6 / 18.3 |  12.9 / 13.2 / 15.0
Prec$@N$ |  32.2 / 32.4 / 36.4 |  32.9 / 33.1 / 37.5

TABLE VI: Comparison of zero-shot retrieval performance. Listed in the
following order: LXMERT/UNITER/EGE-CMP.

Method |  mAP$@100$ |  mAR$@100$ |  Prec$@100$
---|---|---|---
UNITER-single | 86.56 | 80.82 | 80.82
LXMERT-single | 86.05 | 80.59 | 80.59
CAPTURE-single | 88.24 | 83.33 | 83.33
EGE-CMP-single | 91.00 | 85.42 | 86.37

TABLE VII: Ablation study of single-product retrieval. Note that for single-
product retrieval, the metric Prec$@N$ is equivalent to mAR$@N$ since an image
contains just one category. Figure 7: Visualizations of the EGE-CMP retrieval
results. On the left are multi-product query images. Green/red boxes indicate
Correct/Incorrect retrieval images.  Figure 8: More retrieval visualizations
of EGE-CMP on the identical product retrieval task. On the left are the query
images. Green boxes indicate correct retrieval images, whereas red boxes
indicate faulty retrieval images. The captions of the retrieved samples have
been omitted for the sake of simplicity. Figure 9: Visualize the embeddings
generated by UNITER, CAPTURE and EGE-CMP via t-SNE. Points belonging to the
same category are of the same color. Best viewed in color.

###  4.5 Impact of Layer Configuration

We study how the arrangement of transformer layers may affect the performance
of our model in Table V. In the Config column, the triplet represents the
number of Text/Visual Transformer, Cross-Transformer, and Co-Transformer
layers, respectively. To provide a fair comparison, we first eliminate layers
of a certain kind while maintaining the depth of the resultant network at the
same level as EGE- CMP’s, i.e., 12 levels. The terms ’w/o-Cross’, ’w/o-Co’,
and ’w/o-T/V/E’ refer to the model obtained by deleting the EGE-CMP Cross-
Transformer, Co-Transformer, and Text/Visual/Entity Transformer layers. As can
be observed, these three models perform worse than EGE-CMP, demonstrating the
efficacy of its hybrid-stream design. Additionally, we discover that a single
model transformer (SMT) is critical for our model, demonstrating that a fine-
tuned transformer is more beneficial for the overall model’s performance.
Additionally, we investigate the combination of three-layer types in varying
proportions in the second group of Table V (EGE-CMP-A,B,C). It turns out that
the (4,4,4) combination gives the best performance. We examine the performance
of a smaller model (EGE-CMP-S) and a bigger model in greater detail (EGE-
CMP-L). As can be observed, EGE-CMP with the (4,4,4) configuration offers a
more favorable trade-off between accuracy and parameters.

###  4.6 Zero-Shot Instance-Level Retrieval

We propose that a retrieval-based strategy is more applicable to real-world
settings in which the category set is constantly updated and collecting vast
volumes of clean labels is too expensive. In contrast to detection, our
retrieval approach does not rely on a fixed-size collection of predefined
classes or on fine-grained box annotations. To demonstrate this point, we
perform zero-shot retrieval studies and provide the results in Table VI. We
manually remove 5/10/20 brands from the train set and train EGE-CMP on the
remaining samples to avoid discarding the deleted categories during training.
Then we compare EGE-CMP to the classes of these previously unknown brands.
Additionally, we compare our model to the LXMERT two-stream model and the
UNITER single-stream model. As can be observed, EGE-CMP outperforms LXMERT and
UNITER on all three criteria, demonstrating its generalizability well.
Additionally, we display the embeddings created by UNITER, CAPTURE, and EGE-
CMP in Figure 7 using t-SNE [76]. It turns out that the EGE-CMP
characteristics are more discriminative, which is advantageous for the
retrieval job.

###  4.7 Comparisons on Single-Product Retrieval

It’s worth noting that EGE-CMP may be used for both single- 666The single-
product retrieval is a special case in the identical product search. There is
only one instance in all product samples. and multi-product retrieval. Indeed,
it excels at these two tasks and outperforms other baselines when it comes to
single-product retrieval. Specifically, we select a single-product sample from
the gallery set as a query and execute single-product retrieval from the
galleryset’s remaining samples. The performance of four models is compared in
Table VII, namely UNITER-single, LXMERT-single, CAPTURE-single, and EGE-CMP-
single. As can be shown, single-product retrieval performs far better than
multi-product retrieval, as the difficulty is significantly decreased when
there is only one instance/entity in the image/text. Additionally, we see that
the performance of ’EGE-CMP-single’ is superior to that of two other
baselines, demonstrating the superiority of EGE-CMP.

##  5 Conclusion

In this work, we describe the first attempt at extending the scope of
canonical intra-/cross-modal retrieval to a more generalized scenario, namely,
weakly-supervised multi-modal instance-level product retrieval, which has
broad demands potential in E-commerce business. To facilitate multi-modal pre-
training, we present the Product1M, which is one of the biggest available
visual-language retrieval datasets, as well as the only one designed
exclusively for instance-level retrieval in real-world contexts. Additionally,
we present an entity-graph enhanced hybrid-stream transformer, dubbed EGE-CMP,
which excels at capturing potential synergy and understanding common relevant
data from disparate modalities. we incorporate genuine semantic information
into the entity knowledge, focusing cross-modal contrastive learning on the
related regions or words between multi-modal characteristics. Extensive
experimental evidence verifies that our EGE-CMP outperforms several state-of-
art multi-modal pretraining models on a wide variety of metrics. We believe
that the proposed EGE-CMP model, Product1M dataset, and established baselines
will stimulate additional research, making it a more trustworthy and flexible
retrieval search basement

##  6 Acknowledgement

This work was supported in part by Shenzhen Fundamental Research Program
(Project No.RCYX20200714114642083, No. JCYJ20190807154211365), Guangdong
Outstanding Youth Fund (Grant No. 2021B1515020061), National Key R$\&$D
Program of China under Grant (No. 2018AAA0100300), Guangdong Province Basic
and Applied Basic Research (Regional Joint Fund-Key) Grant (No.
2019B1515120039), National Natural Science Foundation of China under Grant
(No. U19A2073) and (No. 61976233).

## References

  * [1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” 2021.
  * [2] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” in _ECCV_. Springer, 2020, pp. 104–120.
  * [3] X. Zhan, Y. Wu, X. Dong, Y. Wei, M. Lu, Y. Zhang, H. Xu, and X. Liang, “Product1m: Towards weakly supervised instance-level product retrieval via cross-modal pretraining,” in _ICCV_ , 2021, pp. 11 782–11 791.
  * [4] A. A. Winecoff, F. Brasoveanu, B. Casavant, P. Washabaugh, and M. Graham, “Users in the loop: a psychologically-informed approach to similar item retrieval,” in _RecSys_ , T. Bogers, A. Said, P. Brusilovsky, and D. Tikk, Eds. ACM, 2019, pp. 52–59.
  * [5] P. Rahayu, D. I. Sensuse, B. Purwandari, I. Budi, F. Khalid, and N. Zulkarnaim, “A systematic review of recommender system for e-portfolio domain,” 2017, pp. 21–26.
  * [6] S. Vogler, A. Vitry _et al._ , “Cancer drugs in 16 european countries, australia, and new zealand: a cross-country price comparison study,” _The Lancet Oncology_ , vol. 17, no. 1, pp. 39–47, 2016.
  * [7] J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” in _NIPS_ , 2019, pp. 13–23.
  * [8] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai, “Vl-bert: Pre-training of generic visual-linguistic representations,” in _ICLR_ , 2020\.
  * [9] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,” _arXiv preprint arXiv:1908.03557_ , 2019.
  * [10] H. Tan and M. Bansal, “Lxmert: Learning cross-modality encoder representations from transformers,” in _EMNLP_ , 2019.
  * [11] J. Li, D. Li, C. Xiong, and S. C. H. Hoi, “BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation,” _CoRR_ , vol. abs/2201.12086, 2022.
  * [12] L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J. Hwang, K. Chang, and J. Gao, “Grounded language-image pre-training,” _CoRR_ , vol. abs/2112.03857, 2021.
  * [13] S. Esmaeilpour, B. Liu, E. Robertson, and L. Shu, “Zero-shot open set detection by extending clip,” _arXiv e-prints_ , pp. arXiv–2109, 2021.
  * [14] H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li, “Clip4clip: An empirical study of CLIP for end to end video clip retrieval,” _CoRR_ , vol. abs/2104.08860, 2021.
  * [15] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, “Open-vocabulary detection via vision and language knowledge distillation,” _arXiv preprint arXiv:2104.13921_ , 2021.
  * [16] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” _CoRR_ , vol. abs/1907.11692, 2019.
  * [17] A. Talmor, Y. Elazar, Y. Goldberg, and J. Berant, “olmpics - on what language model pre-training captures,” _Trans. Assoc. Comput. Linguistics_ , vol. 8, pp. 743–758, 2020.
  * [18] W. Liu, P. Zhou, Z. Zhao, Z. Wang, Q. Ju, H. Deng, and P. Wang, “K-BERT: enabling language representation with knowledge graph,” in _AAAI_. AAAI Press, 2020, pp. 2901–2908.
  * [19] Y. Qin, Y. Lin, R. Takanobu, Z. Liu, P. Li, H. Ji, M. Huang, M. Sun, and J. Zhou, “ERICA: improving entity and relation understanding for pre-trained language models via contrastive learning,” in _ACL_. Association for Computational Linguistics, 2021, pp. 3350–3363.
  * [20] Y. Qi, Y.-Z. Song, H. Zhang, and J. Liu, “Sketch-based image retrieval via siamese convolutional neural network,” in _ICIP_. IEEE, 2016, pp. 2460–2464.
  * [21] C. Bai, L. Huang, X. Pan, J. Zheng, and S. Chen, “Optimization of deep convolutional neural network for large scale image retrieval,” _Neurocomputing_ , vol. 303, pp. 60–67, 2018.
  * [22] A. Qayyum, S. M. Anwar, M. Awais, and M. Majid, “Medical image retrieval using deep convolutional neural network,” _Neurocomputing_ , vol. 266, pp. 8–20, 2017.
  * [23] P. Nurmi, E. Lagerspetz, W. Buntine, P. Floréen, and J. Kukkonen, “Product retrieval for grocery stores,” in _ACM SIGIR_ , 2008, pp. 781–782.
  * [24] B. Wang, Y. Yang, X. Xu, A. Hanjalic, and H. T. Shen, “Adversarial cross-modal retrieval,” in _ACM MM_ , 2017, pp. 154–162.
  * [25] F. Feng, X. Wang, and R. Li, “Cross-modal retrieval with correspondence autoencoder,” in _ACM MM_ , 2014, pp. 7–16.
  * [26] Y. Wei, Y. Zhao, C. Lu, S. Wei, L. Liu, Z. Zhu, and S. Yan, “Cross-modal retrieval with cnn visual features: A new baseline,” _IEEE Trans. Syst. Man Cybern._ , vol. 47, no. 2, pp. 449–460, 2016.
  * [27] Y. Cao, M. Long, J. Wang, Q. Yang, and P. S. Yu, “Deep visual-semantic hashing for cross-modal retrieval,” in _ACM SIGKDD_ , 2016, pp. 1445–1454.
  * [28] K. Wang, R. He, L. Wang, W. Wang, and T. Tan, “Joint feature selection and subspace learning for cross-modal retrieval,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 38, no. 10, pp. 2010–2023, 2015.
  * [29] C. Deng, Z. Chen, X. Liu, X. Gao, and D. Tao, “Triplet-based deep hashing network for cross-modal retrieval,” _IEEE Trans. Image Process._ , vol. 27, no. 8, pp. 3893–3903, 2018.
  * [30] Z. Kuang, Y. Gao, G. Li, P. Luo, Y. Chen, L. Lin, and W. Zhang, “Fashion retrieval via graph reasoning networks on a similarity pyramid,” in _ICCV_ , 2019, pp. 3066–3075.
  * [31] X. Gu, Y. Wong, L. Shou, P. Peng, G. Chen, and M. S. Kankanhalli, “Multi-modal and multi-domain embedding learning for fashion retrieval and analysis,” _IEEE Trans. Multim._ , vol. 21, no. 6, pp. 1524–1537, 2018.
  * [32] Y. Sun, S. Wang, Y. Li, S. Feng, X. Chen, H. Zhang, X. Tian, D. Zhu, H. Tian, and H. Wu, “ERNIE: enhanced representation through knowledge integration,” _CoRR_ , vol. abs/1904.09223, 2019.
  * [33] Y. Sun, S. Wang, S. Feng, S. Ding, C. Pang, J. Shang, J. Liu, X. Chen, Y. Zhao, Y. Lu, W. Liu, Z. Wu, W. Gong, J. Liang, Z. Shang, P. Sun, W. Liu, X. Ouyang, D. Yu, H. Tian, H. Wu, and H. Wang, “ERNIE 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation,” _CoRR_ , vol. abs/2107.02137, 2021.
  * [34] X. Wang, T. Gao, Z. Zhu, Z. Zhang, Z. Liu, J. Li, and J. Tang, “KEPLER: A unified model for knowledge embedding and pre-trained language representation,” _Trans. Assoc. Comput. Linguistics_ , vol. 9, pp. 176–194, 2021.
  * [35] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and visual question answering,” in _CVPR_ , 2018, pp. 6077–6086.
  * [36] Y. Hu, L. Zheng, Y. Yang, and Y. Huang, “Twitter100k: A real-world dataset for weakly supervised cross-media retrieval,” _IEEE Trans. Multim._ , vol. 20, no. 4, pp. 927–938, 2017.
  * [37] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei, “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” _Int. J. Comput. Vis._ , vol. 123, no. 1, pp. 32–73, 2017.
  * [38] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,” _Trans. Assoc. Comput. Linguistics_ , vol. 2, pp. 67–78, 2014\.
  * [39] V. Ordonez, G. Kulkarni, and T. L. Berg, “Im2text: Describing images using 1 million captioned photographs,” in _NIPS_ , 2011, pp. 1143–1151.
  * [40] T. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng, “NUS-WIDE: a real-world web image database from national university of singapore,” in _CIVR_ , 2009.
  * [41] J. Krapac, M. Allan, J. Verbeek, and F. Jurie, “Improving web image search results using query-relative classifiers,” _ICSCCVPR_ , pp. 1094–1101, 2010\.
  * [42] X.-S. Wei, Q. Cui, L. Yang, P. Wang, and L. Liu, “Rpc: A large-scale retail product checkout dataset,” _arXiv preprint arXiv:1901.07249_ , 2019.
  * [43] C. Corbiere, H. Ben-Younes, A. Ramé, and C. Ollion, “Leveraging weakly annotated data for fashion image retrieval and label prediction,” in _ICCV Workshops_ , 2017, pp. 2268–2274.
  * [44] F. Ensan and E. Bagheri, “Document retrieval model through semantic linking,” in _WSDM_ , 2017, pp. 181–190.
  * [45] H. Noh, A. Araujo, J. Sim, T. Weyand, and B. Han, “Large-scale image retrieval with attentive deep local features,” in _ICCV_ , 2017, pp. 3456–3465.
  * [46] S. Büttcher, C. L. Clarke, and G. V. Cormack, _Information retrieval: Implementing and evaluating search engines_. Mit Press, 2016.
  * [47] D. Harman _et al._ , “Information retrieval: the early years,” _Foundations and Trends® in Information Retrieval_ , vol. 13, no. 5, pp. 425–577, 2019.
  * [48] X. Ji, W. Wang, M. Zhang, and Y. Yang, “Cross-domain image retrieval with attention modeling,” in _ACM MM_ , 2017, pp. 1654–1662.
  * [49] Z. Lin, Z. Yang, F. Huang, and J. Chen, “Regional maximum activations of convolutions with attention for cross-domain beauty and personal care product retrieval,” in _ACM MM_ , 2018, pp. 2073–2077.
  * [50] W. Wang, X. Yang, B. C. Ooi, D. Zhang, and Y. Zhuang, “Effective deep learning-based multi-modal retrieval,” _The VLDB Journal_ , vol. 25, no. 1, pp. 79–101, 2016.
  * [51] P. Tang, X. Wang, S. Bai, W. Shen, X. Bai, W. Liu, and A. Yuille, “Pcl: Proposal cluster learning for weakly supervised object detection,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 42, no. 1, pp. 176–191, 2018\.
  * [52] Y. Shen, R. Ji, S. Zhang, W. Zuo, and Y. Wang, “Generative adversarial learning towards fast weakly supervised detection,” in _CVPR_ , 2018.
  * [53] K. Ye, M. Zhang, A. Kovashka, W. Li, D. Qin, and J. Berent, “Cap2det: Learning to amplify weak caption supervision for object detection,” 2019.
  * [54] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei _et al._ , “Oscar: Object-semantics aligned pre-training for vision-language tasks,” in _ECCV_. Springer, 2020, pp. 121–137.
  * [55] D. Gao, L. Jin, B. Chen, M. Qiu, P. Li, Y. Wei, Y. Hu, and H. Wang, “Fashionbert: Text and image matching with adaptive loss for cross-modal retrieval,” in _SIGIR_. ACM, 2020, pp. 2251–2260.
  * [56] W. Li, C. Gao, G. Niu, X. Xiao, H. Liu, J. Liu, H. Wu, and H. Wang, “Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning,” _arXiv preprint arXiv:2012.15409_ , 2020.
  * [57] T. Fu, L. Li, Z. Gan, K. Lin, W. Y. Wang, L. Wang, and Z. Liu, “VIOLET : End-to-end video-language transformers with masked visual-token modeling,” _CoRR_ , vol. abs/2111.12681, 2021.
  * [58] J. Yang, Y. Bisk, and J. Gao, “Taco: Token-aware cascade contrastive learning for video-text alignment,” in _ICCV_. IEEE, 2021, pp. 11 542–11 552.
  * [59] L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu, “FILIP: Fine-grained interactive language-image pre-training,” in _International Conference on Learning Representations_ , 2022. [Online]. Available: https://openreview.net/forum?id=cpDhcsEDC2
  * [60] W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” 2021.
  * [61] L. Yuan, D. Chen, Y. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, C. Liu, M. Liu, Z. Liu, Y. Lu, Y. Shi, L. Wang, J. Wang, B. Xiao, Z. Xiao, J. Yang, M. Zeng, L. Zhou, and P. Zhang, “Florence: A new foundation model for computer vision,” _CoRR_ , vol. abs/2111.11432, 2021\.
  * [62] J. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang, “M6: A chinese multimodal pretrainer,” _CoRR_ , vol. abs/2103.00823, 2021.
  * [63] Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao, “Simvlm: Simple visual language model pretraining with weak supervision,” _CoRR_ , vol. abs/2108.10904, 2021.
  * [64] N. Mu, A. Kirillov, D. Wagner, and S. Xie, “Slip: Self-supervision meets language-image pre-training,” 2021.
  * [65] C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in _ICML_ , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 2021, pp. 4904–4916.
  * [66] X. Zhu, J. Zhu, H. Li, X. Wu, X. Wang, H. Li, X. Wang, and J. Dai, “Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks,” _CoRR_ , vol. abs/2112.01522, 2021.
  * [67] Y. Huo, M. Zhang, G. Liu, H. Lu, Y. Gao, G. Yang, J. Wen, H. Zhang, B. Xu, W. Zheng, Z. Xi, Y. Yang, A. Hu, J. Zhao, R. Li, Y. Zhao, L. Zhang, Y. Song, X. Hong, W. Cui, D. Y. Hou, Y. Li, J. Li, P. Liu, Z. Gong, C. Jin, Y. Sun, S. Chen, Z. Lu, Z. Dou, Q. Jin, Y. Lan, W. X. Zhao, R. Song, and J. Wen, “Wenlan: Bridging vision and language by large-scale multi-modal pre-training,” _CoRR_ , vol. abs/2103.06561, 2021.
  * [68] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in _NAACL-HLT )_ , J. Burstein, C. Doran, and T. Solorio, Eds. ACL, 2019, pp. 4171–4186.
  * [69] W. Xiong, J. Du, W. Y. Wang, and V. Stoyanov, “Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model,” in _ICLR_. OpenReview.net, 2020.
  * [70] L. He, S. Zheng, T. Yang, and F. Zhang, “Klmo: Knowledge graph enhanced pretrained language model with fine-grained relationships,” in _EMNLP_. Association for Computational Linguistics, 2021, pp. 4536–4542.
  * [71] X. Li, H. Zhang, and R. Zhang, “Adaptive graph auto-encoder for general data clustering,” _IEEE Trans. Pattern Anal. Mach. Intell._ , pp. 1–1, 2021.
  * [72] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _ICML_. PMLR, 2020, pp. 1597–1607.
  * [73] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [74] T. Weyand, A. Araujo, B. Cao, and J. Sim, “Google Landmarks Dataset v2 - A Large-Scale Benchmark for Instance-Level Recognition and Retrieval,” in _Proc. CVPR_ , 2020.
  * [75] D. Cai, X. Gu, and C. Wang, “A revisit of deep hashings for large-scale content based image retrieval,” in _arXiv:1711.06016_ , 2017.
  * [76] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _JMLR_ , vol. 9, no. 11, 2008.