# A Generative Approach for Wikipedia-Scale Visual Entity Recognition

Mathilde Caron Ahmet Iscen Alireza Fathi Cordelia Schmid
Google Research

###### Abstract

†† Code: github.com/google-research/scenic/tree/main/scenic/projects/gerald

In this paper, we address web-scale visual entity recognition, specifically
the task of mapping a given query image to one of the 6 million existing
entities in Wikipedia. One way of approaching a problem of such scale is using
dual-encoder models (_e.g_. CLIP), where all the entity names and query images
are embedded into a unified space, paving the way for an approximate $k$NN
search. Alternatively, it is also possible to re-purpose a captioning model to
directly generate the entity names for a given image. In contrast, we
introduce a novel Generative Entity Recognition (ger) framework, which given
an input image learns to auto-regressively decode a semantic and
discriminative “code” identifying the target entity. Our experiments
demonstrate the efficacy of this ger paradigm, showcasing state-of-the-art
performance on the challenging OVEN benchmark. ger surpasses strong
captioning, dual-encoder, visual matching and hierarchical classification
baselines, affirming its advantage in tackling the complexities of web-scale
recognition.

##  1 Introduction

Generative vision-language models such as GPT-4 [30], Flamingo [2] or PALI
[5], are becoming increasingly popular for computer vision applications. They
show an impressive ability to generate free-form text for describing the
contents of an image (captioning), or answering questions based on an image
(visual-question answering). Nevertheless, their potential for _recognition_
tasks [12], which usually require a more concise, structured output, remains
under-explored. The focus of this paper is to explore their application for
the challenging task of web-scale entity recognition. A recent benchmark,
Open-domain Visual Entity recognitioN (OVEN) [12], challenges models to
associate an image with a Wikipedia entity from a pool of over six million
entities. Models must establish a robust association between images across
millions of coarse-grained and fine-grained entities, encompassing a wide
spectrum of concepts such as animals, buildings, locations, and a multitude of
others [12].

Figure 1: We introduce ger, a novel generative paradigm for web-scale visual
entity recognition. We create compact semantic codes for each entity, and
learn to auto-regressively generate them for a given query image at inference.

Traditionally, the predominant methods employed to address the challenge of
visual entity recognition have revolved around either classification or
contrastive dual-encoder paradigm like CLIP [32]. While classification offers
a straightforward approach, it grapples with limitations when confronted with
extensive label spaces such as that of OVEN, resulting in substantial
parameter counts and practical engineering complexities. The dual-encoder
approach on the other hand, learns a unified image-text feature space, thereby
facilitating efficient nearest neighbor searches for recognition. Nonetheless,
this approach exhibits its own drawbacks: (a) it does not directly optimize
for the final recognition task but instead relies on indirect optimization
through contrastive loss where a set of negative data has to be subsampled at
training time [11, 29, 32], (b) compressing either the image or text into an
embedding vector results in loss of information, detrimentally affecting
performance for fine-grained recognition [15] and (c) the memory requirements
for storing dense representations scale proportionally with the size of the
entity set.

These challenges of the dual-encoder paradigm have kindled interest in
alternative strategies. Notably, in Natural Language Processing (NLP) domain,
recent works challenge the dual-encoder approach and use generative models
instead for information retrieval [42, 33, 31, 6, 25, 41]. These works
represent each element of the corpus by a compact code of integers, and learn
an auto-regressive generative model to decode the target code for a given
query. This paradigm promises to overcome some drawbacks of dual-encoders by
simplifying the retrieval pipeline such that the training and inference
objectives are the same, and directly encoding the corpus within the model’s
parameters. Also as an alternative to dual encoders, OVEN paper [12] showcases
the feasibility of extending a generative image captioning model [5] for
visual entity recognition by matching the generated caption to one of the
Wikipedia entity texts [34].

Inspired by these recent explorations, we propose a Generative Entity
Recognition (ger) framework (illustrated in Fig. 1) to facilitate end-to-end
visual entity recognition by leveraging generative auto-regressive models.
Specifically, we represent each Wikipedia entity with a code, _i.e_. a short
sequence of integers. Then, we train models to predict an entity from an input
image by auto-regressively generating the code corresponding to the target
entity. We find that creating unAmbiguous, Language-based and Discriminative
(ald) entity codes results in the best variant of our ger framework, which we
denote by ger-ald. In fact, while we observe that unstructured “atomic” codes
work well in some scenarios, they fail when training data or model capacity
are limited or more importantly, when the entity set reaches the million scale
(see Sec. 4.4.1). Plus, they cannot generalize to new entities. In contrast,
we find that semantically-structured codes based on language improve upon
atomic codes by leveraging generic concepts shared across related entities
(see example in Fig. 1 with “Black colobus” and “Black-and-white colobus”
sharing common code tokens). A simple way of creating codes based on language
is to directly tokenize [20] the entity name, which is akin to image
captioning where the entity name is used as a caption [12, 6]. However, we
find that such tokenized entity names contain clutter and noisy information,
all the more so when the entity name is long (see Sec. 4.4.2). Our ger-ald
method improves over this simple captioning baseline by decoding only the most
discriminative part of the tokenized entity name, _i.e_. the part which makes
the considered entity name the most different compared to all other entities.

Finally, we also propose an entity-based pre-training to condition the ger
models to web-scale entity recognition. Inspired by recent advances in
retrieval-based methods [15, 23], we retrieve a subset of images from a large-
scale image-text dataset typically used for captioning or contrastive pre-
training [5] and re-purpose it by replacing the original text captions with
related OVEN entity names. Overall, our experiments demonstrate the efficacy
of the proposed ger paradigm: ger-ald outperforms previously published numbers
on OVEN benchmark [12] by $+6.7$ top-1 accuracy, while using $42\times$ less
parameters. In summary, our contributions are as follows:

  * •

a generative entity recognition framework (ger) to facilitate end-to-end
visual entity recognition;

  * •

an innovative strategy for encoding Wikipedia entities into unambiguous
language-based discriminative (ald) codes that are highly effective for ger;

  * •

an entity-based pre-training process without requiring human intervention;

  * •

state-of-the-art results in challenging web-scale OVEN entity recognition and
on-par performance to traditional classifiers in smaller-scale label-space
scenarios.

##  2 Related work

Visual entity recognition aims to recognize classes, or entities given visual
inputs [35]. Granularity of visual entity recognition tasks varies from every-
day generic objects [8, 9], to fine-grained domains, such as birds [44], dogs
[17], cars [18], food [4], landmarks [47], faces [50] and natural world
species [43]. Some challenges for the visual entity recognition tasks include
imbalanced training classes following a long-tailed distribution [24], or
noisy training labels [22]. Recent work [12] proposes a new, web-scale dataset
for open-domain entity recognition. This challenging benchmark contains $6$M
entity names derived from Wikipedia page titles, including coarse-grained and
fine-grained entities, encompassing a wide spectrum of concepts such as
animals, buildings, organizations, landmarks, and a multitude of other. The
authors show that generative captioning models (_i.e_. PaLI [5]) outperform
dual encoder models for large-scale entity recognition. In this paper, we
build upon this observation, and study generative models for accurate and
efficient entity recognition.

Extreme classification tackles entity recognition specifically at a very large
scale with a pure classification approach [3, 26, 1]. Typical approaches
explore strategies for scaling to the hundred of thousands scale and
preliminary results are even shown at million scale [1]. By leveraging
generative image-to-text models, we propose a fresh perspective beyond
traditional classification methods typically used in the context of large-
scale visual entity recognition.

Generative auto-regressive retrieval methods are increasingly popular in NLP
[42, 33, 31, 6, 25, 41]. GENRE retrieves Wikipedia entities by generating
their names in an autoregressive fashion. Seminal work DSI [42] shows the
benefit of learning to decode compact codes (created either randomly or with
hierarchical k-means clustering) associated with each document. Neural Corpus
Indexer [46] proposes a specific decoding scheme for generative retrieval and
show the benefit of _query augmentation_ by automatically generating training
queries for documents to be indexed. TIGER [33] studies generative retrieval
in the context of recommender systems. Finally, [31] conducts a systematic
study of generative retrieval systems when scaled to millions of document
passages. Only very few works explore this family of approaches in computer
vision domain, and only in very small-scale and uni-modal scenarios [49].

##  3 Method

Figure 2: Overview of ger-ald method. (a) We utilize a text tokenizer to
create compact and semantic codes, which represents each entity with short,
but discriminative representations. (b) We learn a generative auto-regressive
model, which learns to decode the correct code for given query image and text
pair.

Our goal is to explore how to adapt generative auto-regressive models to the
task of visual entity recognition (ger). While previous works have shown
preliminary signal that it is possible to repurpose autoregressive models for
entity recognition by directly decoding entity names [12, 6], we propose a
more effective strategy. An overview of our framework is in Fig. 2.

###  3.1 Problem definition

Web-scale visual entity recognition. The Open-domain Visual Entity recognitioN
(OVEN) [12] task consists of mapping input visual queries to one of the $6$M
English Wikipedia entities. More specifically, for a given image query $x_{v}$
and text query $x_{t}$, the model needs to recognize the corresponding entity
$e$ among the set $\mathcal{E}$ of all possible entities. The purpose of the
input text $x_{t}$ is to achieve unambiguous recognition. For example, when
several entities are represented in the query image $x_{v}$, the text query
indicates which one needs to be recognized. Each entity $e\in\mathcal{E}$
comes with an entity name, denoted by $t_{e}$, which corresponds to the title
of the entity Wikipedia page.

Representing each entity with a code. In ger, we represent each entity $e$ by
a code denoted by $c^{e}=\\{c^{e}_{1},...,c^{e}_{L}\\}\in\llbracket
1,V\rrbracket^{L}$ where $L$ is the length of the code and $V$ is the size of
the vocabulary of all integer values that each code token $c^{e}_{i}$ can
take. This forms up to $V^{L}$ unique codes. Note that vanilla image
classification and captioning baselines can both be cast into this code
formulation. In fact, with $L=1$ and $V=|\mathcal{E}|$, the codes are
equivalent to the labels used in standard multi-class classification. On the
other hand, if each code token value in $\llbracket 1,V\rrbracket$ maps to a
(sub-)word in a pre-defined vocabulary [20], then the codes simply correspond
to standard tokenized text used in captioning models [19, 39, 45]. In the
following paragraphs, we detail ger-ald, our most effective strategy for
building codes $\mathcal{C}$ to represent all 6M English Wikipedia entities.

###  3.2 ger-ald: Creating ald codes for ger

We design the code set $\mathcal{C}$ so that it has three properties which we
find are important for effective ger models: i) semantically structured thanks
to language, ii) discriminative and compact, and iii) unambiguous. Our
algorithm to create such unambiguous, language-based and discriminative codes,
called ald, is illustrated in Fig. 2 (a) and described in pseudo-code in
Algorithm 1 of the Appendix.

Semantic tokens based on language. We find that entity codes $\mathcal{C}$
benefit from following a semantic structure, especially in scenarios where
memorizing unstructured atomic codes is difficult. We show in Sec. 4.4.1 that
using unstructured atomic codes fail when the amount of training data or the
model capacity are limited or, of particular interest, when the entity set
size increases to the million scale (see Fig. 3). Intuitively, we want
entities that are semantically similar to have some overlapping code tokens.
For example, we wish that entities $e=\small{\text{Q521977}}$ with
corresponding name $t_{\text{Q521977}}=$ “Black colobus” and
$e=\small{\text{Q358813}}$ with corresponding name $t_{\text{Q358813}}=$
“Black-and-white colobos” to share some code tokens, given that these
correspond to two close species.

A simple yet effective way of having semantic codes is to tokenize the entity
names based on text tokenizers [19, 39, 20, 6]. If each of the sub-words in
the entity names are mapped to an integer representing this sub-word, then
entities Q358813 and Q521977 naturally share code tokens: those representing
the phrase “colobus”. We denote by $\Phi(.)$ an off-the-shelf text tokenizer
with a vocabulary of $V_{\Phi}$ sub-words such that
$\Phi(t_{e})=\\{y^{e}_{1},...,y^{e}_{L_{e}}\\}\in\llbracket
1,V_{\Phi}\rrbracket^{L_{e}}$ where $L_{e}$ is the length of the tokenized
entity name $\Phi(t_{e})$. In practice we use the same language tokenizer as
GIT [45] for $\Phi(.)$ and have a vocabulary size of $V=V_{\Phi}=30522$. We
refer to the baseline of using codes $\mathcal{C}$ created by simple
tokenization of the entity name as ger-caption (i.e. we treat the entity name
as a caption) [6]. We show in the following paragraph how ger-ald codes differ
from such ger-caption codes by making them more compact and discriminative.

Discriminative and compact codes. Our goal is to build short and highly
discriminative codes because they are easier to learn for the model, as
validated by our experiments in Sec. 4.4.2. For example, the tokenized entity
name $\Phi(t_{\text{Q358813}})=$ $\Phi(\text{``Black-and-white colobus''})$
counts $L_{\text{Q358813}}=8$ tokens, but clearly not all $8$ tokens are
important to make this entity discriminative compared to all other existing
entities. Hence, we choose to represent each entity with the _bare minimum_ ,
removing all the _clutter_ which is not only non-discriminative but also adds
noise. We achieve this by selecting the most discriminative and rarest tokens
within the tokenized entity name. Specifically, we compute the frequency
$f_{v}$ of each token value $v\in[1,V]$ in the vocabulary over the entire
corpus of tokenized entity names $\\{\Phi(t_{e})\\}_{e\in\mathcal{E}}$. We
have $f_{v}=\frac{n_{v}}{\sum_{u=1}^{V}n_{u}}$ where $n_{v}$ is the number of
times $v$ appears in $\\{\Phi(t_{e})\\}_{e\in\mathcal{E}}$. We create an ald
code $c_{e}$ for each entity by keeping only the $(L-1)$ tokens with the
lowest frequencies and discarding the other ones. For example for entity
Q358813, the 3 tokens with the lowest frequencies are“col”, “ob” and “white”.
Interestingly, these 3 most discriminative tokens appear at the end of the
code for ger-caption. By contrast, they appear right at the beginning of the
code for ger-ald and they constitute the only tokens to be decoded by the
model, which intuitively explains the improved performance of ger-ald codes,
as analyzed later in Sec. 4.4.2 especially when entities have long names (see
Fig. 4). Finally an interesting by-product of using short codes is that they
are faster to decode (the complexity of decoding is $\mathcal{O}(L^{2})$) and
require less memory footprint to store.

Unambiguous codes. Note that several entities might share the same least
frequent $(L-1)^{\text{th}}$ tokens. In this case their code are exactly
identical up to the $(L-1)^{\text{th}}$ token. We use the last $L^{\text{th}}$
token to ensure that each entity has a unique code: we greedily assign the
last code token $c^{e}_{L}$ to the next least frequent word of the tokenized
entity name until the code $c_{e}$ is different from all existing codes. If
this still fails to create a unique code, we assign $c^{e}_{L}$ to a random
token value $v^{\prime}$ so that the resulting code is unique. With code
length $L=4$, only $0.5\%$ of the entities use a random token value.

###  3.3 Training

In this section, we describe the model used to decode entity codes from an
input image-text pair. Importantly, we also introduce our entity-based pre-
training to condition the generative model to the task of entity recognition.

Auto-regressive generative models. We build upon GIT [45], an auto-regressive
image-to-text generative model. The query image-text pair $(x_{v},x_{t})$ is
transformed into a set of $d$-dimensional embeddings using a visual encoder
for $x_{v}$ and the text tokenizer $\Phi(.)$ for $x_{t}$. The resulting output
is represented by $\mathbf{X}_{v}\in\mathbb{R}^{N_{v}\times d}$ (resp.
$\mathbf{X}_{t}\in\mathbb{R}^{N_{t}\times d}$) for image (resp. text) tokens.
We then input $\mathbf{X}_{v}$ and $\mathbf{X}_{t}$ to a decoder network
$g(.)$ whose task is to decode the next code token $c^{e}_{i}$, conditioned on
the previous tokens $c^{e}_{j<i}$. Each code token value $v$ in $\llbracket
1,V\rrbracket$ maps to a learnable $d$-dimensional vector $\mathbf{Y}_{v}$
(gathered in the embedding matrix $\mathbf{Y}\in\mathbb{R}^{(V+1)\times d}$
where $\mathbf{Y}_{0}$ corresponds to the “beginning of code” token). We train
with a language modeling loss:

$\mathcal{L}^{e}=\frac{1}{L}\sum_{i=1}^{L}\ell(c^{e}_{i},g([\mathbf{X}_{v};\mathbf{X}_{t};\mathbf{Y}_{0};\mathbf{Y}_{c^{e}_{0<j<i}}]))\vspace{-0.3cm}$

where $[;]$ corresponds to the concatenation operation in the first dimension
and $\ell$ is the softmax cross-entropy loss with label-smoothing [27]. We
average $\mathcal{L}^{e}$ over a mini-batch and learn the weights of the
visual encoder, decoder $g(.)$ and embedding matrix $\mathbf{Y}$ through back-
propagation. When decoding, we use beam search to obtain the best predicted
entity coded. We find that we do not need to constrain the beam search to
existing codes since more than $99\%$ of the top-$1$ predictions are valid
codes for converged ger models.

Entity-based pre-training. Common auto-regressive models such as GIT [45] or
PaLI [5] are pre-trained for descriptive captioning. As shown in Tab. 5 and
Fig 9 of the Appendix, they generalize poorly to entity recognition. This is
because of the task discrepancy between predicting a descriptive caption and
predicting an entity name. In order to condition our models better for entity
recognition, we propose to collect a significant number of entity-based
pretraining images, each associated with a Wikipedia entity instead of a
generic caption. However, such an entity-based pretraining dataset does not
exist. We create it in an automatic way, without any human supervision.

To do so, we leverage existing large-scale image-caption datasets [37, 38]:
unless specified otherwise we use WebLI [5]. For each Wikipedia entity, we
retrieve in WebLI the image-caption pairs that best represent this entity and
replace their original captions by this entity name [15, 23]. Specifically, we
embed the $6$M entity names of OVEN with a semantic text encoder [32] and find
the top-$k$ most similar captions in WebLI. We retrieve their corresponding
images and replace their original captions by the considered entity name. We
ensure that no image is assigned to multiple entities to avoid instability
during training. We vary the number of retrieved images $k$ per entity from
$2$ to $100$ to produce pre-training datasets of different sizes: from 11M up
to 55M images (see Fig. 6). We denote by Entity-WebLI (resp. Entity-LAION) the
resulting dataset used for entity-based pretraining, built from WebLI (resp.
LAION [38]). This way of creating pre-training data is akin to the query
generation techniques used for generative retrieval in NLP [46]. However,
rather than generating a synthetic input, we simply retrieve input images from
a large-scale dataset.

###  3.4 Baselines

We compare our method to the following different baselines.

Hierarchical classification. Solving million-scale entity recognition with
classification is unpractical due to the very large number of classes. A
workaround is to use hierarchical classifiers. As OVEN does not come with
hierarchical labels we obtain a 3-level hierarchy through k-means of the 6M
entity names encoded with sentence-T5 [28]. We train a multi-class classifier
for each parent node in the hierarchy. To avoid training a huge number of
different classification matrices, we learn a generic classifier matrix per
level which is modified by learnable small modifiers depending on the path in
the hierarchy.

Dual encoders. Another typical workaround to classification is to rely on deep
metric learning approaches [36] such as Noise Contrastive Estimation [11] and
its InfoNCE variant [29] as used in popular dual encoder approaches [32, 16].
Dual encoders learn a unified image-text feature space with separate encoders,
thereby facilitating efficient nearest neighbor searches for recognition. We
use CLIP-L/14 [32].

Visual matching. We also experiment with pure visual matching baselines. We
use off-the-shelf CLIP-L/14 visual encoder and Entity-WebLI (55M) dataset as
the memory. We use $k=500$ for nearest neighbor search with majority voting as
it obtains the best results on OVEN val set.

Captioning. We compare to Git-Large [45] or PaLI [5] image-to-text auto-
regressive captioning models.

ger-baselines: alternative code creation strategies. We compare ger-ald,
_i.e_. the best variant of ger, with several alternatives. First, ger-atomic
refers to using atomic, completely unstructured codes, i.e. each code token
$c^{e}_{i}$ is randomly drawn from $\llbracket 1,V\rrbracket^{L}$ [42] .
Second, we consider two alternatives using semantically structured codes: (i)
ger-hkc where we embed the entity names with a pretrained text encoder before
applying hierarchical k-means clustering on the resulting embeddings [42] and
(ii) ger-caption where we create a code by tokenizing the entity name with
$\Phi(.)$ [12, 6]. Details on the baselines are in Appendix Sec. 6.4.

##  4 Experiments

In this section, we detail our experimental setup, compare our method with
state of the art and baselines, and finally present thorough analyses on code
creation and pretraining.

###  4.1 Experimental setting

OVEN dataset consists of 6,063,945 different entities [12]. We evaluate the
models on the validation and test splits, by reporting the harmonic mean (HM)
of top-1 accuracy scores between “seen” and “unseen” entities. Seen are
entities present in the OVEN training set. Unseen entities are a subset of
entities among the ones not present in the training set. The models are
evaluated on a total of 3192 entities (1721 for seen and 1471 for unseen) for
validation and 15888 entities (8355 for seen and 7533 for unseen) for test. We
call the entities that the model is evaluated on by “positive” entities
(_i.e_. the union of the 3192 validation and 15888 test entities) and all
other entities by “negative” entities.

|  | Pretraining |  | OVEN test
---|---|---|---|---
Method | #par.(B) | dataset | #imgs |  | HM | seen | unseen
Dual encoder approaches |  |  |
$\text{CLIP}_{\text{ViT-L14}}$ | 0.42 | OpenAI | 400M |  |  5.2 | 5.6 | 4.9
$\text{CLIPfusion}_{\text{ViT-L14}}$ | 0.88 | OpenAI | 400M |  |  8.4 | 33.6 | 4.8
$\text{CLIP2CLIP}_{\text{ViT-L14}}$ | 0.86 | OpenAI | 400M |  | 11.5 | 12.6 | 10.5
Captioning approaches |  |  |
GiT-Large | 0.40 | WebLI | 100M |  |  7.0 | 17.6 | 4.3
PaLI-3B | 3 | WebLI | 1B |  |  9.1 | 19.1 | 6.0
PaLI-17B | 17 | WebLI | 1B |  | 16.0 | 28.3 | 11.2
Generative entity recognition |  |  |
$\text{{ger-ald}}^{\ddagger}$ (Ours) | 0.40 | Entity-LAION | 41M |  | 20.9 | 29.1 | 16.3
ger-ald (Ours) | 0.40 | Entity-WebLI | 55M |  | 22.7 | 31.5 | 17.7
Table 1:  Comparison with state-of-the-art approaches on OVEN entity test
split. We report the harmonic mean (HM) of the seen and unseen splits (top-1
accuracy) after finetuning on OVEN training set. Numbers are taken from [12]
except methods based on GiT-Large which are run by us. We indicate the total
number of parameters of each model (“# par.”) in billion and the pretraining
dataset details. $\ddagger$: use only publicly available data.

Pretraining and finetuning. Unless specified otherwise, we pretrain our models
on the entity-WebLI dataset, which we create considering all 6M entity names
as described in Sec. 3.3. After this entity-based pretraining, the models are
finetuned on OVEN training set which consists only of the “seen” entities. All
implementation details are in Sec. 6 in Appendix and code is released in the
scenic library [7].

Preventing data leakage. We remove pretraining images from Entity-WebLI and
Entity-LAION with a cosine similarity (with CLIP-L/14 visual features) above
0.95 with any of the OVEN test or val images. We chose a 0.95 conservative
threshold by looking at some examples: similarity 0.95 corresponds to
conceptually similar images but clearly not duplicates (see Fig. 8 in
Appendix).

###  4.2 Comparison with the state of the art

In Tab. 1, we compare the performance of ger-ald, our best ger variant, on the
OVEN entity benchmark with previously published numbers after finetuning on
the OVEN training set. We see that our method outperforms previously proposed
approaches by significant margins. Notably, ger-ald improves over the
captioning model PALI-17B by $+6.8$ top-1 HM test accuracy (a relative
improvement of $43\%$) while using $42\times$ less parameters.

###  4.3 Comparison with baselines

In Tab. 2, we compare ger-ald with the different baselines described in Sec.
3.4. All baselines use exactly the same pretraining dataset entity-based WebLI
(55M) and model architectures of comparable sizes.

Comparing ger to different paradigms. We see in Tab. 2 that ger outperforms
strong captioning, dual-encoder, visual matching and hierarchical
classification baselines, affirming its advantage in tackling web-scale visual
entity recognition. Our superior performance compared to dual encoders aligns
with previous works observing that CLIP struggles for fine-grained recognition
[12, 15]. Due to query image and entity name similarities being captured only
through a vector dot product, potentially fine-grained interactions are
missed. Also, ger offers significant advantages over dual encoders: its
computational complexity is not a function of entity set size and it does not
require to store entity dense embeddings.

Different ger variants. In Tab. 2, we compare different variants of ger: one
variant using unstructured codes (ger-atomic) and three variants using
semantically-structured codes: ger-caption, ger-hkc and ger-ald. We observe
that ger-ald is the best performing variant, both after entity-based
pretraining and after finetuning on the OVEN seen entities. Compared to ger-
caption, ger-ald use codes that are more discriminative and compact, which
improves the performance particularly for entities with long names (see Sec.
4.4.2). Compared to ger-atomic, ger-ald codes yield a semantic structure which
is crucial for million-scale label-space as shown in Sec. 4.4.1. ger-hkc model
also gets strong performance but relies on an off-the-shelf semantic text
encoder which makes the approach more complex and costly compared to ger-ald.
ger-hkc is a first step towards learning codes and we hope future works will
propose original and better code creation strategies [41].

|  | Entity-based pretraining |  | \+ finetuning on seen
---|---|---|---|---
Method |  | HM | seen | unseen |  | HM | seen | unseen
Dual encoders |  |  9.2 | 8.9 | 9.4 |  | 16.3 | 24.3 | 12.3
Visual matching |  | 16.2 | 15.5 | 17.1 |  | 16.4 | 15.7 | 17.2
Captioning |  | 13.2 | 13.1 | 13.3 |  | 16.8 | 25.9 | 12.5
Hierarchic. classif. |  | 14.7 | 14.8 | 14.6 |  | 21.8 | 29.6 | 17.2
Generative entity recognition |
ger-atomic |  | 15.9 | 15.3 | 16.7 |  | 20.1 | 26.2 | 16.3
ger-caption |  | 14.3 | 16.5 | 12.6 |  | 20.7 | 26.8 | 16.9
ger-hkc |  | 15.8 | 15.5 | 16.0 |  | 21.0 | 25.2 | 17.9
ger-ald |  | 17.7 | 18.3 | 17.2 |  | 22.7 | 31.5 | 17.7
Table 2:  Baseline comparisons. All baselines use exactly the same pretraining
dataset Entity-WebLI (55M) and architectures of comparable number of
parameters ($\sim 400$M). All numbers are obtained with finetuning on seen
split after entity-based pretraining. We report the Harmonic Mean of top-1
accuracy on OVEN test.

###  4.4 Analysis and ablation study

In this section, unless specified otherwise, we report the accuracy on the
OVEN validation set [12] evaluated after pretraining on Entity-WebLI (27M),
_i.e_. no OVEN finetuning.

####  4.4.1 Semantic versus atomic codes

In Fig. 3 (and Appendix Tab. 6), we report the relative improvement of
semantically-structured codes (ger-ald) compared to unstructured codes (ger-
atomic). We vary pretraining data size, model capacity and label-space size. A
relative improvement of $100\%$ means that the performance of ger-ald doubles
compared to ger-atomic.

Figure 3: Semantic vs atomic codes. We report the relative improvement in $\%$
of ger-ald compared to ger-atomic in 3 scenarios: (i) limited pretraining
data, (ii) limited model capacity and (iii) massive-scale label-space. Plots
share a common experiment shown by $\mdblksquare$ which uses a pretraining
dataset size of $27M$, Large model and 6M entity set. The setting reported in
Tab. 2 is ★.

Limited pretraining data. In Fig. 3 (left), we see that semantic codes
outperform atomic codes when the amount of data available for pretraining
diminishes. In fact, the results reported in Tab. 2 corresponds to the most
favorable scenario for ger-atomic with 55M pretraining datapoints (represented
by ★ in Fig. 3). The relative improvement in this case is still of $14\%$
while it grows to more than $1000\%$ when the amount of data is reduced by
$5\times$.

Figure 4: Accuracy per entity name length for ger-ald versus ger-caption
codes. (left): Accuracy averaged per entity name length. (right): Qualitative
examples of predictions for long entity names. Code tokens are symbolized
between brackets.  Figure 5: ald versus captioning codes. (left): Effect of
different code lengths for ger-ald and ger-caption codes. (right): Cumulative
distribution function (CDF) of (in green) the position of the least frequent
token in the tokenized entity name and of (in pink) the length of tokenized
entity name.

Limited model capacity. In Fig. 3 (middle), we see that the model struggles to
learn unstructured codes when its capacity is reduced. When considering the
small version of our model (114M parameters), the performance with atomic
codes is very poor: $0.7$ top-1 accuracy.

Web-scale label-space. In Fig. 3 (right), we vary the number of entities for
pretraining. The “positive” entities (see Sec. 4.1) are always included in the
pretraining set and the amount of “negative” entities is increased,
effectively acting as distractors. First, we see in Fig. 3 (right) that for
relatively small-scale label-space ($\leq 100k$), the benefit of having
semantic codes versus atomic is small. In this regime we find that the model
can memorize all the entities without the need for semantic structure between
them. This aligns with the findings of DSI [42]. We evaluate ger further in
small label-spaces in Sec. 4.5. However, we see that in million-scale label-
space regime, semantic structure becomes important and significantly improves
the performance compared to atomic codes: $+26\%$ relative improvement.

Overall, we find that ger-atomic fail to learn unstructured codes when the
amount of pretraining data or architecture capacity are reduced, or when the
label-space increases to million-scale. Unlike ger-atomic, ger-ald succeed in
these scenarios thanks to the semantic structure easing the learning. Next, we
analyze how ger-ald improves over another type of semantic codes: ger-caption
codes.

####  4.4.2 ald versus captioning codes

We analyze why unambiguous, language-based and discriminative codes (ger-ald)
are more effective for entity recognition than directly decoding the entity
name (ger-caption). In Fig. 5 (left), we report the performance of ger-ald and
ger-caption when varying the length $L$ of the codes. Fixing a code length $L$
to a caption corresponds to keeping only the first $L^{\text{th}}$ tokens of
the entity name. In Fig. 5 (right), we report the cumulative distribution
functions (CDF) of (i) the position within the tokenized entity name of the
least frequent token among the entire corpus (as described in Sec. 3.2) and
(ii) the total number of tokens in the tokenized entity name ($L_{e}$ in the
notations of Sec. 3.2)).

Discriminative tokens versus number of tokens. We observe in Fig. 5 (left)
that the performance of ger-caption increases drastically from $L=2$ to $L=4$.
At the same time, we see in Fig. 5 (right) that for $L=4$, less than half of
the entity names are considered in full while more than $80\%$ of the ger-
caption codes contain the least frequent token of the entire tokenized name.
This hints that what is important for language-based codes is not to describe
the full entity name but to include its most discriminative part. We also
observe that the performance of captioning increases only moderately from
$L=4$ to $L=8$ even though the number of entities considered in full increases
drastically from $46.6\%$ to $100\%$. This confirms our intuition that
decoding all the entity name tokens does not have a major impact on the
performance as long as the most discriminative tokens are decoded. Overall,
these observations motivate the ald design of keeping only the most
discriminative tokens, which is shown in Fig. 5 to lead to improved
performance compared to decoding the full tokenized entity name.

Effect of code length for ger-ald. We see in Fig. 5 (left) that the
performance of ger-ald is the best for $L=4$. With smaller code lengths, we
need to resort to random tokens a lot to achieve unique codes (see Sec. 3.2),
which deters the performance. For example at $L=2$, more than $10\%$ of the
entities use a random code token while this percentage decreases to $0.5\%$ at
$L=4$. We also see that the performance of ger-ald decreases for code length
above $L=4$, which hints that only the few most discriminative tokens are
important while additional ones clutter the entity code. Interestingly we also
observe in Fig. 5 (left) that when considering all the tokens,ger-ald
performance is slightly below that of ger-caption. This might seem surprising
since the same amount of information is present in both cases. However we find
that when considering all the tokens, it is more difficult for the model to
decode tokens ordered by frequencies than tokens ordered syntactically.

Entities with long entity names. In Fig. 4 (left), we report the accuracy per
entity name length for both ger-ald and ger-caption finetuned models. We see
that the longer the entity name, the more ger-ald improves over captioning.
Longer entities tend to have more noise with key information further into the
code. We also show in Fig. 4 qualitative examples of entities with long entity
names (more in Fig. 12 in Appendix). In the left example, we see that ger-ald
use the token combination [col][ob] to represent the semantic concept of
colobus monkey species. The last token is used to efficiently differentiate
between sub-species of colobus. This compact and discriminative way of
encoding the entity allows ger-ald to successfully predict this entity whereas
ger-caption fails to generate the entity tokenized name.

Selection strategy | HM
---|---
Least frequent tokens | 14.4
Most frequent tokens | 12.3
First tokens | 12.0
Random tokens | 11.3

Tokens order | HM
---|---
Least frequent first | 14.4
Syntax order | 14.4
Random order | 13.0
Least frequent last | 12.7

Table 3: Ablation study of ger-ald codes. (left) Word tokens selection.
(right) Tokens order. All variants use $L=4$. Default is in top rows. Non
language-based ger-atomic gets $11.4$ top-1.

Dataset | Codes | HM
---|---|---
WebLI | WebLI caption | 1.8
Entity-WebLI (55M) | WebLI caption |  12.9 (+11.1)
Entity-WebLI (55M) | Entity name |  14.8 (+1.9)
Entity-WebLI (55M) | ALD |  17.5 (+2.7)

Figure 6: Entity-based pretraining ablation. (left): Validation OVEN accuracy.
(right): Examples of original WebLI captions versus corresponding OVEN entity
names.

Figure 7: Pretraining. We vary the size of the pretraining dataset by changing
the amount of retrieved examples from WebLI for each OVEN entity (see Sec.
3.3).

####  4.4.3 Creating codes with ald

Least frequent tokens. In Tab. 3 (left), we validate our choice of selecting
the least frequent tokens by evaluating 3 alternatives: random choice, most
frequent tokens and first-appearing tokens in tokenized entity name. We see
that these alternative strategies hurt the performance significantly.
Qualitative examples in Appendix Fig. 11 show that the kept tokens are less
semantic and discriminative compared to ger-ald strategy of keeping the least
frequent tokens. Note that all these variants are at least as good as ger-
atomic ($11.4$ top-1) which is not based on language at all.

Decoding order. In Tab. 3 (right), we vary the order of the first $L-1$ tokens
in ger-ald codes. Instead of decoding tokens from least to most frequent, we
evaluate most to least frequent, syntax order and random order. Note that the
selected tokens are the same in all variants, only their order changes. We see
that both “least frequent first” and “syntax” orders achieve the best of
performance.

####  4.4.4 Entity-based pretraining

Entity-based pretraining. In Fig. 6, we analyze why our entity-based
pretraining improves over the standard captioning pretraining of PaLI or GiT
models. First, we see that our method of selecting WebLI data relevant to OVEN
entities drastically improves the performance (+11.1 in Fig. 6 (left)). This
is because, by design, we select image-text pairs from WebLI that have
captions similar to OVEN entity names. Hence, this data is directly relevant
for the OVEN entity recognition benchmark. Second, we see that replacing the
original WebLI caption with its corresponding entity name from OVEN leads to
superior performance (+1.9). We see in the qualitative examples of Fig. 6
(right) that original captions contain a lot of descriptive information not
directly relevant to the entity. Lastly, we confirm that using ger-ald codes
is better (+2.7) than tokenized entity name.

Dataset size. In Fig. 7, we evaluate the effect of the pretraining dataset
size for ger models. We control the dataset size by varying the amount of
retrieved examples from WebLI for each of the OVEN entities (see Sec. 3.3). We
see in Fig. 7 that ger-ald, ger-caption and ger-atomic benefit greatly from
more data and do not seem to have reached saturation yet. As analyzed in Sec.
4.4.1, ger-atomic fails when the amount of pretraining data decreases.

###  4.5 Link with classification

Method | ImageNet-LT | WebVision
---|---|---
Classif. MLP | 74.3 | 80.9
ger-atomic $L=2$ | 80.8 | 84.7
ger-ald $L=2$ | 80.9 | 84.8
ger-atomic $L=1$ ($\sim$ Classif. MAP) | 81.0 | 84.8
Previously published numbers
NCR [13] | – | 76.8
CurrNet [10] | – | 79.3
PEL [40] | 78.3 | –
MAM [14]$\dagger$ | 82.3 | 83.6
Table 4: Evaluation of classification models and ger on small-scale label-
spaces. $\dagger$ indicates the use of additional data.

A typical way of tackling visual entity recognition is by training a
classifier into the number of entities [35]. This is not a viable solution for
web-scale problems such as OVEN where a single fully-connected layer for a
$6$M classes has an enormous parameter count of $4.6$B. In this section, we
evaluate ger in cases where learning a classification model is a feasible
choice (smaller number of classes). Classification can be cast in our ger
framework simply by setting $L=1$ and $V=|\mathcal{E}|=$ number of classes
(see Sec. 3.1), making it a special case of atomic codes with $L=1$. Since the
decoder decodes a single token, it is equivalent to a multi-layer Multihead
Attention Pooling (MAP) head [48, 21]. In Tab. 4, we consider two challenging
classification datasets: long-tailed ImageNet-LT [24] and noisy Webvision
[22]. We evaluate ger-{ald, atomic} and a classification baseline using multi-
layer perceptron (MLP) on averaged-pooled patch tokens. Implementation details
are in Sec 6.3 in Appendix.

We see in Tab. 4 that using ger-atomic instead of standard MLP improves
significantly the performance of the classification model (74.3 versus 81.0
for ImageNet-LT). We also observe that ger-atomic and ger-ald have comparable
performance in this relatively small label-space regime (1k classes). As a
matter of fact, this achieves state-of-the-art accuracy for both datasets
(when no additional external data is used). This shows that ger framework not
only excels for large-scale scenarios, but also works well in datasets with
smaller number of visual entities, making ger a general framework for visual
entity recognition.

##  5 Conclusion

In this work, we propose a novel generative framework for web-scale visual
entity recognition. We represent each entity by a compact, discriminative and
semantic code that a generative auto-regressive model learns to decode. In
future work, we will explore ways of creating better entity codes by
leveraging additional information: either from the Wikipedia page such as the
description of the entity and its attached image or also by using external
tools.

##### Acknowledgement.

We thank Xingyi Zhou, Ziniu Hu and Armand Joulin, as well as our teammates for
their precious help, support and discussions around this project.

## References

  * Agrawal et al. [2013] Rahul Agrawal, Archit Gupta, Yashoteja Prabhu, and Manik Varma.  Multi-label learning with millions of labels: Recommending advertiser bid phrases for web pages.  In _Proceedings of the 22nd international conference on World Wide Web_ , pages 13–24, 2013.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Bengio et al. [2019] Samy Bengio, Krzysztof Dembczynski, Thorsten Joachims, Marius Kloft, and Manik Varma.  Extreme Classification (Dagstuhl Seminar 18291).  _Dagstuhl Reports_ , 2019.
  * Bossard et al. [2014] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In _ECCV_ , 2014.
  * Chen et al. [2023] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al.  Pali: A jointly-scaled multilingual language-image model.  _ICLR_ , 2023.
  * De Cao et al. [2020] Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni.  Autoregressive entity retrieval.  _arXiv preprint arXiv:2010.00904_ , 2020.
  * Dehghani et al. [2021] Mostafa Dehghani, Alexey Gritsenko, Anurag Arnab, Matthias Minderer, and Yi Tay.  Scenic: A JAX library for computer vision research and beyond.  _arXiv preprint arXiv:2110.11403_ , 2021.
  * Everingham et al. [2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  _IJCV_ , 88, 2010.
  * Fei-Fei et al. [2004] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In _CVPR_ , 2004.
  * Guo et al. [2018] Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang.  CurriculumNet: Weakly supervised learning from large-scale web images.  In _ECCV_ , pages 135–150, 2018.
  * Gutmann and Hyvärinen [2010] Michael Gutmann and Aapo Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 2010.
  * Hu et al. [2023] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang.  Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities.  _ICCV_ , 2023.
  * Iscen et al. [2022] Ahmet Iscen, Jack Valmadre, Anurag Arnab, and Cordelia Schmid.  Learning with neighbor consistency for noisy labels.  In _CVPR_ , 2022.
  * Iscen et al. [2023] Ahmet Iscen, Alireza Fathi, and Cordelia Schmid.  Improving image recognition by retrieving from web-scale image-text data.  _CVPR_ , 2023.
  * Iscen et al. [2024] Ahmet Iscen, Mathilde Caron, Alireza Fathi, and Cordelia Schmid.  Retrieval-enhanced contrastive vision-text models.  _ICLR_ , 2024.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _ICML_ , 2021.
  * Khosla et al. [2011] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei.  Novel dataset for fine-grained image categorization.  In _First Workshop on Fine-Grained Visual Categorization, CVPR_ , 2011\.
  * Krause et al. [2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _ICCV_ , 2013.
  * Kudo [2018] Taku Kudo.  Subword regularization: Improving neural network translation models with multiple subword candidates.  _arXiv preprint arXiv:1804.10959_ , 2018.
  * Kudo and Richardson [2018] Taku Kudo and John Richardson.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  _arXiv preprint arXiv:1808.06226_ , 2018.
  * Lee et al. [2019] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh.  Set transformer: A framework for attention-based permutation-invariant neural networks.  In _International conference on machine learning_ , 2019.
  * Li et al. [2017] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool.  Webvision database: Visual learning and understanding from web data.  _arXiv preprint arXiv:1708.02862_ , 2017.
  * Liu et al. [2023] Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li.  Learning customized visual models with retrieval-augmented knowledge.  In _CVPR_ , 2023.
  * Liu et al. [2019] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu.  Large-scale long-tailed recognition in an open world.  In _CVPR_ , 2019.
  * Mehta et al. [2022] Sanket Vaibhav Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Q Tran, Jinfeng Rao, Marc Najork, Emma Strubell, and Donald Metzler.  Dsi++: Updating transformer memory with new documents.  _arXiv preprint arXiv:2212.09744_ , 2022.
  * Mittal et al. [2022] Anshul Mittal, Kunal Dahiya, Shreya Malani, Janani Ramaswamy, Seba Kuruvilla, Jitendra Ajmera, Keng-hao Chang, Sumeet Agarwal, Purushottam Kar, and Manik Varma.  Multi-modal extreme classification.  In _CVPR_ , 2022.
  * Müller et al. [2019] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  _Advances in neural information processing systems_ , 32, 2019.
  * Ni et al. [2021] Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang.  Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.  _arXiv preprint arXiv:2108.08877_ , 2021.
  * Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Pradeep et al. [2023] Ronak Pradeep, Kai Hui, Jai Gupta, Adam D Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Q Tran.  How does generative retrieval scale to millions of passages?  _arXiv preprint arXiv:2305.11841_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Rajput et al. [2023] Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan H Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q Tran, Jonah Samost, et al.  Recommender systems with generative retrieval.  _arXiv preprint arXiv:2305.05065_ , 2023.
  * Robertson et al. [2009] Stephen Robertson, Hugo Zaragoza, et al.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 2009.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 2015.
  * Schroff et al. [2015] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In _CVPR_ , 2015.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _arXiv preprint arXiv:2210.08402_ , 2022.
  * Sennrich et al. [2015] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  _arXiv preprint arXiv:1508.07909_ , 2015.
  * Shi et al. [2023] Jiang-Xin Shi, Tong Wei, Zhi Zhou, Xin-Yan Han, Jie-Jing Shao, and Yu-Feng Li.  Parameter-efficient long-tailed recognition.  _arXiv preprint arXiv:2309.10019_ , 2023.
  * Sun et al. [2023] Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten Rijke, and Zhaochun Ren.  Learning to tokenize for generative retrieval.  _NeurIPS_ , 2023.
  * Tay et al. [2022] Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al.  Transformer memory as a differentiable search index.  _Advances in Neural Information Processing Systems_ , 2022.
  * Van Horn et al. [2018] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In _CVPR_ , 2018.
  * Wah et al. [2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * Wang et al. [2022a] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  _arXiv preprint arXiv:2205.14100_ , 2022a.
  * Wang et al. [2022b] Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, et al.  A neural corpus indexer for document retrieval.  _NeurIPS_ , 2022b.
  * Weyand et al. [2020] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim.  Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval.  In _CVPR_ , 2020.
  * Zhai et al. [2022] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In _CVPR_ , 2022.
  * Zhang et al. [2023] Yidan Zhang, Ting Zhang, Dong Chen, Yujing Wang, Qi Chen, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, Fan Yang, et al.  Irgen: Generative modeling for image retrieval.  _arXiv preprint arXiv:2303.10126_ , 2023.
  * Zhu et al. [2022] Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Dalong Du, Jiwen Lu, et al.  Webface260m: A benchmark for million-scale deep face recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.

Supplementary Material

|  | Zero-shot |  | Finetuned on OVEN
---|---|---|---|---
Method |  | HM | Seen | Unseen |  | HM | Seen | Unseen
PaLI-17B [5] |  |  1.8 | 4.4 | 1.2 |  | 16.0 | 28.3 | 11.2
GiT-Large [45] |  |  1.7 | 4.1 | 1.2 |  |  7.0 | 17.6 | 4.3
Table 5:  Transferring captioning models to OVEN. We report the harmonic mean
(HM) of top-1 accuracy on the seen and unseen test splits for two captioning
models: PALI-17B [5] and GiT-Large [45]. Numbers from GiT-Large are run by us.
Note that GiT-Large has $42\times$ less parameters thank PALI-17B.

##  6 Implementation Details

We use the Large version of GIT [45] with a pretrained visual encoder and a
decoder randomly initialized. The visual encoder is pre-trained with GIT
trained for captioning on WebLI dataset [45, 5].

###  6.1 Entity-based pre-training

We use batch size of 4096, learning rate of $1e-5$ for the visual encoder and
$1\text{e}^{-4}$ for the decoder, label smoothing of $0.3$ and no weight
decay. We use standard inception crop data augmentation for the images. By
default and unless specified otherwise, we use code length $L=4$ (see Fig. 5).
Note that we only evaluate codes with $L>1$ on OVEN, as the only way to ensure
unique codes with $L=1$ is to set $V=|\mathcal{E}|$. This is equivalent to the
classification scenario and is not feasible for the million-scale label-space
of OVEN. We evaluate $L=1$ in Sec. 4.5 for datasets with a smaller label-space
of 1k entities: namely ImageNet-LT [24] and Webvision [22]. Unless specified
otherwise our models for the main results (i.e. in Sec. 4.2 and Sec. 4.3) are
trained on Entity-WebLI with 55M images ($k=100$) during 600k steps while
models for ablations are trained on Entity-WebLI with 27M images ($k=20$) for
200k steps.

Preventing data leakage. Webli is already deduplicated against the train, val,
and test splits of 68 common vision/vision-language datasets (see PaLI paper
[5]). To be sure, in our paper, we further removed pretraining images with a
cosine similarity (with CLIP-L/14 visual features) above 0.95 with any of the
OVEN images. We chose a 0.95 conservative threshold by looking at some
examples: similarity 0.95 corresponds to conceptually similar images but
clearly not duplicates (see Fig 8).

Figure 8: Filtering out pretraining data too similar to OVEN test/val.
Algorithm 1 ger-ald codes.

Data: Code length L, Text tokenizer $\Phi(.)$, Entities $\mathcal{E}$

Result: $\mathcal{C}=\\{c_{e}\\}_{e\in\mathcal{E}}$

for  _$v\in[1,V]$_ do

2
$f_{v}=\frac{1}{\sum_{e\in\mathcal{E}}L_{e}}\sum_{e\in\mathcal{E}}\sum_{y^{e}_{i}\in\Phi(t_{e})}\mathbb{1}_{y^{e}_{i}=v}$

3 end for

4$\mathcal{C}\leftarrow\varnothing$ for  _$e\in\mathcal{E}$_ do

5 sort $\Phi(t_{e})$ by decreasing frequencies:
$\\{y^{e}_{s_{i}}\\}_{i\in[1,L_{e}]}$ such that $f_{y^{e}_{s_{i}}}\leq
f_{y^{e}_{s_{i+1}}}$ for  _$i\in[1,L-1]$_ do

6 Set $c^{e}_{i}\leftarrow y^{e}_{s_{i}}$

7 end for

8 $j=0$ while  _$c_{e}\in\mathcal{C}$ and $j\leq(L_{e}-L)$ _ do

9 $c^{e}_{L}\leftarrow y^{e}_{s_{L+j}}$ $j=j+1$

10 end while

11 while  _$c_{e}\in\mathcal{C}$_ do

12 $c^{e}_{L}\leftarrow v^{\prime}\sim[1,V]$

13 end while

14 $\mathcal{C}\leftarrow\\{c_{e}\\}\bigcup\mathcal{C}$

15 end for

###  6.2 Finetuning on OVEN train set

We finetune models on OVEN training set for 30,000 steps with a batch size of
256 and a learning rate of $1\text{e}^{-7}$. Label smoothing is set at $0.1$.
Note that the finetuning schedule is relatively short (30,000 steps) because
we observe that long finetuning (or equivalently, using a large learning rate)
causes the model to forget about the unseen categories.

###  6.3 Training on ImageNet-LT and Webvision

We train the model on ImageNet-LT and Webvision datasets with the batch size
of 512 and a learning rate of $1\text{e}^{-4}$ for both encoder and decoder.
We do not use any label smoothing but apply a dropout of $0.1$. We use $L=2$
for these experiments with ger-ald because unlike very large label-space this
is enough not to resort to random tokens when ensuring that codes are
unambious.

Figure 9: Zero-shot versus finetuned captioning models predictions. We
qualitatively compare the predictions of the captioning GiT-Large model when
evaluated on OVEN in a zero-shot manner or after finetuning on OVEN train set.

###  6.4 Implementation details about the baselines

##### Dual encoder with CLIP-L/14.

We use a learning rate of $3\text{e}^{-7}$, a batch size of $4096$ and we
train for $200,000$ steps since training for longer deteriorates the
performance. During finetuning on OVEN training set, we find it important to
still include some pretraining data: we randomly sample, with a probability of
$90\%$, elements from the pretraining dataset. Otherwise, when finetuning
solely on OVEN, the model becomes too specialized for the seen categories and
is not capable of discriminating between all the negative entities.
Alternatives could be to freeze some layers of the network during finetuning
to prevent catastrophic forgetting.

#####  ger-atomic.

We benchmark different values for the choice of $L$: {$2$, $4$, $8$} and $V$:
{$512$, $4096$, $32768$} when using atomic codes. Our default is to use $L=2$
and $V=4096$. Note that this corresponds to more than $16$M possible different
codes and we use only a subset of $6$M of those unique codes.

#####  ger-hkc.

For HKC, we first represent each Wikipedia entity with a text embedding using
the sentence-t5 [28] encoder. We have experimented with different ways of
creating such embeddings, such as creating text embeddings from the Wikipedia
titles, Wikipedia article summaries, and Wikipedia title and article summaries
combined together. We have observed that Wikipedia title and article summaries
combined together produces the best embeddings. We have tried different values
of $k$: {$10$, $100$, $1000$, $4096$, $8142$}, and found that $k=4096$
achieves the best performance in the validation set.

##  7 More Experimental Results

###  7.1 Failure case analyses

We show in Fig. 10 that our method works well across different entity types.
In many cases, our codes for animals (_e.g_. ‘Glaucous winged gull’), persons
(_e.g_. ‘List of celebrities who own wineries and vineyards’) or organizations
(_e.g_. Gladney Center for Adoption’) are interpretable, as shown in the
qualitative examples in Fig 11 and Fig 12. We see failure cases when semantics
is difficult to infer from entity name alone. This is often the case for
scientific denomination of species as show in Fig. 10. In future work, using
external tools or Wikipedia page content could improve results in such cases.

Figure 10: (left): Accuracy per sub-task in OVEN. (right): A failure case
example in iNaturalist (‘inat’) sub-task.

###  7.2 Zero-shot OVEN with captioning models

##### Quantitative evaluation.

In Table 5, we transfer two captioning models, namely PALI-17B and GiT-Large
[45], both pre-trained on WebLI [5] to the OVEN task. We observe in Table 5
that these models transfer poorly in a zero-shot manner. This can be explained
by the major discrepancy between the pre-training captioning task (_i.e_.
describing an image with a caption) and the target entity recognition task.

##### Visual examples.

We show some visual examples of predictions from the validation test between
the zero-shot and finetuned GiT-Large models in Figure 9 where we clearly see
the difference of output between the zero-shot and finetuned GiT-Large models.
In the left column, we show two examples where both zero-shot and finetuned
models fail. However, even though the finetuned model fails to find the
correct category of castle or plant, it still tries to output a fine-grained
category of castle or plant. This is not the case of the zero-shot model which
gives a generic description of the entity, for example “The castle in the
middle ages.”. In the middle column, we show examples where zero-shot fails
but the finetuned model finds the correct category. Finally, in the last
column we show cases where the zero-shot model succeeds, but even when it does
we observe that the generated caption is cluttered (for example with “a photo
of a picture”) while the finetuned model directly outputs the entity name.

Overall, the observation that models pre-trained from WebLi captions do not
generalize well to OVEN entity recognition motivated us to create our entity-
based pre-training described in Section 3.3.

Figure 11: Token selection strategies in ger-ald. We qualitatively compare
different alternative token selection strategies for ger-ald: most frequent
token or random token selection. We use $L=2$ for this qualitative evaluation
since this is easier to visually interpret that $L=4$, however the trends are
consistent. Quantitative evaluation is in the Table 3 of the main paper.

###  7.3 Entities with long names

In Figure 12, we show more visual examples of ger-ald and ger-caption
predictions for entities with long names.

Figure 12: Qualitative study of ger-ald versus ger-caption. Visual examples of
predictions for long entity name from 9 to 16 tokens. For these visualizations
with ger-ald, we use SentencePiece tokenizer [20] and $L=2$ in this evaluation
since this leads to more visually interpretable codes. Tokens are symbolized
between brackets. We report the top-3 predictions for ger-ald and for ger-
caption codes, and color in green the correct predictions. We observe that
ger-ald codes are easier to predict as they contain less clutter than ger-
caption codes. Interestingly, we see that with ger-ald the top-3 predictions
usually share a common token which re-group different semantically close
entities.

Pretraining size (M) | 10.6 | 14.7 | 26.6 | 40.3 | 54.9
---|---|---|---|---|---
ger-atomic | 0.9 | 6.8 | 11.4 | 13.8 | 15.3
ger-ald | 10.2 | 11.7 | 14.4 | 16.1 | 17.5
Relative $\Delta$ (%) | 1029 | 71 | 26 | 17 | 14

Architecture size (M) | 114 | 179 | 397
---|---|---|---
ger-atomic | 0.7 | 5.4 | 11.4
ger-ald | 5.6 | 9.5 | 14.4
Relative $\Delta$ (%) | 648 | 77 | 26

# entities (M) | 0.02 | 0.03 | 0.12 | 1.00 | 6.08
---|---|---|---|---|---
ger-atomic | 34.6 | 34.0 | 29.7 | 21.5 | 11.4
ger-ald | 33.5 | 33.5 | 30.4 | 23.6 | 14.4
Relative $\Delta$ (%) | -3.1 | -1.6 | 2.2 | 9.6 | 26.4

Table 6: Semantically-structured (ger-ald) versus unstructured (ger-atomic)
codes. We report the numbers corresponding to Figure 3 of the main paper. The
pretraining dataset sizes of 10.6M, 14.7M, 26.6M, 40.3M and 54.9M correspond
respectively to setting $k$ to 2, 5, 20, 50 and 100. The architecture sizes
with 114M, 179M and 397M parameters correspond respectively to variant Small,
Base and Large of the model. The label space sizes with 20549, 30549, 120549,
1000549 and 6084491 different entities correspond respectively to having 0,
10k, 100k, 1M and 6M entities acting as distractors.

###  7.4 Different token selection strategies

In Figure 11, we show visual examples of codes generated with alternatives of
selecting the least frequent token in ger-ald. We compare with selecting
instead the most frequent token and with selecting a random token in the
entity name. Quantitative evaluation is in Table 3 of the main paper. In
Figure 11, we observe that codes generated with least frequent token strategy
are the most semantically structured. Indeed, in this case the entities
“Adoption by celebrities” and “List of celebrities who own wineries and
vineyards.” share a common token (the token corresponding to “celebrities”)
while there is no intersection of token between those two entities for the
most frequent or the random strategies. We observe the same effect across
several group of entities that we intuitively expect to have shared tokens,
for example with “smoking and pregnancy”, “teenage pregnancy in Australia” and
”Immunization during pregrancy”, or with “Denall National Park and Preserve”
and “Bishop Ranch Regional Preserve”.

###  7.5 Numbers corresponding to Fig. 3 main paper

We report in Table 6 the numbers corresponding to the experiments shown in
Figure 3 of the main paper.