NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
Under review as a conference paper at ICLR 2025

NV-EMBED: IMPROVED TECHNIQUES FOR TRAINING
LLMS AS GENERALIST EMBEDDING MODELS

Anonymous authors
Paper under double-blind review

ABSTRACT

Decoder-only large language model (LLM)-based embedding models are begin-
ning to outperform BERT or T5-based embedding models in general-purpose text
embedding tasks, including dense vector-based retrieval. In this work, we introduce
the NV-Embed model, incorporating architectural designs, training procedures,
and curated datasets to significantly enhance the performance of LLM as a versatile
embedding model, while maintaining its simplicity and reproducibility. For model
architecture, we propose a latent attention layer to obtain pooled embeddings,
which consistently improves retrieval and downstream task accuracy compared to
mean pooling or using the last <EOS> token embedding from LLMs. To enhance
representation learning, we remove the causal attention mask of LLMs during
contrastive training. For training algorithm, we introduce a two-stage contrastive
instruction-tuning method. It first applies contrastive training with instructions on
retrieval datasets, utilizing in-batch negatives and curated hard negative examples.
At stage-2, it blends various non-retrieval into instruction tuning, which not only
enhances non-retrieval task accuracy but also improves retrieval performance. For
training data, we utilize the hard-negative mining, synthetic data generation and
existing public available datasets to boost the performance of embedding model. By
combining these techniques, our NV-Embed-v1 model secured the No.1 position
on the Massive Text Embedding Benchmark (MTEB) (as of May 24, 2024), across
56 embedding tasks. NV-Embed-v2 has reclaimed and maintained the top spot
on MTEB since August 30, 2024, demonstrating the sustained effectiveness of
the proposed methods over time. Additionally, it achieved the highest scores in
the Long Doc section and the second-highest scores in the QA section of the AIR
Benchmark, which covers a range of out-of-domain information retrieval topics
beyond those in MTEB.

1

INTRODUCTION

Embedding or dense vector representation of text (Mikolov et al., 2013; Devlin et al., 2018) encodes its
semantic information and can be used for many downstream applications, including retrieval, rerank-
ing, classification, clustering, and semantic textual similarity tasks. The embedding-based retriever
is also a critical component for retrieval-augmented generation (RAG) (Lewis et al., 2020), which
allows LLMs to access the most up-to-date external or proprietary knowledge without modifying the
model parameters (Liu et al., 2024; Guu et al., 2020; Shi et al., 2023; Wang et al., 2023a).

The embedding models built on bidirectional language models (Devlin et al., 2018; Raffel et al.,
2020) have dominated the landscape for years (e.g., Reimers & Gurevych, 2019; Gao et al., 2021;
Wang et al., 2022; Izacard et al., 2021; Ni et al., 2021), although one notable exception is Neelakantan
et al. (2022). The recent work by Wang et al. (2023b) demonstrates that decoder-only LLMs can
outperform frontier bidirectional embedding models (Wang et al., 2022; Ni et al., 2021; Chen et al.,
2023) in retrieval and general-purpose embedding tasks.

In this work, we introduce NV-Embed, a generalist embedding model that significantly enhances the
performance of decoder-only LLMs for embedding and retrieval tasks. Specifically, we make the
following contributions:

1. For model architecture, we propose a novel latent attention layer to obtain pooled embeddings
for a sequence of tokens. In contrast to the popular average pooling in bidirectional embed-

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

Table 1: Top MTEB leaderboard models as of ICLR submission date (2024-10-01). We use the original model
names on the leaderboard for clarity.

Embedding Task
Mertric
NV-Embed-v2
Bge-en-icl (zero shot)
Stella-1.5B-v5
SFR-Embedding-2R
Gte-Qwen2-7B-instruct
NV-Embed-v1
Bge-multilingual-gemma2
Voyage-large-2-instruct
SFR-Embedding
GritLM-7B
E5-mistral-7b-instruct
Text-embed-3-large (OpenAI)

Retrieval (15) Rerank (4)
nDCG@10
62.65
61.67
61.01
60.18
60.25
59.36
59.24
58.28
59.00
57.41
56.9
55.44

MAP
60.65
59.66
61.21
60.14
61.42
60.59
59.72
60.09
60.64
60.49
60.21
59.16

Cluster. (11)
V-Meas.
58.46
57.51
57.69
56.17
56.92
52.80
54.65
53.35
51.67
50.61
50.26
49.01

PairClass. (3) Class. (12)

AP
88.67
86.93
88.07
88.07
85.79
86.91
85.84
89.24
88.54
87.16
88.34
85.72

Acc.
90.37
88.62
87.63
89.05
86.58
87.35
88.08
81.49
78.33
79.46
78.47
75.45

STS (10)
Spear.
84.31
83.74
84.51
81.26
83.04
82.84
83.88
84.58
85.05
83.35
84.66
81.73

Summ.( 1) Avg. (56)

Spear.
30.7
30.75
31.49
30.71
31.35
31.2
31.2
30.84
31.16
30.37
31.4
29.92

72.31
71.24
71.19
70.31
70.24
69.32
69.88
68.28
67.56
66.76
66.63
64.59

ding models (e.g., Wang et al., 2022) and the last <EOS> token embedding in decoder-only
LLMs (Neelakantan et al., 2022; Wang et al., 2023b), our proposed pooling technique consis-
tently improves the accuracy of retrieval and other downstream tasks. To further enhance the
representation learning, we remove the causal attention mask during the contrastive training
of decoder-only LLM, resulting in solid improvements. Our design is simpler yet more
effective compared to recent related work (BehnamGhader et al., 2024; Muennighoff et al.,
2024), which involves an additional training phase with masked token prediction or a mixed
training objective.

2. For model training, we introduce a two-stage contrastive instruction-tuning method, starting
with the pretrained Mistral-7B (Jiang et al., 2023). In the first stage, we apply contrastive
training with instructions on retrieval datasets, utilizing in-batch negative and curated hard-
negative examples. In the second stage, we blend carefully curated non-retrieval datasets into
the stage-one training data. Since in-batch negative samples are misleading for non-retrieval
tasks in some cases, we disable in-batch negative training in stage two. This design not only
improves the accuracy of classification, clustering, and semantic textual similarity tasks, but
also surprisingly enhances retrieval performance. Note, our model is also not fine-tuned from
existing embedding models1.

3. Training data is one of the most crucial factors in achieving state-of-the-art results. We provide
a detailed recipe on the curation of training datasets, including dataset-specific information,
the positive-aware hard-negative mining technique to enhance contrastive training, the syn-
thetic data generation and example-based multi-class labeling. This enables the community
to easily reproduce and even surpass our model, ultimately advancing the development of the
embedding models.

4. Our NV-Embed-v1 model obtained the No.1 position on the Massive Text Embedding
Benchmark (MTEB) (as of May 24, 2024) (Muennighoff et al., 2022) across 56 embedding
tasks. By improving the curation of the training data, NV-Embed-v2 model set a new
record high score of 72.31 and reclaimed the No. 1 spot (as of Aug 30, 2024) on the highly
competitive MTEB leaderboard, further demonstrating the sustained effectiveness of our
approach. Note that our model also attains the highest scores in 15 retrieval tasks (commonly
referred to as BEIR (Thakur et al., 2021)), 11 clustering tasks, and 12 classification tasks in the
MTEB benchmark. See Table 1 for detailed information. Additionally, it secured the highest
scores in Long Doc section and the second scores in QA section on the AIR-Benchmark
which covers a range of out-of-domain information retrieval topics beyond those in MTEB.

We organize the rest of the paper in the following. In § 2, we discuss the related work. We present
the architectural and training method in § 3. We provide detailed recipe of training data curation in
§ 4. We present the experiment results in § 5 and conclude the paper in § 6. AIR-bench results are
shown in § A.

2 RELATED WORK

2.1 BIDIRECTIONAL EMBEDDING MODELS

BERT (Devlin et al., 2018) or T5 (Raffel et al., 2020)-based embedding models have long been
the dominant approaches for general-purpose embedding tasks. Early examples include Sentence-

1For example, SFR-Embedding and Linq-Embed are fine-tuned from E5-mistral-7b-instruct.

2

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

BERT (Reimers & Gurevych, 2019) and SimCSE (Gao et al., 2021), which finetune BERT on natural
language inference (NLI) datasets. In general, these embedding models are first initialized from
pre-trained BERT (Wang et al., 2022; Izacard et al., 2021) or T5 encoders (Ni et al., 2021). Then,
they are further pre-trained with contrastive learning on curated unsupervised (Izacard et al., 2021)
or weakly-supervised text pairs (Wang et al., 2022). Finally, the embedding models (Li et al., 2023;
Wang et al., 2022; Ni et al., 2021; Chen et al., 2023) are fine-tuned on a variety of supervised data,
including MS MARCO (Nguyen et al., 2016), for retrieval and other downstream tasks. Note that
all the state-of-the-art embedding models are trained in this supervised manner. Some of the most
recent frontier models in this category include mxbai-embed-large-v1 (Lee et al., 2024b) (MTEB:
64.68), UAE-Large-V1 (Li & Li, 2023) (MTEB: 64.64), and voyage-large-2-instruct (Voyage-AI,
2024) (MTEB: 68.28).

2.2 DECODER-ONLY LLM-BASED EMBEDDING MODELS

Decoder-only LLMs (Brown et al., 2020) were believed to underperform bidirectional models on
general-purpose embedding tasks for years, because: i) unidirectional attention limits the representa-
tion learning capability, and ii) the scaling of LLMs leads to very high-dimension embeddings, which
may suffer from the curse of dimensionality.

The early work by Neelakantan et al. (2022) initializes embedding models using pre-trained, decoder-
only GPT-3 models (Brown et al., 2020) and applies continued contrastive training. The hidden state
from the final layer, corresponding to the special token <EOS> at the end of the sequence, is used
as the embedding for the input sequence. Its latest successor, text-embedding-3-large, achieves an
MTEB score of 64.59 (OpenAI, 2024). Most recently, E5-Mistral (Wang et al., 2023b) (MTEB:
66.63) applies contrastive learning with task-specific instructions on Mistral 7B (Jiang et al., 2023).
It begins to outperform the state-of-the-art bidirectional models on comprehensive embedding
benchmarks (Muennighoff et al., 2022) by utilizing a massive amount of synthetic data from the
proprietary GPT-4 model. LLM2Vec (BehnamGhader et al., 2024) (MTEB score: 65.01) tries to
build the embedding model from LLMs while only using public available data, but it is still worse
than E5-Mistral.

Given the success of E5-Mistral, SFR-Embedding-Mistral (Meng et al., 2024b) (MTEB: 67.56) and
SFR-Embedding-2R (Meng et al., 2024a) (MTEB: 70.31) further fine-tunes this model on the blend
of non-retrieval and retrieval datasets for improved accuracy on both tasks, which is closely related
to our NV-Embed. However, there are the following key differences: 1) NV-Embed is trained
from scratch on Mistral 7B LLM directly using public available data, and not dependent on other
embedding model or proprietary synthetic data. Consequently, we introduce a new architecture that
eliminates unnecessary causal attention mask and further improves the sequence pooling mechanism
with latent attention layer. 2) SFR-Embedding-Mistral uses task-homogeneous batching, which
constructs batches consisting exclusively of samples from a single task. In contrast, our NV-Embed
uses well-blended batches consisting samples from all tasks to avoid potential “zigzag” gradient
updates, which leads to a new record high score on both full MTEB and retrieval tasks compared to
SFR-Embedding-Mistral.

Over the past year, MTEB has become one of the most competitive leaderboards across all AI
categories, leading to significantly increased competition among participants. Many of the recent
top-performing models (e.g., stella-1.5B-v5, gte-Qwen2-7B-instruct, bge-multilingual-gemma2,
voyage-large-2-instruct, and text-embed-3-large) have not disclosed key technical details necessary
for reproduction, particularly the blend of training data used. Among the recently disclosed works,
GritLM (Muennighoff et al., 2024) (MTEB: 65.66) unifies text embedding and generation into a single
LLM model. In addition, bge-en-icl (Li et al., 2024) (MTEB: 71.24) enhances query embeddings by
introducing few-shot examples on the query side, utilizing the in-context learning (ICL) capabilities
in text embedding tasks. This approach introduces an overhead by supplying task-relevant examples
to the query during the training process. To maintain zero-shot evaluation accuracy, both zero-shot
and few-shot samples are included during training. In our paper, we focus on comparing the zero-shot
evaluation accuracy of the bge-en-icl model to ensure the fair comparisons during the evaluation
phase.

Another area of research focuses on improving data curation processes to enhance the accuracy of
fine-tuning retrieval embedding models. Gecko (Lee et al., 2024a) (MTEB: 66.31) attempts to distill a

3

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Figure 1: The illustration of proposed architecture design comprising of decoder-only LLM followed
by latent attention layer. Latent attention layer functions as a form of cross-attention where the
decoder-only LLM output serves as queries (Q) and trainable latent array passes through the key-
value inputs, followed by MLP. Blue dotted lines indicate the two matrix multiplications involved in
QKV-attentions.

smaller bidirectional embedding model from a decoder-only LLM (Gemini et al., 2023) by generating
synthetic paired data. It refines the data quality by retrieving a set of candidate passages for each query
and relabeling the positive and hard negative passages using the LLM. Linq-embed-mistral (Kim
et al., 2024) utilized LLMs to refine data by generating, filtering, and mining negative samples.
Meanwhile, NV-Retriever (Moreira et al., 2024) introduced a positive-aware hard-negative mining
technique that considers positive relevance scores to more effectively eliminate false negatives. In
this work, we apply this positive-aware hard-negative technique to curate the samples and enhance
the contrastive training.

3 METHODS

In this section, we describe our architecture designs and two-stage instruction-tuning method.

3.1 BIDIRECTIONAL ATTENTION

The causal attention mask in decoder-only LLMs is introduced for next-token prediction task (Vaswani
et al., 2017). In principle, causal mask in decoder blocks prevents information leakage by allowing
the decoder to attend only to previous positions during auto-regressive text generation. However, it
is observed that unidirectional attention limits the model’s representation power, as evidenced by
the poor performance of GPT models compared to similarly sized BERT or T5 models on natural
language understanding benchmarks (e.g., Wang et al., 2019). In recent, LLM2Vec (BehnamGhader
et al., 2024) introduces additional training phase with a specially designed masked token prediction
to warm-up the bidirectional attention. GRIT (Muennighoff et al., 2024) utilizes a hybrid objective
with both bidirectional representation learning and causal generative training. In contrast, we simply
remove the causal attention mask of decoder-only LLM during the contrastive learning and find it
works compellingly well as demonstrated by our results. As a result, we go with simple solution.

3.2 LATENT ATTENTION LAYER

There are two popular methods to obtain the embedding for a sequence of tokens: i) mean pooling,
and ii) the last <EOS> token embedding. Previous bidirectional embedding models typically use
mean pooling (Wang et al., 2022; Izacard et al., 2021), while the last <EOS> token embedding is
more popular for decoder-only LLM based embedding models. However, both methods have certain

4

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

limitations. Mean pooling simply takes the average of token embeddings and may dilute the important
information from key phrases, meanwhile the last <EOS> token embedding may suffer from recency
bias, relying heavily on the output embedding of last token.

In this work, we propose a latent attention layer inspired by Jaegle et al. (2021) to achieve more
expressive pooling of the sequences for general-purpose embedding tasks. Specifically, we denote
the last layer hidden from decoder as the query Q ∈ Rl×d, where l is the length of sequence, and d is
the hidden dimension. They are sent to attend the latent array K = V ∈ Rr×d, which are trainable
“dictionary” used to obtain better representation, where r is the number of latents in the dictionary.
The output of this cross-attention is O ∈ Rl×d,

O = softmax(QK T )V

(1)

which is followed by a regular MLP consists of two linear transformations with a GELU activation
in between. Our model uses latent attention layer with r of 512 and the number of heads as 8 for
multi-head attention. Finally, we apply mean pooling after MLP layers to obtain the embedding of
whole sequences. See Figure 1 for an illustration. It is worth mentioning here that our approach
follows the spirit of dictionary learning to obtain better representation (e.g., Wang et al., 2018), which
is different from the Perceiver IO architecture. We compare the proposed latent attention layer with
normal self-attention and find consistent improvements in our ablation study.

3.3 TWO-STAGE INSTRUCTION-TUNING

Instruction-tuning has been widely applied for training LLM to follow instructions (Wei et al., 2021;
Ouyang et al., 2022) and to perform retrieval-augmented generation (Wang et al., 2023a; Liu et al.,
2024). It has also been recently applied for training retrievers and general-purpose embedding models
that can adapt their output embeddings with different instructions and task types (Asai et al., 2022;
Wang et al., 2023b).

To obtain a generalist embedding model that can appropriately perform on retrieval and non-retrieval
tasks (e.g., classification, clustering), we need take the characteristics of different tasks into account.
For example, the use of in-batch negatives has been demonstrated to be highly efficient for training
dense-embedding-based retrievers (e.g., Karpukhin et al., 2020), because it allows to reuse the
computation and effectively train on B2 question/passage pairs for each mini-batch with only B
questions and corresponding positive passages. However, applying in-batch negatives trick can
mislead the embedding model for classification or clustering task, as the “passages” in the mini-batch
may come from the the class and are not negatives.

Given these considerations, we introduce a two-stage instruction tuning method which first conducts
contrastive training with instructions on a variety of retrieval datasets (details are in section 4.1),
utilizing in-batch negatives and curated hard-negative examples. In the second stage, we perform
contrastive instruction-tuning on a combination of retrieval and non-retrieval datasets (details are in
section 4.2) without applying the trick of in-batch negatives. It is worth mentioning here that retrieval
task presents greater difficulty compared to the other tasks so that our training strategy focuses on
fine-tuning the model for retrieval initially. In second stage, we blend the remaining embedding tasks
into the instruction-tuning.

4 TRAINING DATA

For training data, we employ public retrieval and non-retrieval datasets and synthetically generated
samples to demonstrate our model’s capability in embedding tasks. Our training procedure incorpo-
rates both retrieval and non-retrieval tasks including classification, clustering, and semantic textual
similarity datasets.

Given a relevant query-document pair, the instructed query follows the instruction template as follows:
q+
inst = Instruct : {task_definition} Query : q+
The instruction templates for each {task_definition} are provided in Table 9 for training and
Table 10 for evaluation. Note, we mask out the instruction tokens in the output embeddings during
both training and evaluation, although they still impact the output due to self-attention. We do not
add any instruction prefix to document corpus.

(2)

5

Under review as a conference paper at ICLR 2025

4.1 PUBLIC RETRIEVAL DATASETS

We adopt the retrieval datasets as follows: MSMARCO (Bajaj et al., 2016), HotpotQA (Yang et al.,
2018), Natural Question (Kwiatkowski et al., 2019), PAQ (Lewis et al., 2021), Stack Exchange (Stack-
Exchange-Community, 2023), Natural Language Inference (Group et al., 2022), SQuAD (Rajpurkar
et al., 2016), ArguAna (Wachsmuth et al., 2018), BioASQ (Tsatsaronis et al., 2015), FiQA (Maia
et al., 2018), FEVER (Thorne et al., 2018), HoVer (Jiang et al., 2020), SciFact (Wadden et al., 2022),
NFCorpus, MIRACL (Zhang et al., 2023) and Mr.TyDi (Zhang et al., 2021).

It is important to note that certain datasets (e.g., MSMARCO) are training splits of the MTEB
Benchmark, which we follow the existing practices established by leading generalist embedding
models (Meng et al., 2024b; Wang et al., 2023b; BehnamGhader et al., 2024; Muennighoff et al.,
2024). Table 9 further provides the number of samples used for training. We demonstrate the
zero-shot generalization capability of NV-Embed on AIR-bench in A.

4.1.1 HARDNEGATIVE MINING TECHNIQUE

Embedding models are trained using contrastive learning (Gao et al., 2021), aiming to increase the
similarity between the embeddings of a query and its relevant passages (positives) while reducing
the similarity with irrelevant passages (negatives). Public retrieval datasets typically only contains
the positive query-passage pairs but do not contain its own hardnegatives, making it necessary
to mine of such negative examples. To address this, we apply the recently proposed positive-
aware hard-negative technique (Moreira et al., 2024) that considers the positive relevance scores
for better false negatives removal. Following the ablation studies in Moreira et al. (2024), we use
E5-mistral-7b-instruct (Wang et al., 2023b) as a teacher retrieval model to identify the optimal
hardnegative passages relevant to the query. We set the maximum threshold for negative scores based
on a percentage of the positive score (TopKPercPos) with a 95% margin, described as follows:
max_negative_score_threshold = pos_score * percentage_margin.

4.2 PUBLIC NON-RETRIEVAL DATASETS

Besides retrieval datasets, we utilize public non-retrieval datasets mainly from three sub-tasks in
MTEB benchmark: classification, clustering and semantic similarity (STS). We pre-process the
format of these datasets to become the compatible with retrieval datasets for contrastive training:
query q+, positive document d+ and hard negative documents {d−

0 , ..., d−

n }.

For classification, we utilize the English training splits of various datasets from MTEB Huggingface
datasets (Muennighoff et al., 2022; Lhoest et al., 2021). The classification datasets that we use
are as follows: AmazonReviews (McAuley & Leskovec, 2013a), AmazonCounterfactual (O’Neill
et al., 2021), Banking77 (Casanueva et al., 2020), Emotion (Saravia et al., 2018), IMDB (Maas
et al., 2011), MTOPDomain/MTOPIntent (Li et al., 2021), ToxicConversations (Adams et al., 2019),
TweetSentimentExtraction (Maggie, 2020), AmazonPolarity (McAuley & Leskovec, 2013b), Mas-
siveScenario/MassiveIntent (FitzGerald et al., 2022). For the Emotion and AmazonCounterfactual
classification datasets we use BM25 (Robertson et al., 2009) similarity thresholds to filter out training
data that is similar to the MTEB evaluation set.

For clustering datasets, we utilize the raw_arxiv, raw_biorxiv and raw_medrxiv datasets from MTEB
Huggingface datasets, TwentyNewsgroups (Lang, 1995), Reddit (Geigle et al., 2021), StackEx-
change (Geigle et al., 2021), RedditP2P (Reimers, 2021b) and StackExchangeP2P (Reimers, 2021a)
We filter out any training data that match the MTEB evaluation set.

The classification and clustering datasets provide examples and corresponding class/cluster labels.
The example texts extracted from the appropriate text/title/abstract field are used for the query
q+. For binary classification tasks the label texts are used as documents d+, d−. For multi-class
classification and clustering tasks, a randomly sampled example from the ground-truth class/cluster is
used for the positive document d+ and randomly sampled examples from other classes/clusters are
used for negative documents d−
k . We will present ablation experiments supporting this approach in
section 5.3.4.

For semantic textual similarity datasets, we use the training splits of three semantic similarity datasets
STS12 (Agirre et al., 2012), STS22 (Chen et al., 2022), STS-Benchmark (Cer et al., 2017) from

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

MTEB Huggingface datasets. For any pair of texts with associated relevance scores (ta, tb, score),
we create two examples (q+ = ta, d+ = tb) and (q+ = tb, d+ = ta) if score ≥ 4. We mine the hard
negatives d−
k from the pool of other texts using the same technique as section 4.1.1. Task instructions
are appended to d+, d− since they are symmmetric with the query.

4.3 SYNTHETIC TASKS DATASET

Due to the limited variety of subjects and tasks in public training datasets, the available instruction
templates for training are also restricted. To enhance task-wise generalization, we employ the
Mixtral-8x22B-Instruct-v0.1 model (MistralAI) to create a dataset consisting of 120,000 synthetic
examples across 60,000 synthetic tasks. Following a two-step prompting approach proposed by
E5-mistral-7b-instruct (Wang et al., 2023b), we adjust the prompts for Mixtral-8x22B-Instruct-v0.1
and English text. We generate only the short-long, long-short, and short-short examples (40,000 of
each), as we use public STS datasets and do not assess bitext retrieval tasks. Example prompts for
synthetic data generation can be found in Appendix 12 and 13.

5 EXPERIMENTS

5.1 EXPERIMENTAL DETAILS

In this section, we describe our detailed experimental setups. We use a parameter-efficient finetun-
ing (PEFT) method denoted as low-rank adaptation (LoRA) (Hu et al., 2021) to efficiently finetune
our proposed NV-Embed model. We chose Mistral 7B (Jiang et al., 2023) as the base decoder-only
LLM. We replace the attention mask from causal to bidirectional, and integrate the latent attention
layer with 512 latents, 4096 hidden dimension size, and 8 multi-head attentions.

We train Mistral 7B LLM model end-to-end with a contrastive loss using LoRA with rank 16, alpha 32
and dropout rate of 0.1. We use Adam optimizer with 50 warm-up steps and learning rate 2e-5 for first
stage and 1.5e-5 for second stage with linear decay. The model is finetuned with 128 batch size, where
each batch is composed of a query paired with 1 positive and 7 hard negative documents. Training
samples from different datasets in Table 9 are uniformly sampled. We train using Bfloat16, and set
the maximum sequence length as 512 tokens. The special <BOS> and <EOS> tokens are appended at
the start and end of given query and documents. The whole training is conducted in two stages where
the model is initially trained on retrieval datasets utilizing in-batch negative technique. Subsequently,
the model is trained with blended datasets with both retrieval and non-retrieval embedding tasks.

For evaluation, we assess our model using a maximum length of 512 tokens to ensure fair comparisons
with prior work (Wang et al., 2023b), which also provides evaluation results based on 512 token
limits. Evaluation instructions templates are available in Table 10.

5.2 MTEB RESULTS

We evaluate the proposed NV-Embed model on the full MTEB benchmark (Muennighoff et al.,
2022) across 56 tasks. Table 1 summarizes the averaged MTEB scores for seven sub-category tasks
compared to the frontier models on the MTEB leaderboard2. Our initial model, namely NV-Embed-
v1 get the score of 69.32 and obtain the No.1 position on the MTEB as of May 24, 2024 (detailed
benchmark scores available in Table 2). We then further improve the model through the curation
of training dataset, including adding more retrieval datasets, applying positive-aware hard-negative
mining technique, using synthetic data generation process and constructing example-based multi-class
labels. As a result, our NV-Embed-v2 model sets a new record high score of 72.31 and reclaimed
No.1 (as of Aug 30, 2024) on the highly competitive MTEB leaderboard, further highlighting the
sustained effectiveness of the proposed methods. In following sub-section 5.3, we will present the
ablation studies on the design choices regarding the model architecture, training algorithm and the
curation of training data.

Based on quantitative leaderboard results, we compare our NV-Embed with the recent frontier
embedding models. The e5-mistral-7b-instruct (Wang et al., 2023b) and google-gecko (Lee et al.,

2https://github.com/embeddings-benchmark/mteb

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Table 2: Averaged MTEB scores on seven tasks after first and second stage training using only the
publically available data and before applying the positive-aware hardnegative mining, synthetic data
and example-based multi-class labeling. The averaged score 69.32 corresponds to NV-Embed-v1.

Pool Type
Mask Type
Retrieval(15)
Rerank (4)
Clustering (11)
PairClass. (3)
Classification (12)
STS (10)
Summar. (1)
Average (56)

Pool Type
Mask Type
Retrieval (15)
Rerank (4)
Clustering (11)
PairClass. (3)
Classification (12)
STS (10)
Summar. (1)
Average (56)

EOS

First stage training
Mean

bidirect
57.70
59.76
44.75
86.17
73.17
74.96
29.28
62.68

causal
56.42
57.21
40.83
83.63
69.22
73.45
28.4
60.06

bidirect
58.42
60.02
45.97
87.45
74.62
77.47
29.72
64.00

causal
57.55
59.35
45.42
84.46
72.48
73.60
30.89
62.32

Latent-attention
causal
bidirect
59.00
57.65
59.72
59.59
45.61
45.44
82.02
87.59
72.74
73.93
78.65
79.07
30.94
30.16
63.39
64.18

Self-attention

bidirect
57.89
59.73
45.19
86.51
73.54
76.89
30.22
63.27

causal
57.21
59.51
45.07
85.74
73.32
77.55
31.59
63.11

EOS

Second stage training
Mean

bidirect
58.39
60.37
51.43
84.06
85.85
79.55
30.36
67.85

causal
56.59
59.23
49.81
80.99
85.04
79.12
29.12
66.50

bidirect
58.71
60.77
52.80
87.45
87.06
82.53
30.49
68.97

causal
57.88
60.27
51.58
82.89
86.08
81.74
31.82
68.13

Latent-attention
causal
bidirect
59.36
58.33
60.57
60.54
51.7
52.80
83.45
86.91
86.58
87.35
81.94
82.84
31.87
31.20
69.32
68.47

Self-attention

bidirect
58.64
60.5
53.34
86.12
86.76
82.38
30.105
69.10

causal
57.71
60.38
51.51
84.44
86.25
81.52
31.4
68.16

Table 3: Averaged MTEB scores on seven embedding tasks after two stage training after applying the
positive-aware hardnegative mining, synthetic data and example-based multi-class labeling. Note, the
averaged score 72.31 corresponds to NV-Embed-v2.

Pool Type
Mask Type
Retrieval (15)
Rerank (4)
Clustering (11)
PairClass. (3)
Classification (12)
STS (10)
Summar. (1)
Average (56)

EOS

Mean

bidirect
62.13
60.02
58.24
87.69
90.10
82.27
30.25
71.63

causal
60.30
59.13
57.11
85.05
90.01
81.65
32.75
70.85

bidirect
61.81
60.65
57.44
87.35
89.49
84.35
30.75
71.71

causal
61.01
59.10
57.34
87.35
89.85
84.35
30.88
71.38

Latent-attention
causal
bidirect
62.65
61.15
59.36
60.65
58.46
57.80
87.22
88.67
90.37
90.49
84.13
84.31
30.90
30.70
72.31
71.61

Self-attention

bidirect
61.17
60.67
58.24
87.69
90.10
84.22
30.93
71.61

causal
60.53
59.67
57.11
85.05
90.01
83.81
31.36
70.6

2024a) utilize proprietary synthetic data to train their model in a single stage manner. In contrast,
we recognize that retrieval task presents greater difficulty compared to the other embedding tasks
and prioritizes our training strategy on fine-tuning the model for retrieval first, followed by blending
the remaining sub-tasks into instruction-tuning, leading to substantially improved BEIR and overall
MTEB results.

SFR-Embedding-2R (Meng et al., 2024b) demonstrates competitive scores on the MTEB (70.31) and
BEIR (60.18) benchmarks by continuing to finetune the e5-mistral-7b-instruct model (Wang et al.,
2023b). However, it remains largely constrained by the architectural limitations of its parent model,
such as the causal attention mask and the last token pooling method. In contrast, our NV-Embed
model is trained starting from the Mistral 7B LLM (Jiang et al., 2023) rather than finetuning e5-
mistral-7b-instruct (Wang et al., 2023b). It features a new architecture that removes the unnecessary
causal attention mask and further improves the sequence pooling mechanism with a latent attention
layer. Table 3 and 11 provides a detailed scores of BEIR and MTEB benchmarks.

5.3 ABLATION STUDY

We conduct ablation studies to compare several training, architecture and data curation design
choices: two-stage training, bidirectional attention, latent-attention pooling method, synthetic data
and example-based multi-class labeling.

8

Under review as a conference paper at ICLR 2025

Table 4: Averaged MTEB scores on ablation studies for NV-Embed-v2: two stage training, multi-
class data labeling, positive-aware hardnegative mining and synthetically generated dataset. In the
third part of the table, HN represents hardnegative mining technique, AD means adding public
retrieval datasets and SD refers to adding synthetically generated data. In the fourth part of the
table, we also include NV-Embed-v1, which omits HN, AD, and SD in stage-one training and uses a
label-based approach in stage-two training.

Embedding Task
Single Stage (Inbatch Enabled)
Single Stage (Inbatch Disabled)
Two Stage Training
Reversed Two Stage

Section 5.3.1 Two stage training
Retrieval Rerank
60.64
60.81
60.65
60.98

Cluster.
57.67
58.31
58.46
58.22

61.25
61.37
62.65
61.91

PairClass. Class.
86.6
90.2
90.37
90.26

87.82
88.3
88.67
88.59

STS
83.7
84.5
84.31
83.07

Summ.
30.75
30.96
30.70
31.28

Avg.
70.83
71.94
72.31
71.85

Embedding Task

Section 5.3.4 Multi-class Classification and Clustering Labels in stage-two training
STS
84.25
84.31

PairClass. Class.
89.17
90.37

Cluster.
53.04
58.46

Retrieval Rerank

88.04
88.67

62.40
62.65

59.7
60.65

Label-based approach
Example-based approach

Summ.
30.77
30.70

Avg.
70.82
72.31

Section 5.3.5 Hard-negative mining and Synthetically Generated Dataset in stage-one training
Embedding Task
[S0] Without HN, Without AD, Without SD
[S1] With HN, Without AD, Without SD
[S2] With HN, With AD, Without SD
[S3] With HN, With AD, With SD

Retrieval Rerank
59.85
59.80
60.45
60.65

PairClass. Class.
90.71
90.31
90.34
90.37

Cluster.
57.95
58.01
58.16
58.46

STS
81.98
84.26
84.11
84.31

85.79
88.56
88.38
88.67

59.22
61.52
62.28
62.65

Summ.
29.87
30.36
29.95
30.70

Avg.
70.73
71.83
72.07
72.31

Label-based approach + [S0]

NV-Embed-v1
60.59

52.80

59.36

86.91

87.35

82.84

31.2

69.32

5.3.1 TWO-STAGE TRAINING

We compare the two-stage and single-stage training with and without the use of the in-batch negative
technique, as shown in Table 4. We observe that our proposed two-stage training surpasses single-
stage training because it allows the use of beneficial in-batch negatives for retrieval tasks in the
first stage, while disabling the in-batch technique for non-retrieval tasks in the second stage. In
contrast, single-stage training with in-batch negatives leads to significantly lower MTEB performance,
especially in the classification sub-task. This accuracy degradation occurs because many classification
tasks involve few-class labels (such as binary labels like True/False), meaning that the inbatch negative
labels in the batch can actually be the positive label. While single-stage training without in-batch
negatives produces more comparable results (MTEB scores: 72.31 for two-stage training vs. 71.94 for
single-stage without in-batch), two-stage training significantly outperforms in the retrieval sub-tasks
(BEIR scores: 62.65 for two-stage training vs. 61.37 for single-stage without in-batch). It is worth
highlighting here that the retrieval is considered the most crucial sub-category for the advancement of
RAG technology across the MTEB embedding tasks.

Lastly, we explore another research question: what happens if the order of two-stage training is
reversed? To examine this, we further finetune the Single Stage (Inbatch disabled) model using only
the retrieval datasets with enabling the inbatch negative technique and present the MTEB results
in Table 4. While the retrieval score increased from 61.37 to 61.91 after the reversed two-staged
training, it remains lower than the retrieval score of 62.65 achieved with our proposed two-stage
training method. Furthermore, the scores on other embedding tasks, such as Clustering and STS,
declined compared to the Single Stage (Inbatch disabled) approach. Consequently, the overall MTEB
score for Reversed Two Stage (score: 71.85) is lower than our proposed Two-Stage Training (score:
72.31) as well as the Single Stage with Inbatch disabled (score: 71.94).

5.3.2 CAUSAL ATTENTION VS. BIDIRECTIONAL ATTENTION

To examine the impact of self-attention masks in decoder-only LLM models for embedding applica-
tions, we conducted experiments comparing bidirectional and causal mask types. As illustrated in
Tables 2 and 3, the bidirectional mask consistently outperforms the causal mask based on the average
MTEB scores across 56 tasks for all pooling types. This indicates that embeddings generated with
causal attention masks are significantly less effective than those produced with bidirectional attention
masks.

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

5.3.3 POOLING METHODS

To examine the impact of different pooling methods on embedding models, we conducted experiments
comparing <EOS>-last, mean, latent-attention, and self-attention pooling types. As depicted in Tables
2 and 3, mean pooling consistently outperforms <EOS>-last token embedding based on the average
MTEB scores across 56 tasks. This difference may be due to the last <EOS> token embedding being
influenced by recency bias, showing an excessive dependence on the output of the final token.

To enhance performance beyond mean pooling, we experimented with adding the proposed latent-
attention or self-attention layer (both followed by MLP) before mean pooling to address the issue of
important information from key phrases being diluted. According to Tables 2, self-attention does
not provide additional accuracy improvements for the embedding capabilities of decoder-only LLMs
(i.e., mean pooling 68.97 vs. self-attention 69.10 on MTEB tasks). It even slightly reduces accuracy
on 15 retrieval tasks (i.e., mean pooling 58.71 vs. self-attention 58.64). Table 3 also shows the similar
trends of NV-Embed-v2. This is not surprising, as the LLM already has many self-attention layers
to learn the representation, and adding an additional one does not bring significant additive value.

In contrast, the latent-attention layer proved beneficial for majority of embedding tasks, as shown
in Table 2 and 3. Specifically, the nDCG@10 accuracy of the more challenging 15 retrieval tasks
latent-attention 62.65) in Table 3. We hypothesize that
improved (i.e., mean pooling 61.82 vs.
this is due to the "dictionary learning" provided by the latent array, which offers more expressive
representation. The latent-attention layer effectively learns output embedding representations from
decoder-only LLMs, mitigating the information dilution caused by averaging the output embeddings.

5.3.4 MULTI-CLASS CLASSIFICATION AND CLUSTERING LABELS

We compare the effect of using two possible tech-
niques for constructing positive and negative docu-
ments for multi-class classification and clustering
tasks. In label-based approach, the ground-truth
class/cluster label corresponding to the example
in the query is used as the positive document, and
other class/cluster labels are sampled for negative
documents. In example-based approach, another
example from the same class/cluster as the exam-
ple in the query is used as the positive document,
and examples from other clusters are sampled for
negative documents. We use random sampling
to get a broad coverage across labels and exam-
ples. In this work, all 11 clustering datasets and 5
muti-class classification datasets are constructed
as example-based approach. As shown in Table 4,
the example-based approach leads to significant
improvements over the label-based approach for
both classification and clustering. Table 5 further
shows the detailed ablation study of label-based
and example-based labels for classification and
clustering multi-class samples.

Table 5: Ablation study on using class/cluster
labels vs. sampled class/cluster examples as
positive and negative documents for multi-class
classification and clustering tasks.

+/- Document Format
Emotion-Classification
MassiveIntent-Classification
MassiveScenario-Classification
MTOPDomain-Classification
MTOPIntent-Classification
Arxiv-Clustering-P2P
Arxiv-Clustering-S2S
Biorxiv-Clustering-P2P
Biorxiv-Clustering-S2S
Medrxiv-Clustering-P2P
Medrxiv-Clustering-S2S
Reddit-Clustering
Reddit-Clustering-P2P
StackExchange-Clustering
StackExchange-Clustering-P2P
TwentyNewsgroups-Clustering
Average (16)

Labels Examples
90.83
84.94
90.18
98.84
88.55
53.01
49.19
45.38
42.67
37.58
36.82
59.83
72.58
79.37
48.59
58.41
64.80

93.38
86.10
92.17
99.25
94.37
55.80
51.26
54.09
49.60
46.09
44.86
71.10
74.94
82.10
48.36
64.82
69.27

5.3.5 HARDNEGATIVE MINING AND SYNTHETICALLY GENERATED DATASET

We provide a step-by-step curation of training dataset, incorporating the hard negative mining
technique (S1), additional public retrieval data (S2), and synthetically generated data (S3). As
shown in Table 4, the first step of adding the hard negative mining technique significantly boosted
retrieval accuracy, with the BEIR score increasing from 59.22 to 61.52. In the next step (S2), we
included more public retrieval datasets (HoVer, SciFact, Nfcorpus, MIRACL, Mr.Tydi) followed by
synthetically generated data. Adding the public retrieval datasets further increased the retrieval score
by 0.7 points. Finally, incorporating the synthetic dataset (S3) leads to a modest improvement in the
overall MTEB scores, raising them by 0.24 points.

10

Under review as a conference paper at ICLR 2025

540
541
542
543

544
545
546
547
548
549

550
551
552
553
554

555
556
557
558
559
560

561
562
563
564
565
566

567
568
569
570
571
572

573
574
575
576
577

578
579
580
581
582
583

584
585
586
587
588
589

590
591
592
593

6 CONCLUSION

We introduced the NV-Embed model, a decoder-only LLM designed to outperform existing bidi-
rectional models in general-purpose text embedding tasks. For model architecture, we propose a
latent attention layer to obtain expressive pooled embeddings and remove the unnecessary causal
attention mask of decoder-only LLMs. For training algorithm, we introduce a two-stage contrastive
instruction-tuning scheme to sequentially improve the embedding tasks. By leveraging carefully
curated datasets, hard-negative mining, synthetic data generation and example-based multi-class
labeling, our approach achieve the superior accuracy across diverse embedding tasks. As a result, the
series of NV-Embed models achieved and maintained the No.1 ranking on the MTEB leaderboard
and also demonstrated superior accuracy in out-of-domain tasks in AIR Benchmark. The sustained
effectiveness of NV-Embed highlights the importance of our proposed architecutre design, training
procedures and dataset curation in achieving state-of-the-art performance in the evolving landscape
of text embedding models.

REFERENCES

C.J. Adams, Daniel Borkan, Jeffrey Sorensen, Lucas Dixon, Lucy Vasserman, and Nithum Thain.

Jig-
saw unintended bias in toxicity classification, 2019. URL https://kaggle.com/competitions/
jigsaw-unintended-bias-in-toxicity-classification.

Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. SemEval-2012 task 6: A pilot on semantic
textual similarity. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz
Yuret (eds.), *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume
1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth
International Workshop on Semantic Evaluation (SemEval 2012), pp. 385–393, Montréal, Canada, 7-8 June
2012. Association for Computational Linguistics. URL https://aclanthology.org/S12-1051.

Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi,

and Wen-tau Yih. Task-aware retrieval with instructions. arXiv preprint arXiv:2211.09260, 2022.

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew
McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension
dataset. arXiv preprint arXiv:1611.09268, 2016.

Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and
Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint
arXiv:2404.05961, 2024.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.
Advances in neural information processing systems, 33:1877–1901, 2020.

Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. Efficient intent detection
with dual sentence encoders. In Proceedings of the 2nd Workshop on NLP for ConvAI - ACL 2020, mar
2020. URL https://arxiv.org/abs/2003.04807. Data available at https://github.com/PolyAI-
LDN/task-specific-datasets.

Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic
textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat,
Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens (eds.), Proceedings of the 11th
International Workshop on Semantic Evaluation (SemEval-2017), pp. 1–14, Vancouver, Canada, August 2017.
Association for Computational Linguistics. doi: 10.18653/v1/S17-2001. URL https://aclanthology.
org/S17-2001.

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual,

multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2023.

Xi Chen, Ali Zeynali, Chico Camargo, Fabian Flöck, Devin Gaffney, Przemyslaw Grabowicz, Scott Hale, David
Jurgens, and Mattia Samory. SemEval-2022 task 8: Multilingual news article similarity. In Guy Emerson,
Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, and
Shyam Ratan (eds.), Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-
2022), pp. 1094–1106, Seattle, United States, July 2022. Association for Computational Linguistics. doi:
10.18653/v1/2022.semeval-1.155. URL https://aclanthology.org/2022.semeval-1.155.

11

Under review as a conference paper at ICLR 2025

594
595
596
597

598
599
600
601
602
603

604
605
606
607
608

609
610
611
612
613
614

615
616
617
618
619
620

621
622
623
624
625
626

627
628
629
630
631

632
633
634
635
636
637

638
639
640
641
642
643

644
645
646
647

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional

transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash,
Liam Urbach, Vishesh Kakarala, Richa Singh, et al. Massive: A 1m-example multilingual natural language
understanding dataset with 51 typologically-diverse languages. arXiv preprint arXiv:2204.08582, 2022.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings.

arXiv preprint arXiv:2104.08821, 2021.

Gregor Geigle, Nils Reimers, Andreas Rücklé, and Iryna Gurevych. Tweac: transformer with extendable qa

agent classifiers. arXiv preprint arXiv:2104.07081, 2021.

Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut,
Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models.
arXiv preprint arXiv:2312.11805, 2023.

Stanford NLP Group et al. The stanford natural language inference (snli) corpus, 2022.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language

model pre-training. In International conference on machine learning, pp. 3929–3938. PMLR, 2020.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu

Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and
Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint
arXiv:2112.09118, 2021.

Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda
Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for
structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las
Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv
preprint arXiv:2310.06825, 2023.

Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. Hover: A

dataset for many-hop fact extraction and claim verification. arXiv preprint arXiv:2011.03088, 2020.

Vladimir Karpukhin, Barlas O˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and
Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906,
2020.

Junseong Kim, Seolhwa Lee, Jihoon Kwon, Sangmo Gu, Yejin Kim, Minkyung Cho, Jy yong Sohn, and
Chanyeol Choi. Linq-embed-mistral: Elevating text retrieval with improved gpt data through task-specific
control and quality refinement. linq ai research blog, 2024. URL https://getlinq.com/blog/
linq-embed-mistral/.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle
Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question
answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.

Ken Lang. Newsweeder: Learning to filter netnews. In Machine learning proceedings 1995, pp. 331–339.

Elsevier, 1995.

Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi
Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arXiv
preprint arXiv:2403.20327, 2024a.

Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lipp. Open source strikes bread - new fluffy embeddings

model, 2024b. URL https://www.mixedbread.ai/blog/mxbai-embed-large-v1.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich
Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-
intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.

Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus
Stenetorp, and Sebastian Riedel. Paq: 65 million probably-asked questions and what you can do with them.
Transactions of the Association for Computational Linguistics, 9:1098–1115, 2021.

12

Under review as a conference paper at ICLR 2025

648
649
650
651

652
653
654
655
656
657

658
659
660
661
662

663
664
665
666
667
668

669
670
671
672
673
674

675
676
677
678
679
680

681
682
683
684
685

686
687
688
689
690
691

692
693
694
695
696
697

698
699
700
701

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil,
Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani,
Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina
McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut,
Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and
Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021
Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175–184,
Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
URL https://aclanthology.org/2021.emnlp-demo.21.

Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu.

Making text embedders few-shot learners. arXiv preprint arXiv:2409.15700, 2024.

Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. MTOP: A com-
prehensive multilingual task-oriented semantic parsing benchmark. In Paola Merlo, Jorg Tiedemann, and
Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for
Computational Linguistics: Main Volume, pp. 2950–2962, Online, April 2021. Association for Computa-
tional Linguistics. doi: 10.18653/v1/2021.eacl-main.257. URL https://aclanthology.org/2021.
eacl-main.257.

Xianming Li and Jing Li. Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871, 2023. URL

https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1.

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text

embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023.

Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Mohammad Shoeybi, and Bryan Catanzaro. ChatQA: Surpassing

GPT-4 on conversational QA and RAG. arXiv preprint arXiv:2401.10225, 2024.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning
word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for
Computational Linguistics: Human Language Technologies, pp. 142–150, Portland, Oregon, USA, June
2011. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/
P11-1015.

Wei Chen Maggie, Phil Culliton. Tweet sentiment extraction, 2020. URL https://kaggle.com/

competitions/tweet-sentiment-extraction.

Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and
Alexandra Balahur. Www’18 open challenge: financial opinion mining and question answering. In Companion
proceedings of the the web conference 2018, pp. 1941–1942, 2018.

Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions
with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys ’13, pp.
165–172, New York, NY, USA, 2013a. Association for Computing Machinery. ISBN 9781450324090. doi:
10.1145/2507157.2507163. URL https://doi.org/10.1145/2507157.2507163.

Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with
review text. In Proceedings of the 7th ACM conference on Recommender systems, pp. 165–172, 2013b.

Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embedding-
2: Advanced text embedding with multi-stage training, 2024a. URL https://huggingface.co/
Salesforce/SFR-Embedding-2_R.

Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfrembedding-mistral:

enhance text retrieval with transfer learning. Salesforce AI Research Blog, 3, 2024b.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words

and phrases and their compositionality. Advances in neural information processing systems, 2013.

MistralAI. Mixtral 8x22b. URL https://mistral.ai/news/mixtral-8x22b/.

Gabriel de Souza P Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge.
NV-Retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint
arXiv:2407.15831, 2024.

Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. MTEB: Massive text embedding

benchmark. arXiv preprint arXiv:2210.07316, 2022.

13

Under review as a conference paper at ICLR 2025

702
703
704
705

706
707
708
709
710
711

712
713
714
715
716

717
718
719
720
721
722

723
724
725
726
727
728

729
730
731
732
733
734

735
736
737
738
739

740
741
742
743
744
745

746
747
748
749
750
751

752
753
754
755

Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe

Kiela. Generative representational instruction tuning. arXiv preprint arXiv:2402.09906, 2024.

Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas
Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv
preprint arXiv:2201.10005, 2022.

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS

MARCO: A human-generated machine reading comprehension dataset. 2016.

Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan,
Keith B Hall, Ming-Wei Chang, et al. Large dual encoders are generalizable retrievers. arXiv preprint
arXiv:2112.07899, 2021.

James O’Neill, Polina Rozenshtein, Ryuichi Kiryo, Motoko Kubota, and Danushka Bollegala. I wish i would
have loved this one, but i didn’t–a multilingual dataset for counterfactual detection in product reviews. arXiv
preprint arXiv:2104.06893, 2021.

OpenAI. New embedding models and api updates, 2024.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang,
Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with
human feedback. Advances in neural information processing systems, 2022.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei
Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of
machine learning research, 21(140):1–67, 2020.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine

comprehension of text. arXiv preprint arXiv:1606.05250, 2016.

Nils Reimers. Stackexchange (title, body) pairs, 2021a. URL https://huggingface.co/datasets/

flax-sentence-embeddings/stackexchange_title_body_jsonl.

Nils Reimers. Reddit (title, body) pairs, 2021b. URL https://huggingface.co/datasets/

sentence-transformers/reddit-title-body.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv

preprint arXiv:1908.10084, 2019.

Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Founda-

tions and Trends® in Information Retrieval, 3(4):333–389, 2009.

Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized
affect representations for emotion recognition. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi
Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.
3687–3697, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi:
10.18653/v1/D18-1404. URL https://aclanthology.org/D18-1404.

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and
Wen-tau Yih. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652,
2023.

Stack-Exchange-Community. Stack exchange data dump, 2023.

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous
benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663, 2021.

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for

fact extraction and verification. arXiv preprint arXiv:1803.05355, 2018.

George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R
Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. An
overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC
bioinformatics, 16:1–28, 2015.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser,
and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.

14

Under review as a conference paper at ICLR 2025

Voyage-AI. voyage-large-2-instruct: Instruction-tuned and rank 1 on mteb, 2024.

Henning Wachsmuth, Shahbaz Syed, and Benno Stein. Retrieval of the best counterargument without prior topic
knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics
(Volume 1: Long Papers), pp. 241–251, 2018.

David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi.
Scifact-open: Towards open-domain scientific claim verification. arXiv preprint arXiv:2210.13777, 2022.

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and
Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems.
Advances in neural information processing systems, 32, 2019.

Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, and Bryan Catanzaro.
Instructretro: Instruction tuning post retrieval-augmented pretraining. arXiv preprint arXiv:2310.07713,
2023a.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and
Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533,
2022.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text

embeddings with large language models. arXiv preprint arXiv:2401.00368, 2023b.

Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei
Ren, and Rif A Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech
synthesis. In International conference on machine learning, pp. 5180–5189. PMLR, 2018.

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai,
and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christo-
pher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint
arXiv:1809.09600, 2018.

Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. tydi: A multi-lingual benchmark for dense retrieval.

arXiv preprint arXiv:2108.08787, 2021.

Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li,
Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. Miracl: A multilingual retrieval dataset covering 18 diverse
languages. Transactions of the Association for Computational Linguistics, 11:1114–1131, 2023.

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

A AIR BENCHMARK

In this first appendix section, we present AIR-Bench3 (version of 24.04) that is newly released infor-
mation retrieval benchmark, incorporating the diverse and comprehensive domains such as healthcare,
law, news, book, arxiv, finance and synthetically generated samples using diverse LLMs. Importantly,
AIR-Bench can help us to understand the generalization capability of the embedding/retrieval model,
because the majority of different domain samples do not appear in MTEB benchmarks. Moreover,
the AIR-Bench is designed as a closed-book benchmark whose ground truth is kept confidential. As
a result, the benchmark score can be only obtained through the HuggingFace Hub platform.

In AIR-Benchmark 24.04 version, there are two tasks: QA and Long-Doc. We run evaluations on
8 English datasets in QA task and 15 English datasets on the Long-Doc task. As shown in Table 6,
our NV-Embed-v2 achieves the second highest scores in QA section. As described in Table 7,
our NV-Embed-v2 attained the highest scores of 74.78 on the Long-Doc section, surpassing the
Bge-en-icl model that requires overheads adding in-context examples to query during training. It
is important to highlight that the NV-Embed-v2 model, which achieved higher MTEB accuracy
scores, also demonstrates improved accuracy on both QA and Long-Doc tasks in the AIR-Bench
compared to NV-Embed-v1. Interestingly, this is not always observed in the literature, where
a model performing better on MTEB does not necessarily outperform on the AIR-Bench. For
example, while SFR-Embedding-2R substantially outperforms SFR-Embedding-Mistral in MTEB
scores (SFR-Embedding-2R: 70.31, SFR-Embedding-Mistral: 67.56), it falls short in AIR-Bench
performance both in QA (SFR-Embedding-2R: 49.47, SFR-Embedding-Mistral: 51.58) and Long-doc
(SFR-Embedding-2R: 67.45, SFR-Embedding-Mistral: 69.0).

Table 6: QA (nDCG@10 scores) on AIR benchmark 24.04

Domain
Bge-en-icl (zero-shot)
NV-Embed-v2
SFR-Embedding-Mistral
Stella-1.5B-v5
Gte-Qwen2-7B-instruct
NV-Embed-v1
Linq-Embed-Mistral
SFR-Embedding-2R
E5-mistral-7b-instruct

Wiki Web
54.40
64.61
52.58
65.19
51.27
63.46
50.88
61.99
51.20
63.46
50.42
62.84
48.41
61.04
48.77
63.72
44.41
61.67

News Healthcare
55.11
53.13
52.21
53.87
54.07
51.46
49.44
51.14
48.18

57.25
59.56
58.76
58.81
54.20
58.53
60.18
55.86
56.32

Law
25.10
25.00
23.27
23.22
22.31
20.65
20.34
20.98
19.32

Finance Arxiv Msmarco Avg (8)
52.93
54.81
52.28
53.04
51.58
56.94
51.53
57.26
50.26
58.20
50.02
49.89
49.69
50.04
49.47
54.78
48.56
54.79

63.71
60.8
58.99
61.38
58.39
60.27
60.50
57.66
59.03

48.46
48.94
47.75
44.81
40.27
46.10
47.56
42.84
44.78

Table 7: Long-document (Recall@10 scores) on AIR benchmark 24.04

Domain
NV-Embed-v2
Bge-en-icl (zero-shot)
NV-Embed-v1
Bge-multilingual-gemma2
Linq-Embed-Mistral
Stella-1.5B-v5
SFR-Embedding-Mistral
Text-embed-3-large (OpenAI)
E5-mistral-7b-instruct
SFR-Embedding-2R

Arxiv (4) Book (2) Healthcare (5)

79.27
78.30
77.65
71.77
75.46
73.17
72.79
74.53
72.14
70.51

77.46
78.21
75.49
76.46
73.81
74.38
72.41
73.16
72.44
70.22

73.01
73.65
72.38
73.96
71.58
70.02
67.94
65.83
68.44
67.60

Law (4) Avg. (15)
71.18
67.09
69.55
70.86
68.58
69.32
64.83
64.47
62.92
62.82

74.78
73.75
73.45
72.88
72.11
71.25
69.0
68.77
68.49
67.45

3https://github.com/AIR-Bench/AIR-Bench

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

B EXPERIMENTAL DETAILS AND INSTRUCTION TEMPLATES FOR TRAINING

AND EVALUATION

We use the Adam optimizer for each training stage. The optimizer hyperparameters are included in
Table 8. We restart the optimizer with the same 50 warm-up steps and lower learning rate for the
second stage.

Table 8: Parameters used in the experiments

Parameter
Batchsize
Number of Hardnegatives
Warm-up Steps

Value
128
7
50

Training Steps

Learning Rate

LoRA Params

Weight Decay
Optimizer
Padding Side
Number of Latents (r)
Latent Width (d)
Multi-Attention Heads

First stage - 20k
Second stage - 18k

First stage - 2e-5
Second stage - 1.5e-5

Rank - 16
Alpha - 32
Dropout - 0.1
0.03
Adam
right
512
4096
8

Table 9: Instructions and number of samples used for each training dataset.

Task Name
ArguAna

Natural Language Inference

PAQ, MSMARCO

SQUAD
StackExchange
Natural Question
HotpotQA
FEVER
FiQA2018
BioASQ
HoVer
Nfcorpus
MIRACL
Mr.TyDi
SciFact
STS12, STS22, STSBenchmark
AmazonCounterfactual-Classification
AmazonPolarity-Classification
AmazonReviews-Classification
Banking77-Classification
Emotion-Classification

Instruction Template
Given a claim, retrieve documents that support or refute the claim
Retrieve semantically similar text
Given a premise, retrieve a hypothesis that is entailed by the premise
Given a web search query, retrieve relevant passages that answer the query
Given a question, retrieve passages that answer the question
Given a question, retrieve documents that can help answer the question
Given a question, retrieve passages that answer the question
Given a web search query, retrieve relevant passages that answer the query
Given a question, retrieve passages that answer the question
Given a multi-hop question, retrieve documents that can help answer the question
Given a claim, retrieve documents that support or refute the claim
Given a financial question, retrieve relevant passages that answer the query
Given a query, retrieve documents that can help answer the question
Given a claim, retrieve documents that support or refute the claim
Given a question, retrieve relevant documents that answer the question
Given a question, retrieve passages that answer the question
Given a question, retrieve passages that answer the question
Given a scientific claim, retrieve documents that support or refute the claim
Retrieve semantically similar text.
Classify a given Amazon customer review text as either counterfactual or not-counterfactual
Classify Amazon reviews into positive or negative sentiment
Classify the given Amazon review into its appropriate rating category
Given a online banking query, find the corresponding intents
Classify the emotion expressed in the given Twitter message into one of the six emotions:anger,

fear, joy, love, sadness, and surprise

Number of Samples
16k

270k

500k, 500k

87k
80k
100k
170k
140k
5k
2.4k
17k
3.6k
2k
2k
0.9k
1.8k, 0.3k, 2.7k
6k
20k
40k
10k
16k

Classify the sentiment expressed in the given movie review text from the IMDB dataset
Classify the intent of the given utterance in task-oriented conversation
Classify the intent domain of the given utterance in task-oriented conversation
Given a user utterance as query, find the user intents
Given a user utterance as query, find the user scenarios
Classify the given comments as either toxic or not toxic

Imdb-Classification
MTOPIntent-Classification
MTOPDomain-Classification
MassiveIntent-Classification
MassiveScenario-Classification
ToxicConversationsClassification
TweetSentimentExtractionClassification Classify the sentiment of a given tweet as either positive, negative, or neutral
Arxiv-Clustering-P2P
Arxiv-Clustering-S2S
Biorxiv-Clustering-P2P
Biorxiv-Clustering-S2S
Medrxiv-Clustering-P2P
Medrxiv-Clustering-S2S
Reddit-Clustering
Reddit-Clustering-S2S
Stackexchange-Clustering
Stackexchange-Clustering-S2S
TwentyNewsgroups-Clustering

Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Identify the main and secondary category of Arxiv papers based on the titles
Identify the main category of Biorxiv papers based on the titles and abstracts
Identify the main category of Biorxiv papers based on the titles
Identify the main category of Medrxiv papers based on the titles and abstracts
Identify the main category of Medrxiv papers based on the titles
Identify the main category of Medrxiv papers based on the titles and abstracts
Identify the main category of Medrxiv papers based on the titles and abstracts
Identify the main category of Medrxiv papers based on the titles and abstracts
Identify the main category of Medrxiv papers based on the titles and abstracts
Identify the topic or theme of the given news articles

24k
15k
15k
11k
11k
50k
27k
50k
50k
15k
15k
2.3k
2.3k
50k
40k
50k
40k
1.7k

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

17

Under review as a conference paper at ICLR 2025

Table 10: Instructions used for evaluation on the MTEB benchmark. “STS*” indicates we use the
same instructions for all the STS tasks.

Task Name
ArguAna
ClimateFEVER
DBPedia
FEVER
FiQA2018
HotpotQA
MSMARCO
NFCorpus
Natural Question
QuoraRetrieval
SCIDOCS
SciFact
Touche2020
TREC-COVID
STS
SummEval
AmazonCounterfactualClassification
AmazonPolarityClassification
AmazonReviewsClassification
Banking77Classification
EmotionClassification

Instruction Template
Given a claim, retrieve documents that support or refute the claim
Given a claim about climate change, retrieve documents that support or refute the claim
Given a query, retrieve relevant entity descriptions from DBPedia
Given a claim, retrieve documents that support or refute the claim
Given a financial question, retrieve user replies that best answer the question
Given a multi-hop question, retrieve documents that can help answer the question
Given a web search query, retrieve relevant passages that answer the query
Given a question, retrieve relevant documents that answer the question
Given a question, retrieve passages that answer the question
Given a question, retrieve questions that are semantically equivalent to the given question
Given a scientific paper title, retrieve paper abstracts that are cited by the given paper
Given a scientific claim, retrieve documents that support or refute the claim
Given a question, retrieve passages that answer the question
Given a query on COVID-19, retrieve documents that answer the query
Retrieve semantically similar text.
Given a news summary, retrieve other semantically similar summaries
Classify a given Amazon customer review text as either counterfactual or not-counterfactual
Classify Amazon reviews into positive or negative sentiment
Classify the given Amazon review into its appropriate rating category
Given a online banking query, find the corresponding intents
Classify the emotion expressed in the given Twitter message into one of the six emotions:anger,

fear, joy, love, sadness, and surprise

Classify the sentiment expressed in the given movie review text from the IMDB dataset
Given a user utterance as query, find the user intents
Given a user utterance as query, find the user scenarios
Classify the intent domain of the given utterance in task-oriented conversation
Classify the intent of the given utterance in task-oriented conversation
Classify the given comments as either toxic or not toxic

ImdbClassification
MassiveIntentClassification
MassiveScenarioClassification
MTOPDomainClassification
MTOPIntentClassification
ToxicConversationsClassification
TweetSentimentExtractionClassification Classify the sentiment of a given tweet as either positive, negative, or neutral
ArxivClusteringP2P
ArxivClusteringS2S
BiorxivClusteringP2P
BiorxivClusteringS2S
MedrxivClusteringP2P
MedrxivClusteringS2S
RedditClustering
RedditClusteringP2P
StackExchangeClustering
StackExchangeClusteringP2P
TwentyNewsgroupsClustering
AskUbuntuDupQuestions
MindSmallReranking
SciDocsRR
StackOverflowDupQuestions
SprintDuplicateQuestions
TwitterSemEval2015
TwitterURLCorpus

Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Identify the main and secondary category of Arxiv papers based on the titles
Identify the main category of Biorxiv papers based on the titles and abstracts
Identify the main category of Biorxiv papers based on the titles
Identify the main category of Medrxiv papers based on the titles and abstracts
Identify the main category of Medrxiv papers based on the titles
Identify the topic or theme of Reddit posts based on the titles
Identify the topic or theme of Reddit posts based on the titles and posts
Identify the topic or theme of StackExchange posts based on the titles
Identify the topic or theme of StackExchange posts based on the given paragraphs
Identify the topic or theme of the given news articles
Retrieve duplicate questions from AskUbuntu forum
Retrieve relevant news articles based on user browsing history
Given a title of a scientific paper, retrieve the titles of other relevant papers
Retrieve duplicate questions from StackOverflow forum
Retrieve duplicate questions from Sprint forum
Retrieve tweets that are semantically similar to the given tweet
Retrieve tweets that are semantically similar to the given tweet

C LATENT-ATTENTION VISUALIZATION

Figure 2: Attention over 4096 latents across 8 heads (columns) are visualized for 10 positive and
10 negative reviews (rows) from the AmazonReviewsClassification dataset. The attention weights
are mean pooled across tokens. The attention weights reveal that the latents specialize in learning
features of queries. The latent indicated by the arrows specialized in learning the positivity of reviews.
It has high attention across the positive reviews and low attention across the negative reviews.

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Table 11: Full BEIR and MTEB benchmark

ArguAna
ClimateFEVER
CQADupStack
DBPEDIA
FEVER
FiQA2018
HotpotQA
MSMARCO
NFCorpus
Natural
QuoraRetrieval
SCIDOCS
SciFact
Touche2020
TREC-COVID
BIOSSES
SICK-R
STS12
STS13
STS14
STS15
STS16
STS17
STS22
STSBenchmark
SummEval
SprintDuplicateQuestions
TwitterSemEval2015
TwitterURLCorpus
AmazonCounterfactual
AmazonPolarity
AmazonReviews
Banking77
Emotion
Imdb
MassiveIntent
MassiveScenario
MTOPDomain
MTOPIntent
ToxicConversations
TweetSentimentExtraction
Arxiv-P2P
Arxiv-S2S
Biorxiv-P2P
Biorxiv-S2S
Medrxiv-P2P
Medrxiv-S2S
Reddit
Reddit-P2P
StackExchange
StackExchange-P2P
TwentyNewsgroups
AskUbuntuDupQuestions
MindSmallRerank
SciDocsRR
StackOverflowDupQuestions
MTEB Average (56)

Bge-multilin
gual-gemma2
77.37
39.37
47.94
51.37
90.38
60.04
83.26
45.71
38.11
71.45
90.04
26.93
72.05
30.26
64.27
85.74
82.66
77.71
87.45
83.48
87.63
86.7
91.18
69.02
87.25
31.2
90.94
79.64
86.95
89.48
96.9
61.6
92.53
92.97
96.66
82.05
84.4
98.61
95.51
87.34
78.86
54.91
50.28
52.64
49.2
45.81
44.11
56.03
65.83
66.21
45.74
70.44
64.59
31.79
87.6
54.9
69.88

Gte-Qwen2-
7B-instruct
64.27
45.88
46.43
52.42
95.11
62.03
73.08
45.98
40.6
67
90.09
28.91
79.06
30.57
82.26
81.37
79.28
79.55
88.83
83.87
88.54
86.49
88.73
66.88
86.85
31.35
92.82
77.96
86.59
91.31
97.5
62.56
87.57
79.45
96.75
85.41
89.77
99.04
91.88
85.12
72.58
54.46
51.74
50.09
46.65
46.23
44.13
73.55
74.13
79.86
49.41
53.91
67.58
33.36
89.09
55.66
70.24

NV-Embed-v1 NV-Embed-v2

68.21
34.72
50.51
48.29
87.77
63.1
79.92
46.49
38.04
71.22
89.21
20.19
78.43
28.38
85.88
85.59
82.8
76.22
86.3
82.09
87.24
84.77
87.42
69.85
86.14
31.2
95.94
78.73
86.05
95.12
97.14
55.47
90.34
91.71
97.06
80.07
81.74
96.51
89.77
92.6
80.6
53.76
49.59
48.15
44.74
39.24
36.98
63.2
68.01
74.99
42.04
60.13
67.5
30.82
87.26
56.58
69.32

70.07
45.39
50.24
53.50
93.75
65.73
85.48
45.63
45.17
73.57
89.04
21.90
80.13
88.44
31.78
87.42
82.15
77.89
88.30
84.30
89.04
86.77
90.67
68.12
88.41
30.70
97.02
81.11
87.87
94.28
97.74
63.96
92.42
93.38
97.14
86.10
92.17
99.25
94.37
92.74
80.87
55.80
51.26
54.09
49.60
46.09
44.86
71.10
74.94
82.10
48.36
64.82
67.46
31.76
87.59
55.79
72.31

Stella-en-
1.5B-v5
65.27
46.11
47.75
52.28
94.83
60.48
76.67
45.22
42
71.8
90.03
26.64
80.09
29.94
85.98
83.11
82.89
80.09
89.68
85.07
89.39
87.15
91.35
68.1
88.23
31.49
96.04
80.58
87.58
92.87
97.16
59.36
89.79
84.29
96.66
85.83
90.2
99.01
92.78
88.76
74.84
55.44
50.66
50.68
46.87
46.87
44.65
72.86
75.27
80.29
49.57
61.43
67.33
33.05
89.2
55.25
71.19

bge-en-icl
(zeroshot)
82.76
45.35
47.23
50.42
91.96
58.77
84.98
46.72
40.69
73.85
91.02
25.25
78.33
29.67
78.11
86.35
83.87
77.73
85.98
82.34
87.35
86.54
91.25
68.08
87.92
30.75
95.06
78.54
87.19
92.88
96.86
61.28
91.42
93.31
96.91
82.26
83.92
97.99
93.56
93.16
79.9
54.42
49.17
52.32
48.38
46.13
44.2
71.2
72.17
81.29
45.53
68.51
64.8
30.6
86.9
56.32
71.24

SFR-Embe
dding-2R
62.34
34.43
46.11
51.21
92.16
61.77
81.36
42.18
41.34
73.96
89.58
24.87
85.91
28.18
87.28
87.6
77.01
75.67
82.4
79.93
85.82
84.5
88.93
67.1
83.6
30.71
97.62
78.57
88.03
92.72
97.31
61.04
90.02
93.37
96.8
85.97
90.61
98.58
91.3
91.14
79.7
54.02
48.82
50.76
46.57
46.66
44.18
62.92
72.74
76.48
48.29
66.42
66.71
31.26
87.29
55.32
70.31

19

Under review as a conference paper at ICLR 2025

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Table 12: Prompt template for short-long matching subgroup.

Brainstorm a list of potentially useful text retrieval tasks.

Here are a few examples for your reference:
- Given a web search query, retrieve relevant passages that answer the query
- Given a claim about climate change, retrieve documents that support or refute the claim
- Given a job title, search for job descriptions that provide information about the role

Please adhere to the following guidelines:
- Specify the type of query and the type of desired texts.
- Each retrieval task should cover a wide range of queries, and should not be too specific.
- Cover a wide range of query types and desired text types.

Your output must always be a JSON list of strings only, with about 40 elements, and each element corresponds to
a distinct retrieval task in one sentence. Do not explain yourself or output anything else. Be creative!
You have been assigned a retrieval task: {task}

Your mission is to write one text retrieval example for this task in JSON format. The JSON object must
contain the following keys:
- "user_query": a string, a random example of what is provided as specified by the task description.
- "positive_document": a string, a relevant document for the user query.
- "hard_negative_document1": a string, a hard negative document that is irrelevant but appears relevant to the query.
- "hard_negative_document2": a string, another hard negative document that is irrelevant but appears relevant to the query.

Please adhere to the following guidelines:
- The "user_query" should be {query_type}, {query_length}, {clarity}, and diverse in topic. The "user_query" should
not restate the task and just contain what the task description says is provided.
- All documents must be created independent of the query. Avoid copying the query verbatim. It’s acceptable if
some parts of the "positive_document" are not topically related to the query.
- All documents should be at least {num_words} words long.
- The "hard_negative_document1" may contain little useful information, but it should be less useful or
comprehensive compared to the "positive_document".
- The "hard_negative_document2" may should be about a related but different topic.
- Do not provide any explanation in any document on why it is relevant or not relevant to the query.
- Both the query and documents require {difficulty} level education to understand.

Your output must always be a JSON object only, do not explain yourself or output anything else. Be creative!"""
Placeholders:
“{query_type}” ∈ {extremely long-tail, long-tail, common}
“{query_length}” ∈ {less than 5 words, 5 to 15 words, at least 10 words}
“{difficulty}” ∈ {high school, college, PhD}
“{clarity}” ∈ {clear, understandable with some effort, ambiguous}
“{num_words}” ∈ {50, 100, 200, 300, 400, 500}

20

Under review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Table 13: Prompt template for long-short matching subgroup.

Brainstorm a list of potentially useful text classification tasks.

Please adhere to the following guidelines:
- Tasks should cover a diverse range of domains and task types.

Your output must always be a JSON list of strings only, with about 40 elements, and each element corresponds
to a distinct text classification task in one sentence. Do not explain yourself or output anything else. Be creative!
You have been assigned a text classification task: {task}

Your mission is to write one text classification example for this task in JSON format. The JSON object must
contain the following keys:
- "input_text": a string, the input text specified by the classification task.
- "label": a string, the correct
label of the input text.
- "misleading_label": a string, an incorrect label that is related to the task.

Please adhere to the following guidelines:
- The "input_text" should be {num_words} words and diverse in expression.
- The "misleading_label" must be a valid label for the given task, but not as appropriate as the "label" for the
"input_text".
- Avoid including the values of the "label" and "misleading_label" fields in the "input_text", that would make
the task too easy.
- The "input_text" is {clarity} and requires {difficulty} level education to comprehend.

Your output must always be a JSON object only, do not explain yourself or output anything else. Be creative!
Placeholders:
{num_words} ∈ {"less than 10","at least 10", "at least 50", "at least 100", "at least 200"}
{difficulty} ∈ {high school, college, PhD}
{clarity} ∈ {clear, understandable with some effort, ambiguous}

21

