# EmbedDistill: A Geometric Knowledge Distillation
for Information Retrieval

Seungyeon Kim  Google LLC, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Ankit Singh
Rawat  Google LLC, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Manzil
Zaheer  Google LLC, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Sadeep Jayasumana  Google LLC, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Veeranjaneyulu Sadhanala  Google LLC, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Wittawat
Jitkrittum  Google LLC, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Aditya Krishna Menon  Google LLC, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Rob Fergus
Google LLC, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Sanjiv
Kumar  Google LLC, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Large neural models (such as Transformers) achieve state-of-the-art
performance for information retrieval (IR). In this paper, we aim to improve
distillation methods that pave the way for the resource-efficient deployment
of such models in practice. Inspired by our theoretical analysis of the
teacher-student generalization gap for IR models, we propose a novel
distillation approach that leverages the relative geometry among queries and
documents learned by the large teacher model. Unlike existing teacher score-
based distillation methods, our proposed approach employs embedding matching
tasks to provide a stronger signal to align the representations of the teacher
and student models. In addition, it utilizes query generation to explore the
data manifold to reduce the discrepancies between the student and the teacher
where training data is sparse. Furthermore, our analysis also motivates novel
asymmetric architectures for student models which realizes better embedding
alignment without increasing online inference cost. On standard benchmarks
like MSMARCO, we show that our approach successfully distills from both dual-
encoder (DE) and cross-encoder (CE) teacher models to 1/10th size asymmetric
students that can retain 95-97% of the teacher performance.

##  1 Introduction

Neural models for information retrieval (IR) are increasingly used to model
the true ranking function in various applications, including web search (Mitra
and Craswell, 2018), recommendation (Zhang et al., 2019), and question-
answering (QA) (Chen et al., 2017). Notably, the recent success of
Transformers (Vaswani et al., 2017)-based pre-trained language models (Devlin
et al., 2019; Liu et al., 2019; Raffel et al., 2020) on a wide range of
natural language understanding tasks has also prompted their utilization in IR
to capture query-document relevance (see, e.g., Dai and Callan, 2019b;
MacAvaney et al., 2019a; Nogueira and Cho, 2019; Lee et al., 2019; Karpukhin
et al., 2020a).

A typical IR system comprises two stages: (1) A _retriever_ first selects a
small subset of potentially relevant candidate documents (out of a large
collection) for a given query; and (2) A _re-ranker_ then identifies a precise
ranking among the candidates provided by the retriever. _Dual-encoder_ (DE)
models are the de-facto architecture for retrievers (Lee et al., 2019;
Karpukhin et al., 2020a). Such models independently embed queries and
documents into a common space, and capture their relevance by simple
operations on these embeddings such as the inner product. This enables offline
creation of a document index and supports fast retrieval during inference via
efficient maximum inner product search implementations (Guo et al., 2020;
Johnson et al., 2021), with _online_ query embedding generation primarily
dictating the inference latency. _Cross-encoder_ (CE) models, on the other
hand, are preferred as re-rankers, owing to their excellent performance
(Nogueira and Cho, 2019; Dai and Callan, 2019a; Yilmaz et al., 2019). A CE
model jointly encodes a query-document pair while enabling early interaction
among query and document features. Employing a CE model for retrieval is often
infeasible, as it would require processing a given query with _every_ document
in the collection at inference time. In fact, even in the re-ranking stage,
the inference cost of CE models is high enough (Khattab and Zaharia, 2020) to
warrant exploration of efficient alternatives (Hofstätter et al., 2020;
Khattab and Zaharia, 2020; Menon et al., 2022). Across both architectures,
scaling to larger models brings improved performance at increased
computational cost (Ni et al., 2022; Neelakantan et al., 2022).

_Knowledge distillation_ (Bucilǎ et al., 2006; Hinton et al., 2015) provides a
general strategy to address the prohibitive inference cost associated with
high-quality large neural models. In the IR literature, most existing
distillation methods only rely on the teacher’s query-document relevance
scores (see, e.g., Lu et al., 2020; Hofstätter et al., 2020; Chen et al.,
2021; Ren et al., 2021; Santhanam et al., 2021) or their proxies (Izacard and
Grave, 2021). However, given that neural IR models are inherently embedding-
based, it is natural to ask:

_Is it useful to go beyond matching of the teacher and student models’
_scores_ ,
and directly aim to align their _embedding spaces_?_

With this in mind, we propose a novel distillation method for IR models that
utilizes an _embedding matching_ task to train student models. The proposed
method is inspired by our rigorous treatment of the generalization gap between
the teacher and student models in IR settings. Our theoretical analysis of the
teacher-student generalization gap further suggests novel design choices
involving asymmetric configurations for student DE models, intending to
further reduce the gap by better aligning teacher and student embedding
spaces. Notably, our proposed distillation method supports _cross-architecture
distillation_ and improves upon existing (score-based) distillation methods
for both retriever and re-ranker models. When distilling a large teacher DE
model into a smaller student DE model, for a given query (document), one can
minimize the distance between the query (document) embeddings of the teacher
and student (after compatible projection layers to account for dimension
mismatch, if any). In contrast, a teacher CE model doesn’t directly provide
document and query embeddings, and so to effectively employ embedding
matching-based distillation requires modifying the scoring layer with _dual-
pooling_ (Yadav et al., 2022) and adding various regularizers. Both of these
changes improve geometry of teacher embeddings and facilitate effective
knowledge transfer to the student DE model via embedding matching-based
distillation.

Our key contributions toward improving IR models via distillation are:

  * •

We provide the first rigorous analysis of the teacher-student generalization
gap for IR settings which captures the role of alignment of embedding spaces
of the teacher and student towards reducing the gap (Sec. 3).

  * •

Inspired by our analysis, we propose a novel distillation approach for neural
IR models, namely EmbedDistill, that goes beyond score matching and aligns the
embedding spaces of the teacher and student models (Sec. 4). We also show that
EmbedDistill can leverage synthetic data to improve a student by further
aligning the embedding spaces of the teacher and student (Sec. 4.3).

  * •

Our analysis motivates novel distillation setups. Specifically, we consider a
student DE model with an _asymmetric_ configuration, consisting of a small
query encoder and a frozen document encoder inherited from the teacher. This
significantly reduces inference latency of query embedding generation, while
leveraging the teachers’ high-quality document index (Sec. 4.1).

  * •

We provide a comprehensive empirical evaluation of EmbedDistill (Sec. 5) on
two standard IR benchmarks – Natural Questions (Kwiatkowski et al., 2019a) and
MSMARCO (Nguyen et al., 2016). We also evaluate EmbedDistill on BEIR benchmark
(Thakur et al., 2021) which is used to measure the zero-shot performance of an
IR model.

(a) DE to DE distillation

(b) CE to DE distillation

Figure 1: Proposed distillation method with query embedding matching. Top: In
the DE to DE distillation setting, the student employs an asymmetric DE
configuration with a small query encoder and a large (non-trainable) document
encoder inherited from the teacher DE model. The smaller query encoder ensures
small latency for encoding query during inference, and large document encoder
leads to a good quality document index. Bottom: Similarly the setting of CE to
DE distillation using EmbedDistill, with teacher CE model employing dual
pooling.

Note that prior works have utilized embedding alignment during distillation
for non-IR setting (see, e.g., Romero et al., 2014; Sanh et al., 2019; Jiao et
al., 2020; Aguilar et al., 2020; Zhang and Ma, 2020; Chen et al., 2022).
However, to the best of our knowledge, our work is the first to study
embedding matching-based distillation method for IR settings which requires
addressing multiple IR-specific challenges such as cross-architecture
distillation, partial representation alignment, and enabling novel asymmetric
student configurations. Furthermore, unlike these prior works, our proposed
method is theoretically justified to reduce the teacher-student performance
gap.

##  2 Background

Let $\mathscr{Q}$ and $\mathscr{D}$ denote the query and document spaces,
respectively. An IR model is equivalent to a scorer
$s:\mathscr{Q}\times\mathscr{D}\to\mathbb{R}$, i.e., it assigns a (relevance)
score $s(q,d)$ for a query-document pair
$(q,d)\in\mathscr{Q}\times\mathscr{D}$. Ideally, we want to learn a scorer
such that $s(q,d)>s(q,d^{\prime})$ _iff_ the document $d$ is more relevant to
the query $q$ than document $d^{\prime}$. We assume access to $n$ labeled
training examples
$\mathscr{S}_{n}=\\{(q_{i},\mathbf{d}_{i},\mathbf{y}_{i})\\}_{i\in[n]}$. Here,
$\mathbf{d}_{i}=(d_{i,1},\ldots,d_{i,L})\in\mathscr{D}^{L},~{}\forall
i\in[n]$, denotes a list of $L$ documents and
$\mathbf{y}_{i}=(y_{i,1},\ldots,y_{i,L})\in\\{0,1\\}^{L}$ denotes the
corresponding labels such that $y_{i,j}=1$ iff the document $d_{i,j}$ is
relevant to the query $q_{i}$. Given $\mathscr{S}_{n}$, we learn an IR model
by minimizing

$R(s;\mathscr{S}_{n}):=\frac{1}{n}\sum\nolimits_{i\in[n]}\ell\big{(}s_{q_{i},\mathbf{d}_{i}},\mathbf{y}_{i}\big{)},$
(1)

where $s_{q_{i},\mathbf{d}_{i}}:=(s(q_{i},d_{1,i}),\ldots,s(q_{i},d_{1,L}))$
and $\ell\big{(}s_{q_{i},\mathbf{d}_{i}},\mathbf{y}_{i}\big{)}$ denotes the
loss $s$ incurs on $(q_{i},\mathbf{d}_{i},\mathbf{y}_{i})$. Due to space
constraint, we defer concrete choices for the loss function $\ell$ to Appendix
A.

While this learning framework is general enough to work with any IR models,
next, we formally introduce two families of Transformer-based IR models that
are prevalent in the recent literature.

###  2.1 Transformer-based IR models: Cross-encoders and Dual-encoders

Let query $q$ = $(q^{1},\ldots,q^{m_{1}})$ and document $d$ =
$(d^{1},\ldots,d^{m_{2}})$ consist of $m_{1}$ and $m_{2}$ tokens,
respectively. We now discuss how Transformers-based CE and DE models process
the $(q,d)$ pair.

Cross-encoder model. Let $p=[q;d]$ be the sequence obtained by concatenating
$q$ and $d$. Further, let $\tilde{p}$ be the sequence obtained by adding
special tokens such [CLS] and [SEP] to $p$. Given an encoder-only Transformer
model ${\rm Enc}$, the relevance score for the $(q,d)$ pair is

$s(q,d)=\langle w,{\rm pool}\big{(}{\rm Enc}(\tilde{p})\big{)}\rangle=\langle
w,{\tt emb}_{q,d}\rangle,$ (2)

where $w$ is a $d$-dimensional classification vector, and ${\rm pool}(\cdot)$
denotes a pooling operation that transforms the contextualized token
embeddings ${\rm Enc}(\tilde{p})$ to a joint embedding vector ${\tt
emb}_{q,d}$. [CLS]-pooling is a common operation that simply outputs the
embedding of the [CLS] token as ${\tt emb}_{q,d}$.

Dual-encoder model. Let $\tilde{q}$ and $\tilde{d}$ be the sequences obtained
by adding appropriate special tokens to $q$ and $d$, respectively. A DE model
comprises two (encoder-only) Transformers ${\rm Enc}_{Q}$ and ${\rm Enc}_{D}$,
which we call query and document encoders, respectively.111It is common to
employ dual-encoder models where query and document encoders are shared. Let
${\tt emb}_{q}$ = ${\rm pool}\big{(}{\rm Enc}_{Q}(\tilde{q})\big{)}$ and ${\tt
emb}_{d}$ = ${\rm pool}\big{(}{\rm Enc}_{D}(\tilde{d})\big{)}$ denote the
query and document embeddings, respectively. Now, one can define
$s(q,d)=\langle{\tt emb}_{q},{\tt emb}_{d}\rangle$ to be the relevance score
assigned to the $(q,d)$ pair by the DE model.

###  2.2 Score-based distillation for IR models

Most distillation schemes for IR (e.g., Lu et al., 2020; Hofstätter et al.,
2020; Chen et al., 2021) rely on teacher relevance scores. Given a training
set $\mathscr{S}_{n}$ and a teacher with scorer $s^{\rm t}$, one learns a
student with scorer $s^{\rm s}$ by minimizing

$R(s^{\rm s},s^{\rm
t};\mathscr{S}_{n})=\frac{1}{n}\sum\nolimits_{i\in[n]}\ell_{\rm
d}\big{(}s^{\rm s}_{q,\mathbf{d}_{i}},s^{\rm
t}_{q,\mathbf{d}_{i}}\big{)},\vspace{-1mm}$ (3)

where $\ell_{\rm d}$ captures the discrepancy between $s^{\rm s}$ and $s^{\rm
t}$. See Appendix A for common choices for $\ell_{\rm d}$.

##  3 Teacher-student generalization gap: Inspiration for embedding alignment

Our main objective is to devise novel distillation methods to realize high-
performing student DE models. As a first step in this direction, we rigorously
study the teacher-student generalization gap as realized by standard (score-
based) distillation in IR settings. Informed by our analysis, we subsequently
identify novel ways to improve the student model’s performance. In particular,
our analysis suggests two natural directions to reduce the teacher-student
generalization gap: 1) enforcing tighter alignment between embedding spaces of
teacher and student models; and 2) exploring novel asymmetric configuration
for student DE model.

Let
$R(s)=\mathbb{E}\left[\ell\big{(}s_{q,\mathbf{d}},\mathbf{y}\big{)}\right]$ be
the population version of the empirical risk in Eq. 1, which measures the test
time performance of the IR model defined by the scorer $s$. Thus, $R(s^{\rm
s})-R(s^{\rm t})$ denotes the teacher-student generalization gap. In the
following result, we bound this quantity (see Appendix C.1 for a formal
statement and proof). We focus on distilling a teacher DE model to a student
DE model and $L=1$ (cf. Sec. 2) as it leads to easier exposition without
changing the main takeaways. Our analysis can be extended to $L>1$ or CE to DE
distillation with more complex notation.

######  Theorem 3.1 (Teacher-student generalization gap (informal)).

Let $\mathscr{F}$ and $\mathscr{G}$ denote the function classes for the query
and document encoders for the student model, respectively. Suppose that the
score-based distillation loss $\ell_{\rm d}$ in Eq. 3 is based on binary cross
entropy loss (Eq. 12 in Appendix A). Let one-hot (label-dependent) loss $\ell$
in Eq. 1 be the binary cross entropy loss (Eq. 10 in Appendix A). Further,
assume that all encoders have the same output dimension and embeddings have
their $\ell_{2}$-norm bounded by $K$. Then, we have

$\displaystyle R(s^{\rm s})-R(s^{\rm
t})\leq\mathscr{E}_{n}(\mathscr{F},\mathscr{G})+2KR_{{\rm Emb},Q}({\rm t},{\rm
s};\mathscr{S}_{n})+2KR_{{\rm Emb},D}({\rm t},{\rm s};\mathscr{S}_{n})$ (4)
$\displaystyle\qquad\qquad\qquad\quad+\Delta(s^{\rm
t};\mathscr{S}_{n})+K^{2}\big{(}\mathbb{E}\left[\left|\sigma(s^{\rm
t}_{q,d})-y\right|\right]+\frac{1}{n}\sum_{i\in[n]}\left|\sigma(s^{\rm
t}_{q_{i},d_{i}})-y_{i}\right|\big{)},$

where $\mathscr{E}_{n}(\mathscr{F},\mathscr{G}):=\sup_{s^{\rm
s}\in\mathscr{F}\times\mathscr{G}}\big{|}R(s^{\rm s},s^{\rm
t};\mathscr{S}_{n})-\mathbb{E}\ell_{\rm d}\big{(}s^{\rm s}_{q,d},s_{q,d}^{\rm
t}\big{)}\big{|}$; $\sigma$ denotes the sigmoid function; and $\Delta(s^{\rm
t};\mathscr{S}_{n})$ denotes the deviation between the empirical risk (on
$\mathscr{S}_{n}$) and population risk of the teacher $s^{\rm t}$. Here,
$R_{{\rm Emb},Q}({\rm t},{\rm s};\mathscr{S}_{n})$ and $R_{{\rm Emb},D}({\rm
t},{\rm s};\mathscr{S}_{n})$ measure misalignment between teacher and student
embeddings by focusing on queries and documents, respectively (cf. Eq. 7 & 8
in Sec. 4.1).

The last three quantities in the bound in Thm. 3.1, namely $\Delta(s^{\rm
t};\mathscr{S}_{n})$, $\mathbb{E}[|\sigma(s^{\rm t}_{q,d})-y|]$, and
$\frac{1}{n}\sum_{i\in[n]}|\sigma(s^{\rm t}_{q_{i},d_{i}})-y_{i}|$, are
independent of the underlying student model. These terms solely depend on the
quality of the underlying teacher model $s^{\rm t}$. That said, the teacher-
student gap can be made small by reducing the following three terms: 1)
uniform deviation of the student’s empirical distillation risk from its
population version $\mathscr{E}_{n}(\mathscr{F},\mathscr{G})$; 2) misalignment
between teacher student query embeddings $R_{{\rm Emb},Q}({\rm t},{\rm
s};\mathscr{S}_{n})$; and 3) misalignment between teacher student document
embeddings $R_{{\rm Emb},D}({\rm t},{\rm s};\mathscr{S}_{n})$.

The last two terms motivate us to propose an embedding matching-based
distillation that explicitly aims to minimize these terms during student
training. Even more interestingly, these terms also inspire an asymmetric DE
configuration for the student which strikes a balance between the goals of
reducing the misalignment between the embeddings of teacher and student (by
inheriting teacher’s document encoder) and ensuring serving efficiency (small
inference latency) by employing a small query encoder. Before discussing these
proposals in detail in Sec. 4 and Fig. 1, we explore the first term
$\mathscr{E}_{n}(\mathscr{F},\mathscr{G})$ and highlight how our proposals
also have implications for reducing this term. Towards this, the following
result bounds $\mathscr{E}_{n}(\mathscr{F},\mathscr{G})$. We present an
informal statement of the result (see Appendix C.2 for a more precise
statement and proof).

######  Proposition 3.2.

Let $\ell_{\rm d}$ be a distillation loss which is $L_{\ell_{\rm
d}}$-Lipschitz in its first argument. Let $\mathscr{F}$ and $\mathscr{G}$
denote the function classes for the query and document encoders, respectively.
Further assume that, for each query and document encoder in our function
class, the query and document embeddings have their $\ell_{2}$-norm bounded by
$K$. Then,

$\mathscr{E}_{n}(\mathscr{F},\mathscr{G})\leq\mathbb{E}_{\mathscr{S}_{n}}\frac{48KL_{\ell_{\rm
d}}}{\sqrt{n}}\int_{0}^{\infty}\sqrt{\log\big{(}N(u,\mathscr{F})N(u,\mathscr{G})\big{)}}\;du.$
(5)

Furthermore, with a fixed document encoder, i.e., $\mathscr{G}=\\{g^{*}\\}$,

$\mathscr{E}_{n}(\mathscr{F},\\{g*\\})\leq\mathbb{E}_{\mathscr{S}_{n}}\frac{48KL_{\ell_{\rm
d}}}{\sqrt{n}}\int_{0}^{\infty}\sqrt{\log N(u,\mathscr{F})}\;du.$ (6)

Here, $N(u,\cdot)$ is the $u$-covering number of a function class.

Note that Eq. 5 and Eq. 6 correspond to uniform deviation when we train
_without_ and _with_ a frozen document encoder, respectively. It is clear that
the bound in Eq. 6 is less than or equal to that in Eq. 5 (because
$N(u,\mathscr{G})\geq 1$ for any $u$), which alludes to desirable impact of
employing a frozen document encoder as one of our proposal seeks to do via
inheriting teacher’s document encoder (for instance in an asymmetric DE
configuration). Furthermore, our proposal of employing an embedding-matching
task will regularize the function class of query encoders; effectively
reducing it to $\mathscr{F}^{\prime}$ with
$|\mathscr{F}^{\prime}|\leq|\mathscr{F}|$. The same holds true for document
encoder function class when document encoder is trainable (as in Eq. 5),
leading to an effective function class $\mathscr{G}^{\prime}$ with
$|\mathscr{G}^{\prime}|\leq|\mathscr{G}|$. Since we would have
$N(u,\mathscr{F}^{\prime})\leq N(u,\mathscr{F})$ and
$N(u,\mathscr{G}^{\prime})\leq N(u,\mathscr{G})$, this suggests desirable
implications of embedding matching for reducing the uniform deviation bound.

##  4 Embedding-matching based distillation

Informed by our analysis of teacher-student generalization gap in Sec. 3, we
propose EmbedDistill – a novel distillation method that explicitly focuses on
aligning the embedding spaces of the teacher and student. Our proposal goes
beyond existing distillation methods in the IR literature that only use the
teacher scores. Next, we introduce EmbedDistill for two prevalent settings:
(1) distilling a large DE model to a smaller DE model; 222 CE to CE
distillation is a special case of this with classification vector $w$ (cf. Eq.
2) as trivial second encoder. and (2) distilling a CE model to a DE model.

###  4.1 DE to DE distillation

Given a $(q,d)$ pair, let ${\tt emb}^{\rm t}_{q}$ and ${\tt emb}^{\rm t}_{d}$
be the query and document embeddings produced by the query encoder ${\rm
Enc}^{\rm t}_{Q}$ and document encoder ${\rm Enc}^{\rm t}_{D}$ of the teacher
DE model, respectively. Similarly, let ${\tt emb}^{\rm s}_{q}$ and ${\tt
emb}^{\rm s}_{d}$ denote the query and document embeddings produced by a
student DE model with $({\rm Enc}^{\rm s}_{Q},{\rm Enc}^{\rm s}_{D})$ as its
query and document encoders. Now, EmbedDistill optimizes the following
embedding alignment losses in addition to the score-matching loss from Sec.
2.2 to align query and document embeddings of the teacher and student:

$\displaystyle R_{{\rm Emb},Q}({\rm t},{\rm
s};\mathscr{S}_{n})=\frac{1}{n}\sum\nolimits_{q\in\mathscr{S}_{n}}\|{\tt
emb}^{\rm t}_{q}-{\rm proj}\big{(}{\tt emb}^{\rm s}_{q}\big{)}\|;$ (7)
$\displaystyle R_{{\rm Emb},D}({\rm t},{\rm
s};\mathscr{S}_{n})=\frac{1}{n}\sum\nolimits_{d\in\mathscr{S}_{n}}\|{\tt
emb}^{\rm t}_{d}-{\rm proj}\big{(}{\tt emb}^{\rm s}_{d}\big{)}\|.$ (8)

Asymmetric DE. We also propose a novel student DE configuration where the
student employs the teacher’s document encoder (i.e., ${\rm Enc}^{\rm
s}_{D}={\rm Enc}^{\rm t}_{D}$) and only train its query encoder, which is much
smaller compared to the teacher’s query encoder. For such a setting, it is
natural to only employ the embedding matching loss in Eq. 7 as the document
embeddings are aligned by design (cf. Fig. 1(a)).

Note that this asymmetric student DE does not incur an increase in latency
despite the use of a large teacher document encoder. This is because the large
document encoder is only needed to create a good quality document index
offline, and only the query encoder is evaluated at inference time. Also, the
similarity search cost is not increased as the projection layer ensures the
same small embedding dimension as in the symmetric DE student. Thus, for DE to
DE distillation, we prescribe the asymmetric DE configuration universally. Our
theoretical analysis (cf. Sec. 3) and experimental results (cf. Sec. 5)
suggest that the ability to inherit the document tower from the teacher DE
model can drastically improve the final performance, especially when combined
with query embedding matching task (cf. Eq. 7).

###  4.2 CE to DE distillation

Given that CE models jointly encode query-document pairs, individual query and
document embeddings are not readily available to implement embedding matching
losses as per Eq. 7 and 8. This makes it challenging to employ EmbedDistill
for CE to DE distillation.

As a naïve solution, for a $(q,d)$ pair, one can simply match a joint
transformation of the student’s query embedding ${\tt emb}^{\rm s}_{q}$ and
document embedding ${\tt emb}^{\rm s}_{d}$ to the teacher’s joint embedding
${\tt emb}^{\rm t}_{q,d}$ , produced by (single) teacher encoder ${\rm
Enc}^{t}$. However, we observed that including such an embedding matching task
often leads to severe over-fitting, and results in a poor student. Since
$s^{\rm t}(q,d)=\langle w,{\tt emb}^{\rm t}_{q,d}\rangle$, during CE model
training, the joint embeddings ${\tt emb}^{\rm t}_{q,d}$ for relevant and
irrelevant $(q,d)$ pairs are encouraged to be aligned with $w$ and $-w$,
respectively. This produces degenerate embeddings that do not capture semantic
query-to-document relationships. We notice that even the final query and
document token embeddings lose such semantic structure (cf. Appendix G.2).
Thus, a teacher CE model with $s^{\rm t}(q,d)=\langle w,{\tt emb}^{\rm
t}_{q,d}\rangle$ does not add value for distillation beyond score-matching; in
fact, it _hurts_ to include naïve embedding matching. Next, we propose a
modified CE model training strategy that facilitates EmbedDistill.

CE models with dual pooling. A _dual pooling_ scheme is employed in the
scoring layer to produce two embeddings ${\tt emb}^{\rm t}_{q\leftarrow(q,d)}$
and ${\tt emb}^{\rm t}_{d\leftarrow(q,d)}$ from a CE model that serve as the
_proxy_ query and document embeddings, respectively. Accordingly, we define
the relevance score as $s^{\rm t}(q,d)=\langle{\tt emb}^{\rm
t}_{q\leftarrow(q,d)},{\tt emb}^{\rm t}_{d\leftarrow(q,d)}\rangle$. We explore
two variants of dual pooling: (1) special token-based pooling that pools from
[CLS] and [SEP]; and (2) segment-based weighted mean pooling that separately
performs weighted averaging on the query and document segments of the final
token embeddings. See Appendix B for details.

In addition to dual pooling, we also utilize a reconstruction loss during the
CE training, which measures the likelihood of predicting each token of the
original input from the final token embeddings. This loss encourages
reconstruction of query and document tokens based on the final token
embeddings and prevents the degeneration of the token embeddings during
training. Given proxy embeddings from the teacher CE, we can perform
EmbedDistill with the embedding matching loss defined in Eq. 7 and Eq. 8 (cf.
Fig. 1(b)).

###  4.3 Task-specific online data generation

Data augmentation as a general technique has been previously considered in the
IR literature (see, e.g., Nogueira et al., 2019b; Oğuz et al., 2021; Izacard
et al., 2021), especially in data-limited, out-of-domain, or zero-shot
settings. As EmbedDistill aims to align the embeddings spaces of the teacher
and student, the ability to generate similar queries or documents can
naturally help enforce such an alignment globally on the task-specific
manifold. Given a set of unlabeled task-specific query and document pairs
$\mathscr{U}_{m}$, we can further add the embedding matching losses $R_{\rm
Emb,Q}({\rm t},{\rm s};\mathscr{U}_{m})$ or $R_{\rm Emb,D}({\rm t},{\rm
s};\mathscr{U}_{m})$ to our training objective. Interestingly, for DE to DE
distillation setting, our approach can even benefit from a large collection of
task-specific queries $\mathscr{Q}^{\prime}$ or documents
$\mathscr{D}^{\prime}$. Here, we can independently employ embedding matching
losses $R_{\rm Emb,Q}({\rm t},{\rm s};\mathscr{Q}^{\prime})$ or $R_{\rm
Emb,D}({\rm t},{\rm s};\mathscr{D}^{\prime})$ that focus on queries and
documents, respectively. Please refer to Appendix E describing how the task-
specific data were generated.

##  5 Experiments

Table 1: Full recall performance of various student DE models on NQ dev set,
including symmetric DE student model (67.5M or 11.3M transformer for both
encoders), and asymmetric DE student model (67.5M or 11.3M transformer as
query encoder and document embeddings inherited from the teacher). All
distilled students used the same teacher (110.1M parameter BERT-base models as
both encoders), with the full Recall@5 = 72.3, Recall@20 = 86.1, and
Recall@100 = 93.6.

Method | 6-Layer (67.5M) |  | 4-Layer (11.3M)
---|---|---|---
R@5 | R@20 | R@100 |  | R@5 | R@20 | R@100
Train student directly | 36.2 | 59.7 | 80.0 |  | 24.8 | 44.7 | 67.5
\+ Distill from teacher | 65.3 | 81.6 | 91.2 |  | 44.3 | 64.9 | 81.0
\+ Inherit doc embeddings | 69.9 | 83.9 | 92.3 |  | 56.3 | 70.9 | 82.5
\+ Query embedding matching | 72.7 | 86.5 | 93.9 |  | 61.2 | 75.2 | 85.1
\+ Query generation | 73.4 | 86.3 | 93.8 |  | 64.3 | 77.8 | 87.9
Train student using only |  |  |  |  |  |  |
embedding matching and |  |  |  |  |  |  |
inherit doc embeddings | 71.4 | 84.9 | 92.6 |  | 64.6 | 50.2 | 76.8
\+ Query generation | 71.8 | 85.0 | 93.0 |  | 54.2 | 68.9 | 80.8

We now conduct a comprehensive evaluation of the proposed distillation
approach. Specifically, we highlight the utility of the approach for both DE
to DE and CE to DE distillation. We also showcase the benefits of combining
our distillation approach with query generation methods.

###  5.1 Setup

Benchmarks and evaluation metrics. We consider two popular IR benchmarks —
Natural Questions (NQ) (Kwiatkowski et al., 2019b) and MSMARCO (Nguyen et al.,
2016), which focus on finding the most relevant passage/document given a
question and a search query, respectively. NQ provides both standard test and
dev sets, whereas MSMARCO provides only the dev set that are widely used for
common benchmarks. In what follows, we use the terms query (document) and
question (passages) interchangeably. For NQ, we use the standard full recall
(_strict_) as well as the _relaxed_ recall metric (Karpukhin et al., 2020a) to
evaluate the retrieval performance. For MSMARCO, we focus on the standard
metrics _Mean Reciprocal Rank_ (MRR)@10, and _normalized Discounted Cumulative
Gain_ (nDCG)@10 to evaluate both re-ranking and retrieval performance. For the
re-ranking, we restrict to re-ranking only the top 1000 candidate document
provided as part of the dataset to be fair, while some works use stronger
methods to find better top 1000 candidates for re-ranking (resulting in higher
evaluation numbers)

See Appendix D for a detailed discussion on these evaluation metrics. Finally,
we also evaluate EmbedDistill on the BEIR benchmark (Thakur et al., 2021) in
terms of nDCG@10 and recall@100 metrics.

Model architectures. We follow the standard Transformers-based IR model
architectures similar to Karpukhin et al. (2020a); Qu et al. (2021); Oğuz et
al. (2021). We utilized various sizes of DE models based on BERT-base (Devlin
et al., 2019) (12-layer, 768 dim, 110M parameters), DistilBERT (Sanh et al.,
2019) (6-layer, 768 dim, 67.5M parameters – $\sim$ 2/3 of base), or BERT-mini
(Turc et al., 2019) (4-layer, 256 dim, 11.3M parameters – $\sim$ 1/10 of
base). For query generation (cf. Sec. 4.3), we employ BART-base (Lewis et al.,
2020), an encoder-decoder model, to generate similar questions from each
training example’s input question (query). We randomly mask $10\%$ of tokens
and inject zero mean Gaussian noise with $\sigma=\\{0.1,0.2\\}$ between the
encoder and decoder. See Appendix E for more details on query generation and
Appendix F.1 for hyperparameters.

Table 2: Performance of EmbedDistill for DE to DE distillation on NQ test set.
While prior works listed in the table rely on techniques such as negative
mining and multi-stage training, we explore the orthogonal direction of
embedding-matching that improves single-stage distillation, which can be
combined with them.

Method | #Layers | R@20 | R@100
---|---|---|---
DPR (Karpukhin et al., 2020a) | 12 | 78.4 | 85.4
DPR + PAQ (Oğuz et al., 2021) | 12 | 84.0 | 89.2
DPR + PAQ (Oğuz et al., 2021) | 24 | 84.7 | 89.2
ACNE (Xiong et al., 2021) | 12 | 81.9 | 87.5
RocketQA (Qu et al., 2021) | 12 | 82.7 | 88.5
MSS-DPR (Sachan et al., 2021) | 12 | 84.0 | 89.2
MSS-DPR (Sachan et al., 2021) | 24 | 84.8 | 89.8
Our teacher (Zhang et al., 2022) | 12 (220.2M) | 85.4 | 90.0
EmbedDistill | 6 (67.5M) | 85.1 | 89.8
EmbedDistill | 4 (11.3M) | 81.2 | 87.4

###  5.2 DE to DE distillation

We employ AR2 (Zhang et al.,
2022)333https://github.com/microsoft/AR2/tree/main/AR2 and SentenceBERT-v5
(Reimers et al., 2019)444https://huggingface.co/sentence-transformers/msmarco-
bert-base-dot-v5 as teacher DE models for NQ and MSMARCO. Note that both
models are based on BERT-base. For DE to DE distillation, we consider two
kinds of configurations for the student DE model: (1) _Symmetric_ : We use
identical question and document encoders. We evaluate DistilBERT and BERT-mini
on both datasets. (2) _Asymmetric_ : The student inherits document embeddings
from the teacher DE model and _are not_ trained during the distillation. For
query encoder, we use DistilBERT or BERT-mini which are smaller than document
encoder.

Student DE model training. We train student DE models using a combination of
(i) one-hot loss (cf. Eq. 9 in Appendix A) on training data; (ii) distillation
loss in (cf. Eq. 11 in Appendix A); and (iii) embedding matching loss in Eq.
7. We used [CLS]-pooling for all student encoders. Unlike DPR (Karpukhin et
al., 2020a) or AR2, we do not use hard negatives from BM25 or other models,
which greatly simplifies our distillation procedure.

Results and discussion. To understand the impact of various proposed
configurations and losses, we train models by sequentially adding components
and evaluate their retrieval performance on NQ and MSMARCO dev set as shown in
Table 1 and Table 3 respectively. (See Table 6 in Appendix F.2 for performance
on NQ in terms of the relaxed recall and Table 7 in Appendix F.3 for MSMARCO
in terms of<EMAIL_ADDRESS>
We begin by training a symmetric DE without distillation. As expected, moving
to distillation brings in considerable gains. Next, we swap the student
document encoder with document embeddings from the teacher (non-trainable),
which leads to a good jump in the performance. Now we can introduce
EmbedDistill with Eq. 7 for aligning query representations between student and
teacher. The two losses are combined with weight of $1.0$ (except for BERT-
mini models in the presence of query generation with $5.0$). This improves
performance significantly, e.g.,it provides $\sim$3 and $\sim$5 points
increase in recall@5 on NQ with students based on DistilBERT and BERT-mini,
respectively (Table 1). We further explore the utility of EmbedDistill in
aligning the teacher and student embedding spaces in Appendix G.1.

Table 3: Performance of various DE models on MSMARCO dev set for both _re-
ranking_ and _retrieval_ tasks (full corpus). The teacher model (110.1M
parameter BERT-base models as both encoders) for re-ranking achieves MRR@10 of
36.8 and that for retrieval get MRR@10 of 37.2. The table shows performance
(in MRR@10) of the symmetric DE student model (67.5M or 11.3M transformer as
both encoders), and asymmetric DE student model (67.5M or 11.3M transformer as
query encoder and document embeddings inherited from the teacher).

Method | Re-ranking |  | Retrieval
---|---|---|---
67.5M | 11.3M |  | 67.5M | 11.3M
Train student directly | 27.0 | 23.0 |  | 22.6 | 18.6
\+ Distill from teacher | 34.6 | 30.4 |  | 35.0 | 28.6
\+ Inherit doc embeddings | 35.2 | 32.1 |  | 35.7 | 30.3
\+ Query embedding matching | 36.2 | 35.0 |  | 37.1 | 35.4
\+ Query generation | 36.2 | 34.4 |  | 37.2 | 34.8
Train student using only |  |  |  |  |
embedding matching and |  |  |  |  |
inherit doc embeddings | 36.5 | 33.5 |  | 36.6 | 31.4
\+ Query generation | 36.4 | 34.1 |  | 36.7 | 32.8

On top of the two losses (standard distillation and embedding matching), we
also use $R_{\rm Emb,Q}({\rm t},{\rm s};\mathscr{Q}^{\prime})$ from Sec. 4.3
on $2$ additional questions (per input question) generated from BART. We also
try a variant where we eliminate the standard distillation loss and only
employ the embedding matching loss in Eq. 7 along with inheriting teacher’s
document embeddings. This configuration without the standard distillation loss
leads to excellent performance (with query generation again providing
additional gains in most cases.)

Table 4: Average BEIR performance of our DE teacher and EmbedDistill student
models and their numbers of trainable parameters. Both models are trained on
MSMARCO and evaluated on 14 other datasets (the average does not include
MSMARCO). The full table is at Appendix F.4. With EmbedDistill, student
materializes most of the performance of the teacher on the unforeseen
datasets.

Method | #Layers | nDCG@10 | R@100
---|---|---|---
DPR (Karpukhin et al., 2020b) | 12 | 22.5 | 47.7
ANCE (Xiong et al., 2021) | 12 | 40.5 | 60.0
TAS-B (Hofstätter et al., 2021) | 6 | 42.8 | 64.8
GenQ (Thakur et al., 2021) | 6 | 42.5 | 64.2
Our teacher (Reimers et al., 2019) | 12 (220.2M) | 45.7 | 65.1
EmbedDistill | 6 (67.5M) | 44.0 | 63.5

It is worth highlighting that DE models trained with the proposed methods
(e.g., asymmetric DE with embedding matching and generation) achieve 99% of
the performance in both NQ/MSMARCO tasks with a query encoder that is 2/3rd
the size of that of the teacher. Furthermore, even with 1/10th size of the
query encoder, our proposal can achieve 95-97% of the performance. This is
particularly useful for latency critical applications with minimal impact on
the final performance.

Finally, we take our best student models, i.e., one trained using with
additional embedding matching loss and using data augmentation from query
generation, and evaluate on test sets. We compare with various prior work and
note that most prior work used considerably bigger models in terms of
parameters, depth (12 or 24 layers), or width (upto 1024 dims). For NQ test
set results are reported in Table 2, but as MSMARCO does not have any public
test set, we instead present results for the BEIR benchmark in Table 4. Note
we also provide evaluation of our SentenceBERT teacher achieving very high
performance on the benchmark which can be of independent interest (please
refer to Appendix F.4 for details). For both NQ and BEIR, our approach obtains
competitive student model with fewer than 50% of the parameters: even with 6
layers, our student model is very close (98-99%) to its teacher.

###  5.3 CE to DE distillation

For the CE to DE distillation experiment, we converted SimLM [CLS]-pooled CE
model555https://github.com/microsoft/unilm/tree/master/simlm to a dual-pooled
CE model via standard score-based distillation (cf. Section 2.2). We
subsequently utilize the resulting dual-pooled version of the SimLM CE model
as a teacher to perform CE to DE distillation via embedding alignment. Similar
to DE to DE distillation (cf. Section 5.2), we aim to identify the utility of
various components of EmbedDistill in our exploration. See Table 5 for the
results.

We also explore distilling dual-pooled CE teacher model to an asymmetric DE
student model. In this setting, DE student model simply inherits the document
embeddings from the CE teacher model. Crucially, the inheritance of the
document embedding from the dual-pooled CE teacher model can be done offline
as we feed an empty query along with the document (separated by the [SEP]
token) to obtain the document embedding from the dual-pooled CE teacher model.

Results and discussion. The excellent performance of distillation to an
asymmetric DE model (which inherits document embeddings from the dual-pooled
CE model) not only showcases the power of embedding alignment via EmbedDistill
but also highlights the effectiveness of dual-pooling method for the teacher.

Table 5: Performance of various DE models obtained via CE to DE distillation
on MSMARCO dev set for _re-ranking_ (original top1000). The teacher model is a
dual-pooled version of the SimLM model which achieves MRR@10 of 40.0 nDCG@10
of 45.8. The table shows performance of the symmetric DE student model (67.5M
or 11.3M transformer as both encoders), and asymmetric DE student model (67.5M
or 11.3M transformer as query encoder and document embeddings inherited from
the dual-pooled teacher). Note that the document embeddings used during
inheritance are generated in a query-independent manner from the CE teacher
model (with empty query).

Method | MRR@10 |  | nDCG@10
---|---|---|---
67.5M | 11.3M |  | 67.5M | 11.3M
Train student directly | 27.0 | 23.0 |  | 32.2 | 29.7
\+ Distill from teacher | 33.2 | 28.6 |  | 38.7 | 33.6
\+ Inherit doc embeddings | 35.4 | 30.2 |  | 41.0 | 35.6
\+ Query embedding matching | 36.1 | 31.7 |  | 41.7 | 37.1
\+ Query generation | 36.3 | 32.1 |  | 42.0 | 37.6
Train student using only |  |  |  |  |
embedding matching and |  |  |  |  |
inherit doc embeddings | 36.9 | 34.7 |  | 42.6 | 40.4
\+ Query generation | 36.8 | 35.1 |  | 42.5 | 40.8
Standard distillation |  |  |  |  |
from [CLS]-teacher | 32.8 | 28.5 |  | 38.4 | 33.6

##  6 Related work

Here, we position our EmbedDistill work with respect to prior work on
distillation and data augmentation for Transformers-based IR models. We also
cover prior efforts on aligning representations during distillation for _non-
IR_ settings. Unlike our problem setting where the DE student is factorized,
these works mainly consider distilling a single large Transformer into a
smaller one.

Distillation for IR. Traditional distillation techniques have been widely
applied in the IR literature, often to distill a teacher CE model to a student
DE model (Li et al., 2020; Chen et al., 2021). Recently, distillation from a
DE model (with complex late interaction) to another DE model (with inner-
product scoring) has also been considered (Lin et al., 2021; Hofstätter et
al., 2021). As for distilling across different model architectures, Lu et al.
(2020); Izacard and Grave (2021) consider distillation from a teacher CE model
to a student DE model. Hofstätter et al. (2020) conduct an extensive study of
knowledge distillation across a wide-range of model architectures. Most
existing distillation schemes for IR rely on only teacher scores; by contrast,
we propose a geometric approach that also utilizes the teacher _embeddings_.
Many recent efforts (Qu et al., 2021; Ren et al., 2021; Santhanam et al.,
2021) show that iterative multi-stage (self-)distillation improves upon
single-stage distillation (Qu et al., 2021; Ren et al., 2021; Santhanam et
al., 2021). These approaches use a model from the previous stage to obtain
labels (Santhanam et al., 2021) as well as mine harder-negatives (Xiong et
al., 2021). We only focus on the single-stage distillation in this paper.
Multi-stage procedures are complementary to our work, as one can employ our
proposed embedding-matching approach in various stages of such a procedure.
Interestingly, we demonstrate in Sec. 5 that our proposed EmbedDistill can
successfully benefit from high quality models trained with such complex
procedures (Reimers et al., 2019; Zhang et al., 2022). In particular, our
single-stage distillation method can transfer almost all of their performance
gains to even smaller models. Also to showcase that our method brings gain
orthogonal to how teacher was trained, we conduct experiments with single-
stage trained teacher in Appendix F.5.

Distillation with representation alignments. Outside of the IR context, a few
prior works proposed to utilize alignment between hidden layers during
distillation (Romero et al., 2014; Sanh et al., 2019; Jiao et al., 2020;
Aguilar et al., 2020; Zhang and Ma, 2020). Chen et al. (2022) utilize the
representation alignment to re-use teacher’s classification layer for image
classification. Unlike these works, our work is grounded in a rigorous
theoretical understanding of the teacher-student (generalization) gap for IR
models. Further, our work differs from these as it needs to address multiple
challenges presented by an IR setting: 1) cross-architecture distillation such
as CE to DE distillation; 2) partial representation alignment of query or
document representations as opposed to aligning for the entire input, i.e., a
query-documents pair; and 3) catering representation alignment approach to
novel IR setups such as asymmetric DE configuration. To the best of our
knowledge, our work is first in the IR literature that goes beyond simply
matching scores (or its proxies) for distillation.

Semi-supervised learning for IR. Data augmentation or semi-supervised learning
has been previously used to ensure data efficiency in IR (see, e.g., MacAvaney
et al., 2019b; Zhao et al., 2021). More interestingly, data augmentation have
enabled performance improvements as well. Doc2query (Nogueira et al., 2019b,
a) performs document expansion by generating queries that are relevant to the
document and appending those queries to the document. Query expansion has also
been considered, e.g., for document re-ranking (Zheng et al., 2020). Notably,
generating synthetic (query, passage, answer) triples from a text corpus to
augment existing training data for QA systems also leads to significant gains
(Alberti et al., 2019; Oğuz et al., 2021). Furthermore, even zero-shot
approaches, where no labeled query-document pairs are used, can also perform
competitively to supervised methods (Lee et al., 2019; Izacard et al., 2021;
Ma et al., 2021; Sachan et al., 2022). Unlike these works, we utilize query-
generation capability to ensure tighter alignment between the embedding spaces
of the teacher and student.

Richer transformers-based architectures for IR. Besides DE and CE models (cf.
Sec. 2), intermediate configurations (MacAvaney et al., 2020; Khattab and
Zaharia, 2020; Nie et al., 2020; Luan et al., 2021) have been proposed. Such
models independently encode query and document before applying a more complex
_late interaction_ between the two. Nogueira et al. (2020) explore generative
encoder-decoder style model for re-ranking. In this paper, we focus on basic
DE/CE models to showcase the benefits of our proposed geometric distillation
approach. Exploring embedding matching for aforementioned architectures is an
interesting avenue for future work.

##  7 Conclusion

We propose EmbedDistill — a novel distillation method for IR that goes beyond
simple score matching. En route, we provide a theoretical understanding of the
teacher-student generalization gap in an IR setting which not only motivated
EmbedDistill but also inspired new design choices for the student DE models:
(a) reusing the teacher’s document encoder in the student and (b) aligning
query embeddings of the teacher and student. This simple approach delivers
consistent quality and computational gains in practical deployments and we
demonstrate them on MSMARCO, NQ, and BEIR benchmarks. Finally, we found
EmbedDistill retains 95-97% of the teacher performance to with 1/10th size
students.

Limitations. As discussed in Sec. 4.2 and 5.3, EmbedDistill requires
modifications in the CE scoring function to be effective. In terms of
underlying IR model architectures, we only explore Transformer-based models in
our experiments; primarily due to their widespread utilization. That said, we
expect our results to extend to non-Transformer architectures such as MLPs.
Finally, we note that our experiments only consider NLP domains, and exploring
other modalities (e.g., vision) or multi-modal settings (e.g., image-to-text
search) is left as an interesting avenue for future work.

## References

  * Aguilar et al. (2020) Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Chenlei Guo.  Knowledge distillation from internal representations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 7350–7357, 2020.
  * Alberti et al. (2019) Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins.  Synthetic QA corpora generation with roundtrip consistency.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6168–6173, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1620.  URL https://aclanthology.org/P19-1620.
  * Bengio and Senecal (2008) Yoshua Bengio and Jean-SÉbastien Senecal.  Adaptive importance sampling to accelerate training of a neural probabilistic language model.  _IEEE Transactions on Neural Networks_ , 19(4):713–722, 2008.  doi: 10.1109/TNN.2007.912312.
  * Bousquet et al. (2004) Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi.  _Introduction to Statistical Learning Theory_ , pages 169–207.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2004.  ISBN 978-3-540-28650-9.  doi: 10.1007/978-3-540-28650-9˙8.  URL https://doi.org/10.1007/978-3-540-28650-9_8.
  * Bucilǎ et al. (2006) Cristian Bucilǎ, Rich Caruana, and Alexandru Niculescu-Mizil.  Model compression.  In _Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’06, pages 535–541, New York, NY, USA, 2006. ACM.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada, July 2017. Association for Computational Linguistics.  doi: 10.18653/v1/P17-1171.  URL https://aclanthology.org/P17-1171.
  * Chen et al. (2022) Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen.  Knowledge distillation with the reused teacher classifier.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11933–11942, 2022.
  * Chen et al. (2021) Xuanang Chen, Ben He, Kai Hui, Le Sun, and Yingfei Sun.  Simplified tinybert: Knowledge distillation for document retrieval.  In Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani, editors, _Advances in Information Retrieval_ , pages 241–248, Cham, 2021. Springer International Publishing.  ISBN 978-3-030-72240-1.
  * Dai and Callan (2019a) Zhuyun Dai and Jamie Callan.  Deeper text understanding for IR with contextual neural language modeling.  In Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer, editors, _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019_ , pages 985–988. ACM, 2019a.
  * Dai and Callan (2019b) Zhuyun Dai and Jamie Callan.  Context-aware sentence/passage term importance estimation for first stage retrieval.  _arXiv preprint arXiv:1910.10687_ , 2019b.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics, 2019\.
  * Guo et al. (2020) Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar.  Accelerating large-scale inference with anisotropic vector quantization.  In _International Conference on Machine Learning_ , 2020.  URL https://arxiv.org/abs/1908.10396.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network, 2015.
  * Hofstätter et al. (2020) Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury.  Improving efficient neural ranking models with cross-architecture knowledge distillation.  _CoRR_ , abs/2010.02666, 2020.  URL https://arxiv.org/abs/2010.02666.
  * Hofstätter et al. (2021) Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury.  Efficiently teaching an effective dense retriever with balanced topic aware sampling.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’21, page 113–122, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450380379.  doi: 10.1145/3404835.3462891.  URL https://doi.org/10.1145/3404835.3462891.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave.  Distilling knowledge from reader to retriever for question answering.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=NTEz-6wysdb.
  * Izacard et al. (2021) Gautier Izacard, Mathild Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave.  Unsupervised dense information retrieval with contrastive learning.  _arXiv preprint arXiv:2112.09118_ , 2021.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.  TinyBERT: Distilling BERT for natural language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4163–4174, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.372.  URL https://aclanthology.org/2020.findings-emnlp.372.
  * Johnson et al. (2021) Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with gpus.  _IEEE Transactions on Big Data_ , 7(3):535–547, 2021.  doi: 10.1109/TBDATA.2019.2921572.
  * Karpukhin et al. (2020a) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online, November 2020a. Association for Computational Linguistics.
  * Karpukhin et al. (2020b) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_ , 2020b.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia.  _ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT_ , page 39–48.  Association for Computing Machinery, New York, NY, USA, 2020.  ISBN 9781450380164.
  * Kwiatkowski et al. (2019a) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466, 2019a.  doi: 10.1162/tacl˙a˙00276.  URL https://aclanthology.org/Q19-1026.
  * Kwiatkowski et al. (2019b) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466, 2019b.
  * Ledoux and Talagrand (1991) Michel Ledoux and Michel Talagrand.  _Probability in Banach spaces_.  Springer-Verlag, 1991.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova.  Latent retrieval for weakly supervised open domain question answering.  In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 6086–6096. Association for Computational Linguistics, 2019.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.703.  URL https://aclanthology.org/2020.acl-main.703.
  * Li et al. (2020) Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun.  Parade: Passage representation aggregation for document reranking.  _arXiv preprint arXiv:2008.09093_ , 2020.
  * Lin et al. (2021) Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin.  In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval.  In _Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)_ , pages 163–173, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.repl4nlp-1.17.  URL https://aclanthology.org/2021.repl4nlp-1.17.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Lu et al. (2020) Wenhao Lu, Jian Jiao, and Ruofei Zhang.  Twinbert: Distilling knowledge to twin-structured compressed bert models for large-scale retrieval.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, CIKM ’20, page 2645–2652, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450368599.  doi: 10.1145/3340531.3412747.  URL https://doi.org/10.1145/3340531.3412747.
  * Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins.  Sparse, dense, and attentional representations for text retrieval.  _Transactions of the Association for Computational Linguistics_ , 9:329–345, 2021.  doi: 10.1162/tacl˙a˙00369.  URL https://aclanthology.org/2021.tacl-1.20.
  * Ma et al. (2021) Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald.  Zero-shot neural passage retrieval via domain-targeted synthetic question generation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1075–1088, Online, April 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.eacl-main.92.  URL https://aclanthology.org/2021.eacl-main.92.
  * MacAvaney et al. (2019a) Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian.  CEDR: Contextualized embeddings for document ranking.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR’19, page 1101–1104, New York, NY, USA, 2019a. Association for Computing Machinery.  ISBN 9781450361729.  doi: 10.1145/3331184.3331317.  URL https://doi.org/10.1145/3331184.3331317.
  * MacAvaney et al. (2019b) Sean MacAvaney, Andrew Yates, Kai Hui, and Ophir Frieder.  Content-based weak supervision for ad-hoc re-ranking.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR’19, page 993–996, New York, NY, USA, 2019b. Association for Computing Machinery.  ISBN 9781450361729.  doi: 10.1145/3331184.3331316.  URL https://doi.org/10.1145/3331184.3331316.
  * MacAvaney et al. (2020) Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder.  _Efficient Document Re-Ranking for Transformers by Precomputing Term Representations_ , page 49–58.  Association for Computing Machinery, New York, NY, USA, 2020.  ISBN 9781450380164.
  * Menon et al. (2022) Aditya Menon, Sadeep Jayasumana, Ankit Singh Rawat, Seungyeon Kim, Sashank Reddi, and Sanjiv Kumar.  In defense of dual-encoders for neural ranking.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 15376–15400. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/menon22a.html.
  * Mitra and Craswell (2018) Bhaskar Mitra and Nick Craswell.  An introduction to neural information retrieval.  _Foundations and Trends® in Information Retrieval_ , 13(1):1–126, 2018.  ISSN 1554-0669.  doi: 10.1561/1500000061.  URL http://dx.doi.org/10.1561/1500000061.
  * Neelakantan et al. (2022) Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al.  Text and code embeddings by contrastive pre-training.  _arXiv preprint arXiv:2201.10005_ , 2022.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng.  MS MARCO: A human generated machine reading comprehension dataset.  In Tarek Richard Besold, Antoine Bordes, Artur S. d’Avila Garcez, and Greg Wayne, editors, _Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016_ , volume 1773 of _CEUR Workshop Proceedings_. CEUR-WS.org, 2016.
  * Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang.  Large dual encoders are generalizable retrievers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9844–9855, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  URL https://aclanthology.org/2022.emnlp-main.669.
  * Nie et al. (2020) Ping Nie, Yuyu Zhang, Xiubo Geng, Arun Ramamurthy, Le Song, and Daxin Jiang.  DC-BERT: decoupling question and document for efficient contextual encoding.  In Jimmy Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu, editors, _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_ , pages 1829–1832. ACM, 2020.  doi: 10.1145/3397271.3401271.  URL https://doi.org/10.1145/3397271.3401271.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho.  Passage re-ranking with BERT.  _CoRR_ , abs/1901.04085, 2019.  URL http://arxiv.org/abs/1901.04085.
  * Nogueira et al. (2019a) Rodrigo Nogueira, Jimmy Lin, and AI Epistemic.  From doc2query to doctttttquery.  _Online preprint_ , 6, 2019a.
  * Nogueira et al. (2019b) Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho.  Document expansion by query prediction.  _arXiv preprint arXiv:1904.08375_ , 2019b.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin.  Document ranking with a pretrained sequence-to-sequence model.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 708–718, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.63.  URL https://aclanthology.org/2020.findings-emnlp.63.
  * Oğuz et al. (2021) Barlas Oğuz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Wen-tau Yih, Sonal Gupta, et al.  Domain-matched pre-training tasks for dense retrieval.  _arXiv preprint arXiv:2107.13602_ , 2021.
  * Qu et al. (2021) Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang.  RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering.  In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 5835–5847. Association for Computational Linguistics, 2021\.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Reimers et al. (2019) Nils Reimers, Iryna Gurevych, and Iryna Gurevych.  Sentence-BERT: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2019\.  URL http://arxiv.org/abs/1908.10084.
  * Ren et al. (2021) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen.  Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking.  In _Proceedings of EMNLP_ , 2021.
  * Romero et al. (2014) Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  _arXiv preprint arXiv:1412.6550_ , 2014.
  * Sachan et al. (2021) Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, and Bryan Catanzaro.  End-to-end training of neural retrievers for open-domain question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6648–6662, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.519.  URL https://aclanthology.org/2021.acl-long.519.
  * Sachan et al. (2022) Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer.  Improving passage retrieval with zero-shot question generation.  _arXiv preprint arXiv:2204.07496_ , 2022.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ , 2019.
  * Santhanam et al. (2021) Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia.  Colbertv2: Effective and efficient retrieval via lightweight late interaction.  _CoRR_ , abs/2112.01488, 2021.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych.  BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.  URL https://openreview.net/forum?id=wCu6T5xFjeJ.
  * Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Well-read students learn better: On the importance of pre-training compact models.  _arXiv preprint arXiv:1908.08962_ , 2019.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.  ISBN 9781510860964.
  * Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=zeFrfgyZln.
  * Yadav et al. (2022) Nishant Yadav, Nicholas Monath, Rico Angell, Manzil Zaheer, and Andrew McCallum.  Efficient nearest neighbor search for cross-encoder models using matrix factorization.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2171–2194, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  URL https://aclanthology.org/2022.emnlp-main.140.
  * Yilmaz et al. (2019) Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin.  Cross-domain modeling of sentence-level evidence for document retrieval.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3490–3496, Hong Kong, China, November 2019. Association for Computational Linguistics.
  * Zhang et al. (2022) Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen.  Adversarial retriever-ranker for dense text retrieval.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=MR7XubKUFB.
  * Zhang and Ma (2020) Linfeng Zhang and Kaisheng Ma.  Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors.  In _International Conference on Learning Representations_ , 2020.
  * Zhang et al. (2019) Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay.  Deep learning based recommender system: A survey and new perspectives.  _ACM Comput. Surv._ , 52(1), feb 2019.  ISSN 0360-0300.  doi: 10.1145/3285029.  URL https://doi.org/10.1145/3285029.
  * Zhao et al. (2021) Chen Zhao, Chenyan Xiong, Jordan Boyd-Graber, and Hal Daumé III.  Distantly-supervised dense retrieval enables open-domain question answering without evidence annotation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9612–9622, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.756.  URL https://aclanthology.org/2021.emnlp-main.756.
  * Zheng et al. (2020) Zhi Zheng, Kai Hui, Ben He, Xianpei Han, Le Sun, and Andrew Yates.  BERT-QE: Contextualized Query Expansion for Document Re-ranking.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4718–4728, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.424.  URL https://aclanthology.org/2020.findings-emnlp.424.

EmbedDistill: A Geometric Knowledge Distillation
for Information Retrieval (Supplementary)

##  Appendix A Loss functions

(a) Score-based CE to DE distillation

(b) Score-based DE to DE distillation

Figure 2: Illustration of score-based distillation for IR (cf. Section 2.2).
Fig. 2(a) describes distillation from a teacher [CLS]-pooled CE model to a
student DE model. Fig. 2(b) depicts distillation from a teacher DE model to a
student DE model. Here, both distillation setups employ symmetric DE
configurations where query and document encoders of the student model are of
the same size.

Here, we state various (per-example) loss functions that most commonly define
training objectives for IR models. Typically, one hot training with original
label is performed using softmax-based cross-entropy loss functions:

$\displaystyle\ell\big{(}s_{q,\mathbf{d}_{i}},\mathbf{y}_{i}\big{)}~{}=~{}{-}\sum_{j\in[L]}y_{i,j}\cdot\log\Big{(}\frac{\exp({s(q_{i},d_{i,j})})}{\sum\limits_{j^{\prime}\in[L]}\exp({s(q_{i},d_{i,j^{\prime}})})}\Big{)}.$
(9)

Alternatively, it is also common to employ a one-vs-all loss function based on
binary cross-entropy loss as follows:

$\displaystyle\ell\big{(}s_{q,\mathbf{d}_{i}},\mathbf{y}_{i}\big{)}~{}=~{}{-}\sum_{j\in[L]}\bigg{(}y_{i,j}\cdot\log\Big{(}\frac{1}{1~{}+~{}\exp({-s(q_{i},d_{i,j})})}\Big{)}~{}~{}+$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad(1-y_{i,j})\cdot\log\Big{(}\frac{1}{1~{}+~{}\exp({s(q_{i},d_{i,j})})}\Big{)}\bigg{)}.$
(10)

Note that $\mathbf{d}_{i}=\\{d_{i,j}\\}_{j\in[L]}$ can be expanded to include
various forms of negatives such as in-batch negatives [Karpukhin et al.,
2020b] and sampled negatives [Bengio and Senecal, 2008].

As for distillation (cf. Fig. 2), one can define a distillation objective
based on the softmax-based cross-entropy loss as:666It is common to employ
temperature scaling with softmax operation. We do not explicitly show the
temperature parameter for ease of exposition.

$\displaystyle\ell_{\rm d}\big{(}s^{\rm s}_{q,\mathbf{d}_{i}},s^{\rm
t}_{q,\mathbf{d}_{i}}\big{)}=-\sum_{j\in[L]}\bigg{(}\frac{\exp({s^{\rm
t}_{i,j}})}{\sum_{j^{\prime}\in[L]}\exp({s^{\rm
t}_{i,j^{\prime}}})}\cdot\log\Big{(}\frac{\exp({s^{\rm
s}_{i,j}})}{\sum_{j^{\prime}\in[L]}\exp({s^{\rm
s}_{i,j^{\prime}}})}\Big{)}\bigg{)},$ (11)

where $s^{\rm t}_{i,j}:=s^{\rm t}(q_{i},d_{i,j})$ and $s^{\rm s}_{i,j}:=s^{\rm
s}(q_{i},d_{i,j})$ denote the teacher and student scores, respectively. On the
other hand, the distillation objective with the binary cross-entropy takes the
form:

$\displaystyle\ell_{\rm d}\big{(}s^{\rm s}_{q,\mathbf{d}_{i}},s^{\rm
t}_{q,\mathbf{d}_{i}}\big{)}~{}$
$\displaystyle=~{}{-}\sum_{j\in[L]}\bigg{(}\frac{1}{1~{}+~{}\exp({-s^{\rm
t}_{i,j}})}\cdot\log\Big{(}\frac{1}{1~{}+~{}\exp({-s^{\rm
s}_{i,j}})}\Big{)}~{}~{}~{}+$
$\displaystyle\qquad\qquad\qquad\qquad\frac{1}{1~{}+~{}\exp({s^{\rm
t}_{i,j}})}\cdot\log\Big{(}\frac{1}{1~{}+~{}\exp({s^{\rm
s}_{i,j}})}\Big{)}\bigg{)}.$ (12)

Finally, distillation based on the meas square error (MSE) loss (aka. logit
matching) employs the following loss function:

$\displaystyle\ell_{\rm d}\big{(}s^{\rm s}_{q,\mathbf{d}_{i}},s^{\rm
t}_{q,\mathbf{d}_{i}}\big{)}=\sum\limits_{j\in[L]}\big{(}s^{\rm
t}(q_{i},d_{i,j})-s^{\rm s}(q_{i},d_{i,j})\big{)}^{2}.$ (13)

##  Appendix B Dual pooling details

In this work, we focus on two kinds of dual pooling strategies:

  * •

Special tokens-based dual pooling. Let ${\rm pool}_{{\rm CLS}}$ and ${\rm
pool}_{{\rm SEP}}$ denote the pooling operations that return the embeddings of
the [CLS] and [SEP] tokens, respectively. We define

$\displaystyle{\tt emb}^{\rm t}_{q\leftarrow(q,d)}={\rm pool}_{{\tt
CLS}}\big{(}{\rm Enc}^{\rm t}(\tilde{o})\big{)},$ $\displaystyle{\tt emb}^{\rm
t}_{d\leftarrow(q,d)}={\rm pool}_{{\tt SEP}}\big{(}{\rm Enc}^{\rm
t}(\tilde{o})\big{)},$ (14)

where $\tilde{o}$ denotes the input token sequence to the Transformers-based
encoder, which consists of { query, document, special } tokens.

  * •

Segment-based weighted-mean dual pooling. Let ${\rm Enc}^{\rm
t}(\tilde{o})|_{Q}$ and ${\rm Enc}^{\rm t}(\tilde{o})|_{D}$ denote the final
query token embeddings and document token embeddings produced by the encoder,
respectively. We define the _proxy_ query and document embeddings

$\displaystyle{\tt emb}^{\rm t}_{q\leftarrow(q,d)}={\rm mean_{wt}}\big{(}{\rm
Enc}^{\rm t}(\tilde{o})|_{Q}\big{)},$ $\displaystyle{\tt emb}^{\rm
t}_{d\leftarrow(q,d)}={\rm mean_{wt}}\big{(}{\rm Enc}^{\rm
t}(\tilde{o})|_{D}\big{)},$ (15)

where ${\rm mean_{wt}}(\cdot)$ denotes the weighted mean operation. We employ
the specific weighting scheme where each token receives a weight equal to the
inverse of the square root of the token-sequence length.

##  Appendix C Deferred details and proofs from Section 3

In this section we present more precise statements and proofs of Theorem 3.1
and Proposition 3.2 (stated informally in Section 3 of the main text) along
with the necessary background. First, for the ease of exposition, we define
new notation which will facilitate theoretical analysis in this section.

Notation. Denote the query and document encoders as
$f\colon\mathscr{Q}\to\mathbb{R}^{k}$ and
$g\colon\mathscr{D}\to\mathbb{R}^{k}$ for the student, and
$F\colon\mathscr{Q}\to\mathbb{R}^{k},G\colon\mathscr{D}\to\mathbb{R}^{k}$ for
the teacher (in the dual-encoder setting). With $q$ denoting a query and $d$
denoting a document, $f(q)$ and $g(d)$ then denote query and document
embeddings, respectively, generated by the student. We define $F(q)$ and
$G(d)$ similarly for embeddings by the teacher.777Note that, as per the
notations in the main text, we have $(f,g)=({\rm Enc}^{\rm s}_{Q},{\rm
Enc}^{\rm s}_{D})$ and $(F,G)=({\rm Enc}^{\rm t}_{Q},{\rm Enc}^{\rm t}_{D})$.
Similarly, we have $({\tt emb}^{\rm t}_{q},{\tt emb}^{\rm t}_{d})=(f(q),g(d))$
and $({\tt emb}^{\rm t}_{q},{\tt emb}^{\rm t}_{d})=(F(q),G(d))$.

######  Theorem C.1 (Formal statement of Theorem 3.1).

Let $\mathscr{F}$ and $\mathscr{G}$ denote the function classes for the query
and document encoders for the student model, respectively. Given $n$ examples
$\mathscr{S}_{n}=\\{(q_{i},d_{i},y_{i})\\}_{i\in[n]}\subset\mathscr{Q}\times\mathscr{D}\times\\{0,1\\}$,
let $s^{\rm s}(q,d):=s^{f,g}(q_{i},d_{i})=f(q_{i})^{T}g(d_{i})$ be the scores
assigned to the $(q_{i},d_{i})$ pair by a dual-encoder model with
$f\in\mathscr{F}$ and $g\in\mathscr{G}$ as query and document encoders,
respectively. Let $\ell$ and $\ell_{\rm d}$ be the binary cross-entropy loss
(cf. Eq. 10 with $L=1$) and the distillation-specific loss based on it (cf.
Eq. 12 with $L=1$), respectively. In particular,

$\displaystyle\ell(s^{F,G}(q_{i},d_{i}),y_{i})$
$\displaystyle:=-y_{i}\log\sigma\left(F(q_{i})^{\top}G(d_{i})\right)-(1-y_{i})\log\left[1-\sigma\left(F(q_{i})^{\top}G(d_{i})\right)\right]$
$\displaystyle\ell_{\rm d}(s^{f,g}(q_{i},d_{i}),s^{F,G}(q_{i},d_{i}))$
$\displaystyle:=-\sigma\left(F(q_{i})^{\top}G(d_{i})\right)\cdot\log\sigma\left(f(q_{i})^{\top}g(d_{i})\right)\;-$
$\displaystyle\qquad\qquad[1-\sigma\left(F(q_{i})^{\top}G(d_{i})\right)]\cdot\log\left[1-\sigma\left(f(q_{i})^{\top}g(d_{i})\right)\right],$

where $\sigma$ is the sigmoid function and $s^{\rm t}:=s^{F,G}$ denotes the
teacher dual-encoder model with $F$ and $Q$ as its query and document
encoders, respectively. Assume that

  1. 1.

All encoders $f,g,F,$ and $G$ have the same output dimension.

  2. 2.

$\exists~{}K\in(0,\infty)$ such that
$\sup_{q\in\mathcal{Q}}\max\left\\{\|f(q)\|_{2},\|F(q)\|_{2}\right\\}\leq K$
and $\sup_{d\in\mathcal{D}}\max\left\\{\|g(d)\|_{2},\|G(d)\|_{2}\right\\}\leq
K$.

Then, we have

$\displaystyle\underbrace{\mathbb{E}\left[s^{f,g}(q,d)\right]}_{:=R(s^{\rm
s})=R(s^{f,g})}-\underbrace{\mathbb{E}\left[s^{F,G}(q,d)\right]}_{:=R(s^{\rm
t})=R(s^{F,G})}\leq\underbrace{\sup_{(f,g)\in\mathscr{F}\times\mathscr{G}}\left|R(s^{f,g},s^{F,G};\mathscr{S}_{n})-\mathbb{E}\left[\ell_{\rm
d}\big{(}s^{f,g}(q,d),s^{F,G}(q,d)\big{)}\right]\right|}_{:=\mathscr{E}_{n}(\mathscr{F},\mathscr{G})}\;$
$\displaystyle\qquad\qquad\quad+2K\Big{(}\underbrace{\frac{1}{n}\sum_{i\in[n]}\|g(d_{i})-G(d_{i})\|_{2}}_{:=R_{{\rm
Emb},D}({\rm t},{\rm
s};\mathscr{S}_{n})}\;+\underbrace{\frac{1}{n}\sum_{i\in[n]}\|f(q_{i})-F(q_{i})\|_{2}\Big{)}}_{:=R_{{\rm
Emb},Q}({\rm t},{\rm
s};\mathscr{S}_{n})}\;+\underbrace{R(s^{F,G};\mathscr{S}_{n})-R(s^{F,G})}_{:=\Delta(s^{\rm
t};\mathscr{S}_{n})}$
$\displaystyle\qquad\qquad\quad+K^{2}\Big{(}\mathbb{E}\left[\left|\sigma(F(q)^{\top}G(d))-y\right|\right]+\frac{1}{n}\sum_{i\in[n]}\left|\sigma\left(F(q_{i})^{\top}G(d_{i})\right)-y_{i}\right|\Big{)}.$
(16)

###### Proof.

Note that

$\displaystyle
R(s^{f,g})-R(s^{F,G})=R(s^{f,g})-R(s^{f,g},s^{F,G})+R(s^{f,g},s^{F,G})-R(s^{F,G})$
$\displaystyle\qquad\overset{(a)}{\leq}K^{2}\mathbb{E}\left[\left|\sigma(F(q)^{\top}G(d))-y\right|\right]+R(s^{f,g},s^{F,G})-R(s^{F,G})$
$\displaystyle\qquad=K^{2}\mathbb{E}\left[\left|\sigma(F(q)^{\top}G(d))-y\right|\right]+R(s^{f,g},s^{F,G})-R(s^{f,g},s^{F,G};\mathscr{S}_{n})\;+$
$\displaystyle\qquad\qquad R(s^{f,g},s^{F,G};\mathscr{S}_{n})-R(s^{F,G})$
$\displaystyle\qquad\overset{(b)}{\leq}K^{2}\mathbb{E}\left[\left|\sigma(F(q)^{\top}G(d))-y\right|\right]+\mathscr{E}_{n}(\mathscr{F},\mathscr{G})+R(s^{f,g},s^{F,G};\mathscr{S}_{n})-R(s^{F,G})$
$\displaystyle\qquad=K^{2}\mathbb{E}\left[\left|\sigma(F(q)^{\top}G(d))-y\right|\right]+\mathscr{E}_{n}(\mathscr{F},\mathscr{G})+R(s^{f,g},s^{F,G};\mathscr{S}_{n})-R(s^{F,G};\mathscr{S}_{n})\;+$
$\displaystyle\qquad\qquad R(s^{F,G};\mathscr{S}_{n})-R(s^{F,G})$
$\displaystyle\qquad\overset{(c)}{\leq}K^{2}\mathbb{E}\left[\left|\sigma(F(q)^{\top}G(d))-y\right|\right]+\mathscr{E}_{n}(\mathscr{F},\mathscr{G})+\underbrace{R(s^{F,G};\mathscr{S}_{n})-R(s^{F,G})}_{:=\Delta(s^{\rm
t};\mathscr{S}_{n})}\;+$
$\displaystyle\qquad\qquad\frac{2K}{n}\sum_{i\in[n]}\|g(d_{i})-G(d_{i})\|_{2}\;+\frac{2K}{n}\sum_{i\in[n]}\|f(q_{i})-F(q_{i})\|_{2}\;+$
$\displaystyle\qquad\qquad\frac{K^{2}}{n}\sum_{i\in[n]}\left|\sigma\left(F(q_{i})^{\top}G(d_{i})\right)-y_{i}\right|$
(17)

where $(a)$ follows from Lemma C.3, $(b)$ follows from the definition of
$\mathscr{E}_{n}(\mathscr{F},\mathscr{G})$, and $(c)$ follows from Proposition
C.2. ∎

###  C.1 Bounding the difference between student’s empirical _distillation_
risk and teacher’s empirical risk

######  Lemma C.2.

Given $n$ examples
$\mathscr{S}_{n}=\\{(q_{i},d_{i},y_{i})\\}_{i\in[n]}\subset\mathscr{Q}\times\mathscr{D}\times\\{0,1\\}$,
let $s^{f,g}(q_{i},d_{i})=f(q_{i})^{T}g(d_{i})$ be the scores assigned to the
$(q_{i},d_{i})$ pair by a dual-encoder model with $f$ and $g$ as query and
document encoders, respectively. Let $\ell$ and $\ell_{\rm d}$ be the binary
cross-entropy loss (cf. Eq. 10 with $L=1$) and the distillation-specific loss
based on it (cf. Eq. 12 with $L=1$), respectively. In particular,

$\displaystyle\ell(s^{F,G}(q_{i},d_{i}),y_{i})$
$\displaystyle:=-y_{i}\log\sigma\left(F(q_{i})^{\top}G(d_{i})\right)-(1-y_{i})\log\left[1-\sigma\left(F(q_{i})^{\top}G(d_{i})\right)\right]$
$\displaystyle\ell_{\rm d}(s^{f,g}(q_{i},d_{i}),s^{F,G}(q_{i},d_{i}))$
$\displaystyle:=-\sigma\left(F(q_{i})^{\top}G(d_{i})\right)\cdot\log\sigma\left(f(q_{i})^{\top}g(d_{i})\right)\;-$
$\displaystyle\qquad\qquad[1-\sigma\left(F(q_{i})^{\top}G(d_{i})\right)]\cdot\log\left[1-\sigma\left(f(q_{i})^{\top}g(d_{i})\right)\right],$

where $\sigma$ is the sigmoid function and $s^{F,G}$ denotes the teacher dual-
encoder model with $F$ and $Q$ as its query and document encoders,
respectively. Assume that

  1. 1.

All encoders $f,g,F,$ and $G$ have the same output dimension $k\geq 1$.

  2. 2.

$\exists~{}K\in(0,\infty)$ such that
$\sup_{q\in\mathcal{Q}}\max\left\\{\|f(q)\|_{2},\|F(q)\|_{2}\right\\}\leq K$
and $\sup_{d\in\mathcal{D}}\max\left\\{\|g(d)\|_{2},\|G(d)\|_{2}\right\\}\leq
K$.

Then, we have

$\displaystyle\frac{1}{n}\sum_{i\in[n]}\ell_{\rm
d}\big{(}s^{f,g}(q_{i},d_{i}),s^{F,G}(q_{i},d_{i})\big{)}-\frac{1}{n}\sum_{i\in[n]}\ell\big{(}s^{F,G}(q_{i},d_{i}),y_{i}\big{)}\leq\;$
$\displaystyle\qquad\qquad\qquad\qquad\frac{2K}{n}\sum_{i\in[n]}\|g(d_{i})-G(d_{i})\|_{2}\;+\frac{2K}{n}\sum_{i\in[n]}\|f(q_{i})-F(q_{i})\|_{2}\;+$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\frac{K^{2}}{n}\sum_{i\in[n]}\left|\sigma\left(F(q_{i})^{\top}G(d_{i})\right)-y_{i}\right|.$
(18)

###### Proof.

We first note that the distillation loss can be rewritten as

$\displaystyle\ell_{\rm d}\big{(}s^{f,g}(q,d),s^{F,G}(q,d)\big{)}$
$\displaystyle=\left(1-\sigma(F(q)^{\top}G(d)\right)f(q)^{\top}g(d)+\gamma(-f(q)^{\top}g(d)),$

where $\gamma(v):=\log[1+e^{v}]$ is the softplus function. Similarly, the one-
hot (label-dependent) loss can be rewritten as

$\displaystyle\ell\big{(}s^{F,G}(q,d),y\big{)}$
$\displaystyle=(1-y)F(q)^{\top}G(d)+\gamma(-F(q)^{\top}G(d)).$

Recall from our notation in Section 2 that

$\displaystyle R(s^{f,g},s^{F,G};\mathscr{S}_{n})$
$\displaystyle:=\frac{1}{n}\sum_{i\in[n]}\ell_{\rm
d}\big{(}s^{f,g}(q_{i},d_{i}),s^{F,G}(q_{i},d_{i})\big{)},$ (19)
$\displaystyle R(s^{F,G};\mathscr{S}_{n})$
$\displaystyle:=\frac{1}{n}\sum_{i\in[n]}\ell\big{(}s^{F,G}(q_{i},d_{i}),y_{i}\big{)},$
(20)

as the empirical risk based on the distillation loss, and the empirical risk
based on the label-dependent loss, respectively. With this notation, the
quantity to upper bound can be rewritten as

$\displaystyle
R(s^{f,g},s^{F,G};\mathscr{S}_{n})-R(s^{F,G};\mathscr{S}_{n})=\underbrace{R(s^{f,g},,s^{F,G};\mathscr{S}_{n})-R(s^{f,G},s^{F,G};\mathscr{S}_{n})}_{:=\square_{1}}+$
$\displaystyle\qquad\qquad\quad\underbrace{R(s^{f,G},s^{F,G};\mathscr{S}_{n})-R(s^{F,G},s^{F,G};\mathscr{S}_{n})}_{:=\square_{2}}+\underbrace{R(s^{F,G},s^{F,G};\mathscr{S}_{n})-R(s^{F,G};\mathscr{S}_{n})}_{:=\square_{3}}.$
(21)

We start by bounding $\square_{1}$ as

$\displaystyle\square_{1}$
$\displaystyle=\frac{1}{n}\sum_{i\in[n]}\Big{(}\ell_{\rm
d}\big{(}s^{f,g}(q_{i},d_{i}),s^{F,G}(q_{i},d_{i})\big{)}-\ell_{\rm
d}\big{(}s^{f,G}(q_{i},d_{i}),s^{F,G}(q_{i},d_{i})\big{)}\Big{)}$
$\displaystyle=\frac{1}{n}\sum_{i\in[n]}\Big{(}\left(1-\sigma(F(q_{i})^{\top}G(d_{i}))\right)f(q_{i})^{\top}g(d_{i})+\gamma(-f(q_{i})^{\top}g(d_{i}))$
$\displaystyle\qquad\qquad-\left(1-\sigma(F(q_{i})^{\top}G(d_{i}))\right)f(q_{i})^{\top}G(d_{i})-\gamma(-f(q_{i})^{\top}G(d_{i}))\Big{)}$
$\displaystyle=\frac{1}{n}\sum_{i\in[n]}\Big{(}f(q_{i})^{\top}\big{(}g(d_{i})-G(d_{i})\big{)}\left(1-\sigma(F(q_{i})^{\top}G(d_{i}))\right)\;$
$\displaystyle\qquad\qquad+\gamma(-f(q_{i})^{\top}g(d_{i}))-\gamma(-f(q_{i})^{\top}G(d_{i}))\Big{)}$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\frac{1}{n}\sum_{i\in[n]}\Big{(}f(q_{i})^{\top}\big{(}g(d_{i})-G(d_{i})\big{)}\left(1-\sigma(F(q_{i})^{\top}G(d_{i}))\right)+\left|f(q_{i})^{\top}g(d_{i})-f(q_{i})^{\top}G(d_{i})\right|\Big{)}$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}\frac{1}{n}\sum_{i\in[n]}\Big{(}\|f(q_{i})\|\|g(d_{i})-G(d_{i})\|\left(1-\sigma(F(q_{i})^{\top}G(d_{i}))\right)+\|f(q_{i})\|\|g(d_{i})-G(d_{i})\|\Big{)}$
$\displaystyle\stackrel{{\scriptstyle}}{{\leq}}\frac{K}{n}\sum_{i\in[n]}\|g(d_{i})-G(d_{i})\|_{2}\left(2-\sigma(F(q_{i})^{\top}G(d_{i}))\right)\Big{)}$
$\displaystyle\leq\frac{2K}{n}\sum_{i\in[n]}\|g(d_{i})-G(d_{i})\|_{2},$ (22)

where at $(a)$ we use the fact that $\gamma$ is a Lipschitz continuous
function with Lipschitz constant 1, and at $(b)$ we use Cauchy-Schwarz
inequality.

Similarly for $\square_{2}$, we proceed as

$\displaystyle\square_{2}$
$\displaystyle=\frac{1}{n}\sum_{i\in[n]}\Big{(}\ell_{\rm
d}\big{(}s^{f,G}(q_{i},d_{i}),s^{F,G}(q_{i},d_{i})\big{)}-\ell_{\rm
d}\big{(}s^{F,G}(q_{i},d_{i}),s^{F,G}(q_{i},d_{i})\big{)}\Big{)}$
$\displaystyle=\frac{1}{n}\sum_{i\in[n]}\Big{(}\left(1-\sigma(F(q_{i})^{\top}G(d_{i}))\right)f(q_{i})^{\top}G(d_{i})+\gamma(-f(q_{i})^{\top}G(d_{i}))$
$\displaystyle\qquad\qquad-\left(1-\sigma(F(q_{i})^{\top}G(d_{i}))\right)F(q_{i})^{\top}G(d_{i})-\gamma(-F(q_{i})^{\top}G(d_{i}))\Big{)}$
$\displaystyle=\frac{1}{n}\sum_{i\in[n]}\Big{(}G(d_{i})^{\top}(f(q_{i})-F(q_{i}))\left(1-\sigma(F(q_{i})^{\top}G(d_{i}))\right)\;$
$\displaystyle\qquad\qquad+\gamma(-f(q_{i})^{\top}G(d_{i}))-\gamma(-F(q_{i})^{\top}G(d_{i}))\Big{)}$
$\displaystyle\leq\frac{1}{n}\sum_{i\in[n]}\Big{(}\|G(d_{i})\|\|f(q_{i})-F(q_{i})\|+\left|f(q_{i})^{\top}G(d_{i})-F(q_{i})^{\top}G(d_{i})\right|\Big{)}$
$\displaystyle\leq\frac{2K}{n}\sum_{i\in[n]}\|f(q_{i})-F(q_{i})\|_{2}.$ (23)

$\square_{3}$ can be bounded as

$\displaystyle\square_{3}$
$\displaystyle=R(s^{F,G},s^{F,G};\mathscr{S}_{n})-R(s^{F,G};\mathscr{S}_{n})$
$\displaystyle=\frac{1}{n}\sum_{i\in[n]}\Big{(}\ell_{\rm
d}\big{(}s^{F,G}(q_{i},d_{i}),s^{F,G}(q_{i},d_{i})\big{)}-\ell\big{(}s^{F,G}(q_{i},d_{i}),y_{i}\big{)}\Big{)}$
$\displaystyle=\frac{1}{n}\sum_{i\in[n]}\Big{(}\left(1-\sigma(F(q_{i})^{\top}G(d_{i}))\right)F(q_{i})^{\top}G(d_{i})+\gamma(-F(q_{i})^{\top}G(d_{i}))$
$\displaystyle\qquad\qquad-(1-y_{i})F(q_{i})^{\top}G(d_{i})-\gamma(-F(q_{i})^{\top}G(d_{i}))\Big{)}$
$\displaystyle=\frac{1}{n}\sum_{i\in[n]}\Big{(}\big{(}1-\sigma(F(q_{i})^{\top}G(d_{i}))-(1-y_{i})\big{)}F(q_{i})^{\top}G(d_{i})\Big{)}$
$\displaystyle\leq\frac{K^{2}}{n}\sum_{i\in[n]}\left|\sigma(F(q_{i})^{\top}G(d_{i}))-y_{i}\right|.$
(24)

Combining Eq. 21, 22, 23, and 24 establishes the bound in Eq. C.2. ∎

######  Lemma C.3.

Given an example $(q,d,y)\in\mathscr{Q}\times\mathscr{D}\times\\{0,1\\}$, let
$s^{f,g}(q,d)=f(q)^{T}g(d)$ be the scores assigned to the $(q,d)$ pair by a
dual-encoder model with $f$ and $g$ as query and document encoders,
respectively. Let $\ell$ and $\ell_{\rm d}$ be the binary cross-entropy loss
(cf. Eq. 10 with $L=1$) and the distillation-specific loss based on it (cf.
Eq. 12 with $L=1$), respectively. In particular,

$\displaystyle\ell(s^{f,g}(q,d),y)$
$\displaystyle:=-y\log\sigma\left(f(q)^{\top}g(d)\right)-(1-y)\log\left[1-\sigma\left(f(q)^{\top}g(d)\right)\right]$
$\displaystyle\ell_{\rm d}(s^{f,g}(q,d),s^{F,G}(q,d))$
$\displaystyle:=-\sigma\left(F(q)^{\top}G(d)\right)\cdot\log\sigma\left(f(q)^{\top}g(d)\right)\;-$
$\displaystyle\qquad\qquad[1-\sigma\left(F(q)^{\top}G(d)\right)]\cdot\log\left[1-\sigma\left(f(q)^{\top}g(d)\right)\right],$

where $\sigma$ is the sigmoid function and $s^{F,G}$ denotes the teacher dual-
encoder model with $F$ and $Q$ as its query and document encoders,
respectively. Assume that

  1. 1.

All encoders $f,g,F,$ and $G$ have the same output dimension $k\geq 1$.

  2. 2.

$\exists~{}K\in(0,\infty)$ such that
$\sup_{q\in\mathcal{Q}}\max\left\\{\|f(q)\|_{2},\|F(q)\|_{2}\right\\}\leq K$
and $\sup_{d\in\mathcal{D}}\max\left\\{\|g(d)\|_{2},\|G(d)\|_{2}\right\\}\leq
K$.

Then, we have

$\displaystyle\underbrace{\mathbb{E}\big{[}\ell\big{(}s^{f,g}(q,d),y\big{)}\big{]}}_{:=R(s^{f,g})}-\underbrace{\mathbb{E}\left[\ell_{\rm
d}\big{(}s^{f,g}(q,d),s^{F,G}(q,d)\big{)}\right]}_{:=R(s^{f,g},s^{F,G})}\leq\;K_{Q}K_{D}\mathbb{E}\left[\left|\sigma(F(q)^{\top}G(d))-y\right|\right]$
(25)

where expectation are defined by a joint distribution $\mathbb{P}(q,d,y)$ over
$\mathscr{Q}\times\mathscr{D}\times\\{0,1\\}$

###### Proof.

Similar to the proof of Proposition C.2, we utilize the fact that

$\displaystyle\ell\big{(}s^{F,G}(q,d),y\big{)}$
$\displaystyle=(1-y)F(q)^{\top}G(d)+\gamma(-F(q)^{\top}G(d)),$
$\displaystyle\ell_{\rm d}\big{(}s^{f,g}(q,d),s^{F,G}(q,d)\big{)}$
$\displaystyle=\left(1-\sigma(F(q)^{\top}G(d)\right)f(q)^{\top}g(d)+\gamma(-f(q)^{\top}g(d)),$

where $\gamma(v):=\log[1+e^{v}]$ is the softplus function. Now,

$\displaystyle\mathbb{E}\left[\ell\big{(}s^{f,g}(q,d),y\big{)}-\ell_{\rm
d}\big{(}s^{f,g}(q,d),s^{F,G}(q,d)\big{)}\right]$ (26)
$\displaystyle\qquad\qquad\qquad{=}\mathbb{E}\left[(1-y)f(q)^{\top}g(d)+\gamma(-f(q)^{\top}g(d))\right]$
$\displaystyle\qquad\qquad\qquad\phantom{=}-\mathbb{E}\left[\left(1-\sigma(F(q)^{\top}G(d))\right)f(q)^{\top}g(d)+\gamma(-f(q)^{\top}g(d))\right]$
$\displaystyle\qquad\qquad\qquad=\mathbb{E}\left[\Big{(}1-y-\big{(}1-\sigma(F(q)^{\top}G(d))\big{)}\Big{)}F(q)^{\top}G(d)\right]$
$\displaystyle\qquad\qquad\qquad\leq
K^{2}\mathbb{E}\left[\left|\sigma(F(q)^{\top}G(d))-y\right|\right],$ (27)

which completes the proof. ∎

###  C.2 Uniform deviation bound

Let $\mathscr{F}$ denote the class of functions that map queries in
$\mathscr{Q}$ to their embeddings in $\mathbb{R}^{k}$ via the query encoder.
Define $\mathscr{G}$ analogously for the doc encoder, which consists of
functions that map documents in $\mathscr{D}$ to their embeddings in
$\mathbb{R}^{k}$. To simplify exposition, we assume that each training example
consists of a single relevant or irrelevant document for each query, i.e.,
$L=1$ in Section 2. Let

$\mathscr{F}\mathscr{G}=\\{(q,d)\mapsto
f(q)^{\top}g(d)\;\mid\;f\in\mathscr{F},g\in\mathscr{G}\\}$

Given $\mathscr{S}_{n}=\\{(q_{i},d_{i},y_{i}):i\in[n]\\}$, let
$N(\epsilon,\mathscr{H})$ denote the ${\epsilon}$-covering number of a
function class $\mathscr{H}$ with respect to $L_{2}(\mathbb{P}_{n})$ norm,
where
$\|h\|_{L_{2}(\mathbb{P}_{n})}^{2}:=\|h\|_{n}^{2}:=\frac{1}{n}\sum_{i=1}^{n}\|h(q_{i},d_{i})\|_{2}^{2}$.
Depending on the context, the functions in $\mathscr{H}$ may map to
$\mathbb{R}$ or $\mathbb{R}^{d}$.

######  Proposition C.4.

Let $s^{\mathrm{t}}$ be scorer of a teacher model and $\ell_{\rm d}$ be a
distillation loss function which is $L_{\ell_{\rm d}}$-Lipschitz in its first
argument. Let the embedding functions in $\mathscr{F}$ and $\mathscr{G}$
output vectors with $\ell_{2}$ norms at most $K$. Define the uniform deviation

$\mathscr{E}_{n}(\mathscr{F},\mathscr{G})=\sup_{f\in\mathscr{F},g\in\mathscr{G}}\left|\frac{1}{n}\sum\nolimits_{i\in[n]}\ell_{\rm
d}\big{(}f(q_{i})^{\top}g(d_{i}),s_{q_{i},d_{i}}^{\rm
t}\big{)}-\mathbb{E}_{q,d}\ell_{\rm d}\big{(}f(q)^{\top}g(d),s_{q,d}^{\rm
t}\big{)}\right|.$

For any $g^{*}\in\mathscr{G}$, we have

$\displaystyle\mathbb{E}_{\mathscr{S}_{n}}\mathscr{E}_{n}(\mathscr{F},\mathscr{G})$
$\displaystyle\leq\mathbb{E}_{\mathscr{S}_{n}}\frac{48KL_{\ell_{\rm
d}}}{\sqrt{n}}\int_{0}^{\infty}\sqrt{\log N(u,\mathscr{F})+\log
N(u,\mathscr{G})}\;du,$
$\displaystyle\mathbb{E}_{\mathscr{S}_{n}}\mathscr{E}_{n}(\mathscr{F},\\{g^{*}\\})$
$\displaystyle\leq\mathbb{E}_{\mathscr{S}_{n}}\frac{48KL_{\ell_{\rm
d}}}{\sqrt{n}}\int_{0}^{\infty}\sqrt{\log N(u,\mathscr{F})}\;du.$

###### Proof of Proposition C.4.

We first symmetrize excess risk to get Rademacher complexity, then bound the
Rademacher complexity with Dudley’s entropy integral.

For a training set $\mathscr{S}_{n}$, the empirical Rademacher complexity of a
class of functions $\mathscr{H}$ that maps $\mathscr{Q}\times\mathscr{D}$ to
$\mathbb{R}$ is defined by

$\mathrm{Rad}_{n}(\mathscr{H})=\mathbb{E}_{\sigma}\sup_{h\in\mathscr{H}}\frac{1}{n}\sum_{i=1}^{n}\varepsilon_{i}h(q_{i},d_{i}),$

where $\\{\varepsilon_{i}\\}$ denote i.i.d. Rademacher random variables taking
the value in $\\{+1,-1\\}$ with equal probability. By symmetrization [Bousquet
et al., 2004] and the fact that $\ell_{\rm d}$ is $L_{\ell_{\rm d}}$-Lipschitz
in its first argument, we get

$E_{\mathscr{S}_{n}}\mathscr{E}_{n}(\mathscr{F},\mathscr{G})\leq 2L_{\ell_{\rm
d}}\mathbb{E}_{\mathscr{S}_{n}}\mathrm{Rad}_{n}(\mathscr{F}\mathscr{G}).$

Then, Dudley’s entropy integral [see, e.g., Ledoux and Talagrand, 1991] gives

$\mathrm{Rad}_{n}(\mathscr{F}\mathscr{G})\leq\frac{12}{\sqrt{n}}\int_{0}^{\infty}\sqrt{\log
N(u,\mathscr{F}\mathscr{G})}\;du.$

From Lemma C.5 with $K_{Q}=K_{D}=K$, for any $u>0$,

$N(u,\mathscr{F}\mathscr{G})\leq
N\left(\frac{u}{2K},\mathscr{F}\right)N\left(\frac{u}{2K},\mathscr{G}\right).$

Putting these together,

$\displaystyle\mathbb{E}_{\mathscr{S}_{n}}\mathscr{E}_{n}(\mathscr{F},\mathscr{G})\leq\frac{24L_{\ell_{\rm
d}}}{\sqrt{n}}\int_{0}^{\infty}\sqrt{\log N(u/2K,\mathscr{F})+\log
N(u/2K,\mathscr{G})}\;du.$ (28)

Following the same steps with $\mathscr{G}$ replaced by $\\{g^{*}\\}$, we get

$\displaystyle\mathbb{E}_{\mathscr{S}_{n}}\mathscr{E}_{n}(\mathscr{F},\\{g^{*}\\})\leq\frac{24L_{\ell_{\rm
d}}}{\sqrt{n}}\int_{0}^{\infty}\sqrt{\log N(u/2K,\mathscr{F})}\;du$ (29)

By changing variable in Eq. 28 and Eq. 29, we get the stated bounds. ∎

For $f:\mathscr{Q}\to\mathbb{R}^{k},g:\mathscr{D}\to\mathbb{R}^{k}$, define
$fg:\mathscr{Q}\times\mathscr{D}\to\mathbb{R}$ by $fg(q,d)=f(q)^{\top}g(d).$

######  Lemma C.5.

Let $f_{1},\dots,f_{N}$ be an ${\epsilon}$-cover of $\mathscr{F}$ and
$g_{1},\dots,g_{M}$ be an ${\epsilon}$-cover of $\mathscr{G}$ in
$L_{2}(\mathbb{P}_{n})$ norm. Let
$\sup_{f\in\mathscr{F}}\sup_{q\in\mathscr{Q}}\|f(q)\|_{2}\leq K_{Q}$ and
$\sup_{g\in\mathscr{G}}\sup_{d\in\mathscr{D}}\|g(d)\|_{2}\leq K_{D}$. Then,

$\\{f_{i}g_{j}\mid i\in[N],j\in[M]\\}$

is a $(K_{Q}+K_{D}){\epsilon}$-cover of $\mathscr{F}\mathscr{G}$.

###### Proof of Lemma C.5.

For arbitrary $f\in\mathscr{F},g\in\mathscr{G}$, there exist
$\tilde{f}\in\\{f_{1},\dots,f_{N}\\},\tilde{g}\in\\{g_{1},\dots,g_{M}\\}$ such
that $\|f-\tilde{f}\|_{n}\leq{\epsilon},\|g-\tilde{g}\|_{n}\leq{\epsilon}.$ It
is sufficient to show that
$\|fg-\tilde{f}\tilde{g}\|_{n}\leq(K_{Q}+K_{D}){\epsilon}.$ Decomposing using
triangle inequality,

$\displaystyle\|fg-\tilde{f}\tilde{g}\|_{n}$
$\displaystyle=\|fg-f\tilde{g}+f\tilde{g}-\tilde{f}\tilde{g}\|_{n}$
$\displaystyle\leq\|fg-f\tilde{g}\|_{n}+\|f\tilde{g}-\tilde{f}\tilde{g}\|_{n}.$
(30)

To bound the first term, using Cauchy-Schwartz inequality, we can write

$\frac{1}{n}\sum_{i=1}^{n}\left(f(q_{i})^{\top}g(d_{i})-\tilde{f}(q_{i})^{\top}\tilde{g}(d_{i})\right)^{2}\leq\sup_{q\in\mathscr{Q}}\|f(q)\|_{2}^{2}\cdot\frac{1}{n}\sum_{i=1}^{n}\|(g-\tilde{g})(d_{i})\|_{2}^{2}.$

Therefore

$\|fg-f\tilde{g}\|_{n}\leq K_{Q}\|g-\tilde{g}\|_{n}\leq K_{Q}{\epsilon}.$

Similarly

$\|f\tilde{g}-\tilde{f}\tilde{g}\|_{n}\leq K_{D}\|f-\tilde{f}\|_{n}\leq
K_{D}{\epsilon}$

Plugging these in Eq. 30, we get

$\|fg-\tilde{f}\tilde{g}\|_{n}\leq(K_{Q}+K_{D}){\epsilon}.$

This completes the proof. ∎

##  Appendix D Evaluation metric details

For NQ, we evaluate models with full _strict_ recall metric, meaning that the
model is required to find a _golden_ passage from the whole set of candidates
(21M). Specifically, for $k\geq 1$, recall@$k$ or R@$k$ denotes the percentage
of questions for which the associated golden passage is among the $k$ passages
that receive the highest relevance scores by the model. In addition, we also
present results for _relaxed_ recall metric considered by Karpukhin et al.
[2020a], where R@$k$ denotes the percentage of questions where the
corresponding answer string is present in at least one of the $k$ passages
with the highest model (relevance) scores.

For both MSMARCO retrieval and re-ranking tasks, we follow the standard
evaluation metrics _Mean Reciprocal Rank_(MRR)@10 and _normalized Discounted
Cumulative Gain_ (nDCG)@10. For retrieval tasks, these metrics are computed
with respect to the whole set of candidates passages (8.8M). On the other
hand, for re-ranking task, the metrics are computed with respect to BM25
generated 1000 candidate passages – _the originally provided_ – for each
query. Please note that some papers use more powerful models (e.g., DE models)
to generate the top 1000 candidate passages, which is not a standard re-
ranking evaluation and should not be compared directly. We report 100 $\times$
MRR@10 and 100 $\times$ nDCG@10, as per the convention followed in the prior
works.

##  Appendix E Query generation details

We introduced query generation to encourage geometric matching in local
regions, which can aid in transferring more knowledge in confusing
neighborhoods. As expected, this further improves the distillation
effectiveness on top of the embedding matching in most cases. To focus on the
local regions, we generate queries from the observed examples by adding local
perturbation in the data manifold (embedding space). Specifically, we employ
an off-the-shelf encoder-decoder model – BART-base Lewis et al. [2020]. First,
we embed an observed query in the corresponding dataset. Second, we add a
small perturbation to the query embedding. Finally, we decode the perturbed
embedding to generate a new query in the input space. Formally, the generated
query $x^{\prime}$ given an original query $x$ takes the form
$x^{\prime}=\mathrm{Dec}(\mathrm{Enc}(x)+\epsilon)$, where $\mathrm{Enc}()$
and $\mathrm{Dec}()$ correspond to the encoder and the decoder from the off-
the-shelf model, respectively, and $\epsilon$ is an isotropic Gaussian noise.
Furthermore, we also randomly mask the original query tokens with a small
probability. We generate two new queries from an observed query and use them
as additional data points during our distillation procedure.

As a comparison, we tried adding the same size of random sampled queries
instead of the ones generated via the method described above. That did not
show any benefit, which justifies the use of our query/question generation
method.

##  Appendix F Experimental details and additional results

###  F.1 Additional training details

Optimization. For all of our experiments, we use ADAM weight decay optimizer
with a short warm up period and a linear decay schedule. We use the initial
learning rate of $10^{-5}$ and $2.8\times 10^{-5}$ for experiments on NQ and
MSMARCO, respectively. We chose batch sizes to be $128$.

###  F.2 Additional results on NQ

See Table 6 for the performance of various DE models on NQ, as measured by the
relaxed recall metric.

Table 6: Relaxed recall performance of various student DE models on NQ dev
set, including symmetric DE student model (67.5M or 11.3M transformer for both
encoders), and asymmetric DE student model (67.5M or 11.3M transformer as
query encoder and document embeddings inherited from the teacher). All
distilled students used the same teacher (110M parameter BERT-base models as
both encoders), with the performance (in terms of relaxed recall) of Recall@5
= 87.2, Recall@20 = 94.7, Recall@100 = 98.1. _Note: the proposed method can
achieve 100% of teacher’s performance even with 2/3 rd size of the query
encoder, and 92-97% with even 1/10th size._

Method | Recall@5 |  | Recall@20 |  | Recall@100
---|---|---|---|---|---
67.5M | 11.3M |  | 67.5M | 11.3M |  | 67.5M | 11.3M
Train student directly | 62.5 | 49.7 |  | 82.5 | 73.0 |  | 93.7 | 88.2
\+ Distill from teacher | 82.7 | 66.1 |  | 92.9 | 84.0 |  | 97.3 | 93.1
\+ Inherit document embeddings | 84.7 | 73.0 |  | 93.7 | 85.4 |  | 97.6 | 93.3
\+ Query embedding matching | 87.2 | 77.6 |  | 95.0 | 88.0 |  | 97.9 | 94.3
\+ Query generation | 87.8 | 80.3 |  | 94.8 | 89.9 |  | 98.0 | 95.6
Train student only using embedding |  |  |  |  |  |  |  |
matching and inherit doc embeddings | 86.4 | 69.1 |  | 94.2 | 81.6 |  | 97.7 | 89.9
\+ Query generation | 86.7 | 72.9 |  | 94.4 | 84.9 |  | 97.8 | 92.2

###  F.3 Additional results on MSMARCO

####  F.3.1 DE to DE distillation

See Table 7 for DE to DE distillation results on MSMARCO retrieval and re-
ranking task, as measured by the nDCG@10 metric (see Section 5.2 for the
results on MRR@10 metric).

Table 7: Performance of various DE models on MSMARCO dev set for both _re-
ranking_ (original top1000) and _retrieval_ tasks (full corpus). The teacher
model (110.1M parameter BERT-base models as both encoders) for reranking
achieves nDCG@10 of 42.7 and that for retrieval get nDCG@10 44.2. The table
shows performance (in nDCG@10) of the symmetric DE student model (67.5M or
11.3M transformer as both encoders), and asymmetric DE student model (67.5M or
11.3M transformer as query encoder and document embeddings inherited from the
teacher).

Method | Re-ranking |  | Retrieval
---|---|---|---
67.5M | 11.3M |  | 67.5M | 11.3M
Train student directly | 32.2 | 29.7 |  | 27.2 | 22.5
\+ Distill from teacher | 40.2 | 35.8 |  | 41.3 | 34.1
\+ Inherit doc embeddings | 41.0 | 37.7 |  | 42.2 | 36.2
\+ Query embedding matching | 42.0 | 40.8 |  | 43.8 | 41.9
\+ Query generation | 42.0 | 40.1 |  | 43.8 | 41.2
Train student using only |  |  |  |  |
embedding matching and |  |  |  |  |
inherit doc embeddings | 42.3 | 39.3 |  | 43.3 | 37.6
\+ Query generation | 42.3 | 39.9 |  | 43.4 | 39.2

###  F.4 Additional results on BEIR benchmark

See Table 8 (NDCG@10) and Table 9 (Recall@100) for BEIR benchmark results. All
numbers are from BEIR benchmark paper [Thakur et al., 2021]. As common
practice, non-public benchmark sets888https://github.com/beir-cellar/beir,
{BioASQ, Signal-1M(RT), TREC-NEWS, Robust04}, are removed from the table.
Following the original BEIR paper [Thakur et al., 2021] (Table 9 and Appendix
G from the original paper), we utilized Capped Recall@100 for TREC-COVID
dataset.

Table 8: In-domain and zero-shot retrieval performance on BEIR benchmark
[Thakur et al., 2021], as measured by nDCG@10. All the baseline number in the
table are taken from Thakur et al. [2021]. We exclude (in-domain) MSMARCO from
average computation as common practice.

Model ($\rightarrow$) | Lexical | Sparse | Dense
---|---|---|---
Dataset ($\downarrow$) | BM25 | DeepCT | SPARTA | docT5query | DPR | ANCE | TAS-B | GenQ |  SentenceBERT _(our teacher)_ |  EmbedDistill _(ours)_
MS MARCO | 22.8 |  29.6‡ |  35.1‡ |  33.8‡ | 17.7 |  38.8‡ |  40.8‡ |  40.8‡ |  47.1‡ |  46.6‡
TREC-COVID | 65.6 | 40.6 | 53.8 | 71.3 | 33.2 | 65.4 | 48.1 | 61.9 |  75.4 |  72.3
NFCorpus | 32.5 | 28.3 | 30.1 | 32.8 | 18.9 | 23.7 | 31.9 | 31.9 |  31.0 |  30.7
NQ | 32.9 | 18.8 | 39.8 | 39.9 |  47.4‡ | 44.6 | 46.3 | 35.8 |  51.5 |  50.8
HotpotQA | 60.3 | 50.3 | 49.2 | 58.0 | 39.1 | 45.6 | 58.4 | 53.4 |  58.0 |  56.0
FiQA-2018 | 23.6 | 19.1 | 19.8 | 29.1 | 11.2 | 29.5 | 30.0 | 30.8 |  31.8 |  29.5
ArguAna | 31.5 | 30.9 | 27.9 | 34.9 | 17.5 | 41.5 | 42.9 | 49.3 |  38.5 |  34.9
Touché-2020 | 36.7 | 15.6 | 17.5 | 34.7 | 13.1 | 24.0 | 16.2 | 18.2 |  22.9 |  24.7
CQADupStack | 29.9 | 26.8 | 25.7 | 32.5 | 15.3 | 29.6 | 31.4 | 34.7 |  33.5 |  30.6
Quora | 78.9 | 69.1 | 63.0 | 80.2 | 24.8 | 85.2 | 83.5 | 83.0 |  84.2 |  81.4
DBPedia | 31.3 | 17.7 | 31.4 | 33.1 | 26.3 | 28.1 | 38.4 | 32.8 |  37.7 |  35.9
SCIDOCS | 15.8 | 12.4 | 12.6 | 16.2 | 07.7 | 12.2 | 14.9 | 14.3 |  14.8 |  14.4
FEVER | 75.3 | 35.3 | 59.6 | 71.4 | 56.2 | 66.9 | 70.0 | 66.9 |  76.7 |  76.9
Climate-FEVER | 21.3 | 06.6 | 08.2 | 20.1 | 14.8 | 19.8 | 22.8 | 17.5 |  23.5 |  22.5
SciFact | 66.5 | 63.0 | 58.2 | 67.5 | 31.8 | 50.7 | 64.3 | 64.4 |  59.8 |  55.5
AVG (w/o MSMARCO) | 43.0 | 31.0 | 35.5 | 44.4 | 25.5 | 40.5 | 42.8 | 42.5 |  45.7 |  44.0

Table 9: In-domain and zero-shot retrieval performance on BEIR benchmark
[Thakur et al., 2021], as measured by Recall@100. All the baseline number in
the table are taken from Thakur et al. [2021]. $\ddagger$ indicates in-domain
retrieval performance. $\ast$ indicates capped recall following original
benchmark setup. We exclude (in-domain) MSMARCO from average computation as
common practice.

Model ($\rightarrow$) | Lexical | Sparse | Dense
---|---|---|---
Dataset ($\downarrow$) | BM25 | DeepCT | SPARTA | docT5query | DPR | ANCE | TAS-B | GenQ |  SentenceBERT _(our teacher)_ |  EmbedDistill _(ours)_
MS MARCO | 65.8 | 75.2‡ | 79.3‡ | 81.9‡ | 55.2 | 85.2‡ | 88.4‡ | 88.4‡ |  91.7‡ |  90.6‡
TREC-COVID | 49.8∗ | 34.7∗ | 40.9∗ | 54.1∗ | 21.2∗ | 45.7∗ | 38.7∗ | 45.6∗ |  54.1∗ |  48.8∗
NFCorpus | 25.0 | 23.5 | 24.3 | 25.3 | 20.8 | 23.2 | 28.0 | 28.0 |  27.7 |  26.7
NQ | 76.0 | 63.6 | 78.7 | 83.2 | 88.0‡ | 83.6 | 90.3 | 86.2 |  91.1 |  89.9
HotpotQA | 74.0 | 73.1 | 65.1 | 70.9 | 59.1 | 57.8 | 72.8 | 67.3 |  69.7 |  68.3
FiQA-2018 | 53.9 | 48.9 | 44.6 | 59.8 | 34.2 | 58.1 | 59.3 | 61.8 |  62.0 |  60.1
ArguAna | 94.2 | 93.2 | 89.3 | 97.2 | 75.1 | 93.7 | 94.2 | 97.8 |  89.2 |  87.8
Touché-2020 | 53.8 | 40.6 | 38.1 | 55.7 | 30.1 | 45.8 | 43.1 | 45.1 |  45.3 |  45.5
CQADupStack | 60.6 | 54.5 | 52.1 | 63.8 | 40.3 | 57.9 | 62.2 | 65.4 |  63.9 |  61.3
Quora | 97.3 | 95.4 | 89.6 | 98.2 | 47.0 | 98.7 | 98.6 | 98.8 |  98.5 |  98.1
DBPedia | 39.8 | 37.2 | 41.1 | 36.5 | 34.9 | 31.9 | 49.9 | 43.1 |  46.0 |  42.6
SCIDOCS | 35.6 | 31.4 | 29.7 | 36.0 | 21.9 | 26.9 | 33.5 | 33.2 |  32.5 |  31.5
FEVER | 93.1 | 73.5 | 84.3 | 91.6 | 84.0 | 90.0 | 93.7 | 92.8 |  93.9 |  93.8
Climate-FEVER | 43.6 | 23.2 | 22.7 | 42.7 | 39.0 | 44.5 | 53.4 | 45.0 |  49.3 |  47.6
SciFact | 90.8 | 89.3 | 86.3 | 91.4 | 72.7 | 81.6 | 89.1 | 89.3 |  88.9 |  87.2
AVG (w/o MSMARCO) | 63.4 | 55.9 | 56.2 | 64.7 | 47.7 | 60.0 | 64.8 | 64.2 |  65.1 |  63.5

###  F.5 Additional results with single-stage trained teachers

Hereby we evaluate EmbedDistill with a simple single-stage trained teachers
instead of teachers trained in complex multi-stage frameworks, in order to
test the generalizability of the method.

Similar to Table 1, we conducted an experiment on top of single-stage trained
teacher based on RoBERTa-base instead of AR2 [Zhang et al., 2022] in the main
text. We also changed the student to be based on DistilRoBERTa or RoBERTa-mini
accordingly for simplicity to use same tokenizer.

Table 10 demonstrates that EmbedDistill provides a significant boost of the
performance on top of standard distillation techniques similar to what we
observed in Table 1.

Table 10: Full recall performance of various student DE models on NQ dev set,
including symmetric DE student model, and asymmetric DE student models. All
students used the same _in-house teacher_ (124M parameter RoBERTa-base models
as both encoders), with the full Recall@5 = 64.6, Recall@20 = 81.7, and
Recall@100 = 91.5.

Method | 6-Layer (82M) |  | 4-Layer (16M)
---|---|---|---
R@5 | R@20 | R@100 |  | R@5 | R@20 | R@100
Train student directly | 41.9 | 64.5 | 82.0 |  | 39.5 | 59.9 | 76.3
\+ Distill from teacher | 48.3 | 67.2 | 80.9 |  | 44.9 | 61.1 | 74.8
\+ Inherit doc embeddings | 56.9 | 74.3 | 85.4 |  | 47.2 | 64.0 | 77.0
\+ Query embedding matching | 61.8 | 78.7 | 89.0 |  | 56.7 | 74.6 | 85.9
\+ Query generation | 61.7 | 79.4 | 89.6 |  | 57.1 | 75.2 | 86.7
Train student using only |  |  |  |  |  |  |
embedding matching and |  |  |  |  |  |  |
inherit doc embeddings | 63.7 | 80.3 | 90.3 |  | 57.9 | 74.6 | 85.7
\+ Query generation | 64.1 | 80.5 | 90.4 |  | 58.9 | 76.0 | 86.6

Furthermore, we also consider a in-house trained teacher (RoBERTa-base) for
MSMARCO re-ranking task. Table 11 demonstrates a similar pattern to Table 3,
providing evidence of generalizability of EmbedDistill.

Table 11: Reranking performance of various DE models on MSMARCO dev set. We
utilize a RoBERTa-base in-house trained teacher achieving MRR@10 of 33.1 and
nDCG@10 of 38.8 is used. The table shows performance of the symmetric DE
student model and asymmetric DE student models.

Method | MRR@10 |  | nDCG@10
---|---|---|---
82M | 16M |  | 82M | 16M
Train student directly | 29.7 | 26.3 |  | 35.2 | 31.4
\+ Distill from teacher | 31.6 | 28.4 |  | 37.2 | 33.5
\+ Inherit doc embeddings | 32.4 | 30.2 |  | 38.0 | 35.8
\+ Query embedding matching | 32.8 | 31.9 |  | 38.6 | 37.6
\+ Query generation | 33.0 | 32.0 |  | 38.8 | 37.7
Train student only using embedding |  |  |  |  |
matching and inherit doc embeddings | 32.7 | 31.8 |  | 38.5 | 37.5
\+ Query generation | 33.0 | 31.8 |  | 38.9 | 37.5

These result showcase that our method brings performance boost orthogonal to
how teacher was trained, whether single-staged or multi-staged.

##  Appendix G Embedding analysis

###  G.1 DE to DE distillation

Traditional score matching-based distillation might not result in transfer of
relative geometry from teacher to student. To assess this, we look at the
discrepancy between the teacher and student query embeddings for all
$q,q^{\prime}$ pairs: $\|{\tt emb}^{\rm t}_{q}-{\tt emb}^{\rm
t}_{q^{\prime}}\|-\|{\tt emb}^{\rm s}_{q}-{\tt emb}^{\rm s}_{q^{\prime}}\|$.
Note that the analysis is based on NQ, and we focus on the teacher and student
DE models based on BERT-base and DistilBERT, respectively. As evident from
Fig. 3, embedding matching loss significantly reduces this discrepancy.

###  G.2 CE to DE distillation

Figure 3: Histogram of teacher-student distance discrepancy in queries.

We qualitatively look at embeddings from CE model in Fig. 4. The embedding
${\tt emb}^{\rm t}_{q,d}$ from [CLS]-pooled CE model does not capture semantic
similarity between query and document as it is solely trained to classify
whether the query-document pair is relevant or not. In contrast, the (proxy)
query embeddings ${\tt emb}^{\rm t}_{q\leftarrow(q,d)}$ from our Dual-pooled
CE model with reconstruction loss do not degenerate and its embeddings groups
same query whether conditioned on positive or negative document together.
Furthermore, other related queries are closer than unrelated queries. Such
informative embedding space would aid distillation to a DE model via embedding
matching.

Figure 4: Illustration of geometry expressed by [CLS]-pooled CE and our Dual-
pooled CE model on 6 queries from MSMARCO and 12 passages based on pairwise
distance matrix across these 72 pairs. [CLS]-pooled CE embeddings degenerates
as all positive and negative query-document pairs almost collapse to two
points and fail to capture semantic information. In contrast, our Dual-pooled
CE model leads to much richer representation that can express semantic
information.