RouterRetriever: Exploring the Benefits of Routing over Multiple Expert Embedding Models
Hyunji Lee Luca Soldaini Arman Cohan Minjoon Seo Kyle Lo


Pointwise and uniform bounds for functions of the Laplacian on non-compact symmetric spaces
    [
    September 9, 2024
===========================================================================================



§ ABSTRACT 
Information retrieval methods often rely on a single embedding model trained on large, general-domain datasets like MSMARCO. While this approach can produce a retriever with reasonable overall performance, models trained on domain-specific data often yield better results within their respective domains. While prior work in information retrieval has tackled this through multi-task training, the topic of combining multiple domain-specific expert retrievers remains unexplored, despite its popularity in language model generation. 
In this work, we introduce , a retrieval model that leverages multiple domain-specific experts along with a routing mechanism to select the most appropriate expert for each query. It is lightweight and allows easy addition or removal of experts without additional training. Evaluation on the BEIR benchmark demonstrates that outperforms both MSMARCO-trained (+2.1 absolute nDCG@10) and multi-task trained (+3.2) models. This is achieved by employing our routing mechanism, which surpasses other routing techniques (+1.8 on average) commonly used in language modeling. 
Furthermore, the benefit generalizes well to other datasets, even in the absence of a specific expert on the dataset. 
To our knowledge, is the first work to demonstrate the advantages of using multiple domain-specific expert embedding models with effective routing over a single, general-purpose embedding model in retrieval tasks[Code in <https://github.com/amy-hyunji/RouterRetriever>].





§ INTRODUCTION

While a single embedding model trained on large-scale general-domain datasets like MSMARCO <cit.> often performs well, research shows that models trained on domain-specific datasets, even if smaller, can achieve superior results within those domains <cit.>. 
Moreover, finetuning on MSMARCO after pretraining with contrastive learning can sometimes degrade performance on specific datasets <cit.>. To improve embedding models for domain-specific datasets, previous studies have explored approaches such as data construction <cit.> and domain adaptation methods <cit.>. However, less attention has been paid to leveraging multiple expert embedding models and routing among them to select the most suitable one during inference. 

In this work, we introduce , a retrieval model that leverages multiple domain-specific experts with a routing mechanism to select the most suitable expert for each instance. For each domain, we train gates (experts), and during inference, the model determines the most relevant expert by computing the average similarity between the query and a set of pilot embeddings representing each expert, selecting the expert with the highest similarity score. 
is lightweight, as it only requires the training of parameter-efficient LoRA module <cit.> for each expert, resulting in a minimal increase in parameters. Additionally, offers significant flexibility: unlike a single model that requires retraining when domains are added or removed, simply adds or removes experts without the need for further training.

Evaluation on the BEIR benchmark <cit.> with various combinations of experts highlights the benefits of having multiple expert embedding models with a routing mechanism compared to using a single embedding model. 
When keeping the total number of training datasets constant, consisted of only domain-specific experts without an MSMARCO expert outperforms both a model trained on the same dataset in a multi-task manner and a model trained with MSMARCO.
Also, adding domain-specific experts tends to improve performance even when an expert trained on a large-scale general-domain dataset like MSMARCO is already present, suggesting that, despite the capabilities of a general-domain experts, domain-specific experts provide additional benefits, underscoring their importance.
Moreover, consistently improves performance as new experts are added, whereas multi-task training tends to show performance degradation when a certain number of domains are included. This indicates the advantage of having separate experts for each domain and using a routing mechanism to select among them. Notably, the benefits of generalize not only to datasets that have corresponding experts but also to additional datasets without specific experts.

We further explore the factors behind these performance benefits.
First, consistently shows improved performance with the addition of more experts (gates), suggesting that broader domain coverage by experts enhances retrieval accuracy. This trend holds even in an oracle setting, where the gate that maximizes performance is always selected. Notable, adding a new expert for a different domain yields greater performance gains than adding additional experts within the same domain.
Second, we observe that parametric knowledge influences embedding extraction. This observation supports the idea that training with domain-specific knowledge improves the quality of embedding extraction of the domain. 
Last, the performance difference between an instance-level oracle (which routes each instance to its best expert) and a dataset-level oracle (which routes queries to the expert with the highest average performance for the dataset) suggests that queries may benefit from a knowledge of other domains, supporting the effectiveness of our routing technique. 
Our results point to potential research opportunities in improving routing techniques among multiple expert retrievers, a direction that leads to the development of a retriever system that performs well across both general and domain-specific datasets.



§ RELATED WORKS




  
Domain Specific Retriever
There exists substantial research on retrieval models that aim to improve performance on domain-specific tasks.
One approach focuses on dataset augmentation. As domain-specific training datasets are often unavailable and can be costly to construct, researchers have developed methods that either train models in an unsupervised manner <cit.> or fine-tune models on pseudo-queries generated for domain-specific datasets <cit.>.
Another approach is developing domain-specific embeddings. A common approach is training in a multi-task manner over domain-specific datasets <cit.>. 
Recent works have aimed to improve domain-specific retrievers by developing instruction-following retrieval models <cit.>; instruction contains such domain knowledge.
Another example is <cit.> which trains a soft token for domain-specific knowledge. 
While these methods also aim to extract good representative embeddings for the input text, these methods rely on a single embedding model and produce domain-specific embeddings by additionally including domain-specific knowledge (e.g., appended as instructions) to the input.
differs from these prior methods by allowing for the employment of multiple embedding models where rather than providing the domain knowledge to the input, added to the model as parametric knowledge to produce the domain representative embeddings.



  
Routing Techniques

Various works have focused on developing domain-specific experts and routing mechanisms to improve general performance in generation tasks.
One approach simultaneously trains experts (gates) and the routing mechanism <cit.>.
Another line of work includes post-hoc techniques that do not require additional training for routing.
Some approaches use the model itself as the knowledge source by training it on domain-specific knowledge <cit.>, incorporate domain-specific knowledge in the token space <cit.>, or select the most relevant source from a sampled training dataset of each domain <cit.>. 
Routing techniques have also been investigated for improving generation quality in retrieval-augmented generation tasks; <cit.> explores routing to decide whether to utilize external knowledge and <cit.> focuses on routing to choose among different retrieval approaches.
However, there has been less emphasis on applying these techniques to information retrieval tasks. In this work, we investigate the benefits of leveraging multiple domain-specific experts and routing mechanisms in information retrieval, contrasting this approach with the traditional methods of using a single embedding model trained on a general-domain dataset or multi-task training across various domains. Additionally, we find that simply adapting routing techniques from generation tasks to information retrieval does not yield high performance, underscoring the importance of developing routing techniques tailored specifically for information retrieval.




§ ROUTER RETRIEVER






Constructing Pilot Embedding Library
 



In this section, we introduce , a retrieval model composed of a base retrieval model and multiple domain-specific experts (gates). As shown in Figure <ref>, for a given input query, 1 the most appropriate embedding is selected using a routing mechanism. Then, 2 the query embedding is generated by passing the query through the selected gate alongside the base encoder. 

In the offline time, we train the experts (gates) with domain-specific training datasets and construct a pilot embedding library. This library contains pairs of pilot embeddings for each domain along with the corresponding expert trained on that domain. Please note that this process is performed only once.
During inference (online time), when given an input query, a routing mechanism determines the appropriate expert. We calculate the similarity score between the input query embedding and the pilot embeddings in the pilot embedding library, and then choose the expert with the highest average similarity score.

We use Contriever <cit.> as the base encoder and train parameter-efficient LoRA <cit.> for each domain as the gate for that domain keeping the model lightweight. 
For example, in the case of Figure <ref>, includes a base encoder with three gates (experts): Gate A, Gate B, and Gate C, and the Expert Encoder A is composed of the base encoder with Gate A (LoRA trained on a dataset from domain A) added. This approach allows for the flexible addition or removal of domain-specific gates, enabling various gate combinations without requiring further training for the routing mechanism.




  
Experts (Gates)
For each domain D_i, where i = 1, …, T and T is the total number of domains, we train a separate expert (gate) g_i using the corresponding domain dataset. After the training step, we have a total of T different gates, 𝒢 = { g_1, g_2, …, g_T }, with each gate g_i specialized for a specific domain.



  
Pilot Embedding Library

Given a domain-specific training dataset D_i = {x_1, …, x_k} where x_j is an instance in D_i, we perform inference using all gates 𝒢 to identify which gate provides the most suitable representative embedding for each instance (line 4-7 in Alg. <ref>). For each instance x_j, we select g_max, the gate that demonstrates the highest performance, defined as g_max(x_i) = max_g_j ∈𝒢Performance(g_j, x_i). This process produces pairs (x_j, g_max) for all instances in the dataset D_i.

Next, we group these pairs by g_max, constructing T groups, one for each domain. Then for each group, we perform k-means clustering with cluster size 1 to get the pilot embedding (line 8-19 in Alg. <ref>).
In specific, with the constructed pairs (x_j, g_max), we group them by the ones that have the same g_max, Group_m, which contains list of instances x_j with same gate as the max gate. This results in T groups, one for each domain (m=1,⋯, T). 
If the Group_m is not empty, we first extract all embeddings for instances in the group with the base encoder (BaseModel).
We then apply k-means clustering <cit.> to these embeddings with a cluster size of one. The centroid of this cluster 𝐜_m is taken as the pilot embedding for the domain. This results in one pilot embedding per group, yielding a maximum of T pilot embeddings for the training dataset D_i. Each of these embeddings is associated with a different gate, representing the most suitable one for that domain. Please note that since when Group_m is empty, we do not extract pilot embedding for the empty group (cluster), thereby the number of pilot embeddings for the training dataset could be less than T. 

By repeating this process across all domain-specific training datasets D_1, …, D_T, we obtain T pilot embeddings for each gate, one from each domain-specific training dataset (repeating line 3-19 in Alg. <ref> for all training dataset D_1 ⋯ D_T). Consequently, the pilot embeddings contains a maximum of T^2 pilot embeddings, with each of the T domain-specific training datasets contributing up to T pilot embeddings. 


For example, consider a scenario with three experts, each trained on one of the following datasets: SciFact, FiQA-2018, and HotpotQA. 
To construct the pilot embedding library, we first perform inference on all training instances used to train the SciFact expert across all three experts to determine which expert produces the most suitable embedding (line 4-7 in Alg. <ref>).
The chosen gate can be any of the three experts.
Next, we group all training instances from the SciFact dataset according to the expert that achieved the highest performance for each instance, resulting in up to three groups: instances where the SciFact expert, FIQA-2018 expert, or HotpotQA expert performed best. For each group, we apply k-means clustering with k=1 to compute the centroid, which serves as the pilot embedding for that group. This pilot embedding is added to the pilot embedding library, with the corresponding expert as the key. For example, if the centroid is extracted from the group where the HotpotQA expert performed best, the HotpotQA expert is considered the most suitable expert, even if the instances are from the SciFact dataset. This process adds three pilot embeddings to the library, one for each expert (lines 8-19 in Alg. <ref>).
We repeat this process for all domains (the example on top is for SciFact and we repeat the process for FIQA-2018 and HotpotQA), ultimately creating a total of a maximum of nine pilot embeddings in the library, with three pilot embeddings associated with each expert.




  
Routing Mechanism

When given an input query, we calculate the similarity between the query embedding extracted from the base encoder and the T^2 pilot embeddings in the pilot embedding library. We then average the similarity scores for T pilot embeddings associated with the same gate, resulting in a mean similarity score for each gate. The gate corresponding to the highest mean similarity score is selected as the most suitable embedding model.



§ EXPERIMENTAL SETUP



  
Baselines
We compare the performance of with when training on the same dataset in a multi-task manner (Multi-Task) and training on a large-scale general-domain dataset MSMARCO (MSMARCO-Trained). Additionally, following previous works <cit.>, we evaluate performance using two oracle settings: Best Individual and Oracle. The Best Individual setting is a dataset-level oracle that routes all queries in a dataset to the expert with the highest average performance for that dataset, while the Oracle setting is an instance-level oracle that routes each individual instance to its best-performing expert. 

We also conduct experiments with various other routing techniques commonly used in language modeling tasks; ExpertClassifierRouter <cit.>, ClassificationHeadRouter <cit.>, and DatasetRouter <cit.>. 
ExpertClassifierRouter employs a binary classifier for each gate to calculate the probability of that gate being selected. The gate with the highest probability is chosen for the final selection. 
ClassificationHeadRouter uses a single classifier layer to determine the appropriate expert for each instance. 
DatasetRouter is the most similar to , as it selects the gate by retrieving the instance with the highest similarity score. However, there are two key differences: uses the predicted label, whereas DatasetRouter relies on the original dataset label. Also, incorporates a clustering step to group instances, while DatasetRouter randomly samples 100 instances from the training dataset.
Further details of the baselines and training methods for each are provided in the supplementary materials.



  
Dataset
We used datasets in BEIR benchmark <cit.>, which includes 14 datasets across 6 domains: Bio-Medical, Wikipedia, Finance, Misc., Quora, and Scientific[Details of datasets and domains in supplementary].
To train domain-specific gates, we utilize the training sets provided by BEIR. Due to the limited number of datasets with available training sets, we also employed generated queries provided by BEIR[<https://huggingface.co/BeIR>].
The models were evaluated using the test sets. We categorize the datasets in the Misc. domain as separate general domains, Wikipedia as a general domain, and Bio-Medical, Finance, Quora, and Scientific as domain-specific datasets based on how broadly each instance is distributed. As illustrated in Figure <ref>, which shows the embeddings extracted from the pre-trained Contriever model (our base model), datasets in the Misc. domain are often widely dispersed even within the same domain. Although the Wikipedia datasets are generally close to others within the same domain, they also exhibit a broad spread. In contrast, datasets from the Bio-Medical, Finance, Quora, and Scientific domains tend to be more compact and closely clustered.







  
Hyperparameters

We use the pre-trained Contriever <cit.> as our base encoder and train gates (LoRA) according to the settings in <cit.>, with a rank of 8, an alpha of 32 per gate, thereby training approximately 0.5% of the parameters (about 1M parameters) per gate. 
For training, we adopt the few-shot hyperparameters from <cit.>: a learning rate of 1e-4, a batch size of 256 with in-batch negatives, and a maximum of 500 epochs with early stopping. 
Gates are applied only to the query encoder, keeping the context encoder frozen, as our focus is on understanding the impact of routing by query instances, thereby eliminating the influence of routing on the context encoder. We also include the results of applying gates to the context encoder in the supplementary materials.




§ EXPERIMENTAL RESULTS & DISCUSSIONS





 §.§ Overall Performance


Table <ref> shows the performance of compared to baseline models using seven domain-specific gates (AR, MS, HO, NF, SF, QU, and FI)[All gates, except for MSMARCO, are selected to have the smallest training dataset from each domain to ensure that the total number of training dataset is equal to that of MSMARCO when excluding the MS gate.].
outperforms the MSMARCO-trained model, indicating that even with a large-scale general-domain training dataset, incorporating additional domain-specific gates further enhances performance. 
Also, when keeping the training dataset the same when comparing to the Multi-Task model, which is trained with the same training datasets, consistently shows higher performance. 
Moreover, (w/o MS expert), which excludes the MSMARCO gate but maintains the same total number of training datasets as the MSMARCO-trained model, still achieves superior performance. These results underscore the importance of having separate embedding models (gates) for each domain and dynamically selecting the most appropriate gate for each query rather than relying on a single model to handle multiple domains. 
For additional results with different combinations of experts, please refer to the supplementary material.





 §.§ Affect of Dataset Size when Training Experts


Figure <ref> shows the relationship between performance (y-axis) and the number of training samples (x-axis) across various datasets. 
For in-domain evaluation datasets, performance generally improves as the number of training samples increases. However, in out-of-domain evaluation datasets, simply increasing the number of training samples does not necessarily lead to better performance.
When the same number of training samples is used, for in-domain cases, experts consistently achieve the highest performance across all evaluation datasets. 
Interestingly, for out-of-domain cases, experts perform better when trained on general domains (e.g., Arguana and MSMARCO) compared to domain-specific experts (e.g., SciFact and NFcorpus). We attribute this to the broader coverage and stability of general-domain datasets, as illustrated in Figure <ref>. 
These results suggest that while a larger training dataset is generally beneficial for expert in-domain performance, the coverage of the training dataset has a more significant impact on out-of-domain performance.








 §.§ Impact of Number of Gates

Figure <ref> shows that adding gates (x-axis) consistently improves the performance of (y-axis). Notably, outperforms the MSMARCO-trained model even with just three gates, indicating that despite not having as diverse or large a training dataset as MSMARCO, the advantage of having multiple embedding models and the ability to select the most suitable one leads to better performance. also shows a small gap with the Best Individual performance which is the in-domain performance for each expert (Oracle performance for dataset-wise). 
The performance in multi-task training tends to fluctuate as the number of domains (gates) increases. We hypothesize that with a large number of domains, the model struggles to find the optimal embedding for general cases due to the high variance across training datasets.

Figure <ref> illustrates the performance when we use 7 gates and increase the number of experts that the model can choose from, selecting the one with the maximum performance for each instance. Oracle represents the performance when the model can route through all 7 gates and choose the best-performing one instance-wise. As the number of gates increases, performance consistently improves. Notably, the rate of improvement is higher when adding gates initially, and as the number of gates grows, the rate of increase diminishes, regardless of the order in which experts are added. We believe this tendency arises because the routing technique tends to be more distracted as more gates are added. Nonetheless, the consistent improvement with additional gates highlights the potential for further enhancement with better routing techniques, emphasizing the importance of investigating these techniques across various expert retrievers. We randomly varied the order and combination of gates in the figure but observed that the trend remained consistent. Details are provided in the supplementary materials.









 §.§ Maximum Performing Gate Rates and Analysis of Gate Selection

Figure <ref> shows the rate at which each gate achieves the highest performance across different evaluation datasets. For general-domain datasets (AR, MS, HO), the best-performing gate is often distributed across multiple experts. However, for domain-specific datasets (SF, NF, FI, QU), the best performance is typically achieved by the gate trained specifically on that domain. 
This indicates that while gates generally perform well on general-domain datasets, having a domain-specific expert model is essential for achieving high performance in specialized areas.

Figure <ref> shows the rate at which gates are selected by our routing technique for each evaluation dataset. The MS gate is often chosen, likely because, as shown in Figure <ref>, MSMARCO instances are broadly distributed, leading to more generalized pilot embeddings. Since the MS gate performs well across all datasets (as seen in Figure <ref>), this selection seems reasonable. For domain-specific datasets like SF and FI, the routing strongly favors the gate trained on the respective dataset, which we assume is likely because these datasets cluster closely together in Figure <ref>. We add a detailed error analysis of the routing technique in the supplementary.




 §.§ Impact of Expert Combinations

We experiment with various combinations of experts to assess their impact on performance. Our findings suggest that broad coverage across domains is critical. Within a single domain, the specific expert chosen does not significantly affect performance as long as it is trained with a sufficient amount of training dataset. Adding an expert from a new domain tends to significantly improve performance while adding additional experts to a domain that already has an expert doesn't yield as much improvement.

Adding domain-specific gates like SciDocs in the Science domain or TREC-COVID in the Bio-Medical domain improves performance for those datasets, with SciDocs increasing from 44.3 to 56.2 and TREC-COVID from 15.1 to 16.1. However, the overall average performance across all datasets remains relatively stable. We also observed that performance tends to improve when using experts trained on larger datasets, consistent with our earlier observation in Figure <ref> that expert performance generally increases with the size of the training dataset. Detailed results are provided in the supplementary materials.




 §.§ Various Routing Techniques


We experiment with various routing techniques commonly used in language modeling and compared them with our proposed routing mechanism. Results in Table <ref> show that the routing technique used in consistently achieves the highest performance. 
In fact, ClassificationHeadRouter and ExpertClassifierRouter approaches tend to underperform compared to when using a single expert trained solely on MSMARCO (MSMARCO-Trained). DatasetRouter, which is the closest to , tends to show higher performance than MSMARCO-Trained but also consistently shows lower performance than . These results suggest that these routing techniques are not well-suited for information retrieval and may even degrade performance compared to using a single expert. We hypothesize that the differences in the effectiveness of routing techniques between language modeling and information retrieval can be explained from two perspectives. First, in language modeling, experts are often trained to handle distinct tasks, making them easier to differentiate. In contrast, information retrieval involves domain classification, which may be more challenging. Second, in language modeling, routing decisions are often made at the token level, which allows for greater flexibility and reduces the impact of any single choice. However, in information retrieval, where a single representative embedding is required, the choice of expert is made only once per instance, making the process more vulnerable to the routing technique used, and thus requiring greater precision.







 §.§ General Performance of over various datasets

Table <ref> demonstrates that consistently outperforms other baselines that rely on a single general-purpose embedding model[Detailed numbers of the table are in Supplementary.]. This is evident not only in datasets that have their experts (w/ Experts) but also across various other datasets that do not have their experts (w/o Experts).
These findings suggest that the benefits of having multiple experts and routing across them extend well beyond the datasets for which specific experts were trained. 



 §.§ Where does the benefit come from?


We hypothesize that the benefit of having domain-specific gates comes from the model's tendency to be influenced by its parametric knowledge; models trained on domain-specific datasets are likely to have domain-specific knowledge embedded in their parametric space, enabling them to produce more meaningful embeddings related to those domains.
To test the hypothesis, we conduct experiments with the dataset from  <cit.>, which contains both original NQ <cit.> contexts that align with the retriever's parametric knowledge and conflicting contexts for each instance. We experiment with RepLlama <cit.> and E5-Mistral <cit.>[We used these two models not Contriever to ensure that the NQ contexts from <cit.> align with their parametric knowledge] and found that the retrievers surprisingly for all case prefer contexts that align with their parametric knowledge; they consistently retrieve the original NQ contexts over conflicting contexts[We exclude contexts containing an extensive length of contexts (context with table information) as they tend to introduce bias <cit.>]. 
This finding supports our hypothesis that embedding models are influenced by parametric knowledge when extracting embedding thus their knowledge of domain-specific datasets are better able to extract meaningful embeddings relevant to their domain knowledge. Further details are in the supplementary.



 §.§ Efficiency

achieves high efficiency by using parameter-efficient LoRA gates, which account for only about 0.5% of the parameters per gate. This makes the addition of new gates relatively insignificant in terms of parameter count.
In terms of training, it uses the same amount of training data as in a multi-task approach. However, unlike multi-task training, which requires retraining the entire model when adding, removing, or changing domains, allows for these modifications without additional training, as our routing technique is training-free.
However, during inference, computing the query embedding involves two forward passes: the first to identify the appropriate gate (routing), and the second to generate the final query embedding. Improving the computation efficiency of this routing technique is a direction for future work.




§ CONCLUSION

In this paper, we present , a retrieval model that integrates multiple domain-specific experts with a routing mechanism to extract the most suitable embedding for each query. This approach is both lightweight and flexible, allowing for the addition or removal of experts without additional training. Our experiments demonstrate that it consistently outperforms single embedding models, showcasing the advantages of integrating domain-specific experts. Additionally, it surpasses various widely used routing techniques in language modeling, emphasizing the significance of effective routing for information retrieval tasks. These results highlight the crucial role of domain-specific experts in improving retrieval performance and suggest that combining them with efficient routing techniques can significantly enhance results, potentially approaching oracle performance.



§ ACKNOWLEDGMENTS

We thank Nandan Thakur, Orion Weller, Jiyeon Kim, and Hanseok Oh for helpful discussions and constructive feedback.










§ EXPERIMENTAL SETUP



 §.§ Baselines



  
MSMARCO
This baseline uses a single MSMARCO gate, which is trained on a large-scale, general-domain dataset without any routing techniques applied.


  
Multi-Task
In this approach, we train a single embedding model on all datasets simultaneously in a multi-task manner. We keep the number of training datasets for each label the same, keeping to the one with the minimum value by sampling. 


  
Best Individual
This represents the oracle performance when selecting the single best-performing gate for each dataset. For example, if the SciFact gate shows the highest overall performance on the SciDocs evaluation dataset compared to other gates, the performance of the SciFact gate is recorded as the best individual performance for SciDocs.


  
Oracle
This is the oracle performance when selecting the best-performing gate for each individual instance. For example, within the SciDocs dataset, certain instances might achieve the highest performance with the SciFact gate, while others might perform better with the MSMARCO gate. This baseline measures the performance when, for each instance, the gate that yields the best result is selected.


  
ExpertClassifierRouter
This routing technique, inspired from <cit.>, uses a binary classifier for each gate. For each instance, the classifier calculates the probability of selecting or not selecting a specific gate. The gate with the highest probability of being selected is chosen.

To construct the training dataset, we use the predicted label (g_max) from the Pilot Embedding Library. For each (x_i, g_max) pair, we randomly sample instances where the maximum gate differs, which are used to train the "not choosing the gate" label. The dataset is balanced across labels, with the following number of training instances for each dataset: AR (16,108), FI (1070), SF (1,414), NF (892), HO (4,618), QU (4,326), and MS (4,252). Please note that the training datasets only consist of instances where only a single gate shows maximum performance.
We then train a binary classifier for each gate to predict whether an instance is likely to achieve the highest performance through that gate.




  
ClassificationHeadRouter
This routing technique, inspired from <cit.>, uses a classification head where the number of labels corresponds to the number of gates. The gate with the highest predicted probability is selected as the one likely to yield the best performance. To ensure balance, we equalize the number of training instances for each label, matching the dataset with the fewest instances (NFcorpus with 892 instances, other numbers in ExpertClassifierRouter paragraph). AS a result, the total number of training instances is 6,244. 




  
DatasetRouter
This routing technique, inspired from <cit.>, is the closest baseline to . It samples 100 training instances from each dataset and when given a query, it retrieves the most relevant instances from these samples. The gate trained on the dataset from which the sample originated is then used. 

The key differences between DatasetRouter and are as follows. (1) uses the predicted label to map an instance to a gate, while DatasetRouter relies on the original dataset label. For example, if a training instance from MSMARCO performs best with the sciFact gate, will select the Scifact gate for a similar query, whereas DatasetRouter will select the MSMARCO gate. (2) incorporates a clustering step, grouping similar instances together and using centroid embeddings, rather than treating each instance individually.



 §.§ Datasets





  
Stats of Training Dataset

Table <ref> presents the statistics and details of the datasets in the BEIR benchmark, which we used for training and evaluation. We sampled datasets from Quora to ensure that the number of training instances for AR, HO, NF, SF, FI, and QU matches that of MS.







  
Examples of Oracle
Table <ref> shows examples of questions where a gate from a different dataset outperforms the gate trained on the dataset to which the question belongs. We observe that questions related to biology often achieve higher performance with the NFCorpus gate, while those involving scientific knowledge tend to favor the SciFact gate, and questions requiring arguments perform better with the Arguana gate. This pattern suggests that, even within a single dataset, some instances may be more closely aligned with other datasets, likely because the datasets were not labeled or constructed to avoid overlap with existing datasets.




 §.§ Hyperparameters


We trained the Contriever model <cit.> using an asymmetric architecture, where the query encoder encodes the query and the context encoder encodes the context. In our experiments, we fine-tuned only the LoRA (Low-Rank Adaptation) parameters of the query encoder, training approximately 1 million parameters per gate (which accounts for 0.5% of the total model parameters).
For evaluation, we used the NDCG@10 metric, consistent with previous works <cit.>, which measures the ranking quality of the top 10 retrieved documents. All results were calculated using the official BEIR evaluation code.
The experiments were conducted on 8 or fewer A6000 GPUs (each with 40GB of memory). We utilized checkpoints from all pretrained models available on Huggingface[<https://huggingface.co/facebook/contriever>]. The experiments were performed over various combinations of gates, with all random seeds set to 10.





  
When unfreeze context encoder

In our main experiments, we focus on scenarios where the context encoder is frozen, and only the LoRA of the query encoder is trainable to isolate the impact of routing on the query encoder alone. However, we observe that the overall performance trend remains similar even when the context encoder is not frozen, with the unfrozen models generally achieving higher performance. Table <ref> presents the results when the context encoder is frozen. In these experiments, consistently outperforms the MSMARCO-trained model and the Multi-Task model.



§ EXPERIMENTAL RESULTS & DISCUSSIONS






 §.§ Performance of each gates



To analyze the performance trends of each gate, we evaluate them individually without applying any routing techniques in Table <ref>. 
The performance generally shows the highest when the evaluation dataset matches the training dataset of the gate. Additionally, the performance gap between matching and non-matching datasets is larger for domain-specific datasets (NF, TR, SD, SF, QU, FI). In contrast, gates trained on general-domain datasets (AR, MS, HO) tend to perform well across a broader range of datasets. 





 §.§ Affect of Number of Pilot Embeddings


We experiment with how the number of pilot embeddings affects performance. In Figure <ref>, we observe that performance tends to degrade as the number of pilot embeddings increases. We hypothesize that this decline is due to the increased number of pilot embeddings becoming distracting, leading to less effective routing decisions.




 §.§ Impact of Number of Gates

To investigate the impact of number of gates, we randomly shuffle the gate order and experiment how adding gates tend to affect performance.
The order of gates added in Figure 4 and Figure 5 is AR, FI, SF, NF, HO, QU, and MS.
We tried various other combinations and could see that the findings are stabilized (Figure <ref>): (1) performance tend to increase with more gates added and (2) the improvement rate tend to be higher when adding gates initially and as the number of gates grows, the rate of increase diminishes.




 §.§ Detailed numbers by gates

In this section, we show detailed number of performance with different combinations of gates. 
Table <ref> shows performance with AR, NF, SF, FI as gates. 
Table <ref> shows performance with AR, HO, NF, SF, FI as gates.
Table <ref> shows performance with AR, HO, NF, SF, QU, FI as gates.
Table <ref> shows performance with AR, MS, HO, NF, SF, QU, FI as gates.
Figure 4 shows only with three gates, outperforms the MSMARCO-trained ones thereby in all results, we can see that outperforms the MSMARCO-trained ones and multi-task baselines.





























 §.§ Routing Mechanism Error Analysis


Figure 7 illustrates the rate at which each router selects a gate, while Figure 6 shows the rate at which each gate tends to deliver high performance for the dataset. The discrepancy between these two heatmaps highlights the gap between and the oracle performance.
For Arguana, the maximum gate distribution is evenly spread, and the routing tends to follow this distribution closely.  
For Quora, while the maximum gate rate is high overall, the routing often favors the HotpotQA gate in many cases.  
For MSMARCO, the gate trained on MSMARCO generally shows high performance, but the routing technique tends to distribute selections across different gates.  
For HotpotQA, selecting the HotpotQA gate most frequently results in the highest performance, with MSMARCO being the next best option. The routing technique tends to reflect this pattern.  
For SciFact, choosing the SciFact gate is crucial in both cases.  
For NFCorpus, selecting the NFCorpus gate is important, yet the routing technique often opts for the Arguana gate in many instances.  
For FiQA-2018, the best performance is achieved by selecting the FiQA-2018 gate, and the routing technique successfully identifies this gate most of the time.

We specifically investigated why NFCorpus often fails to select the NFCorpus gate and instead tends to choose the Arguana gate. Upon examining the representative embeddings for Arguana, we found that many of them are confused with Arguana embeddings that were extracted from the NFCorpus dataset. These instances originally belong to NFCorpus but show the highest performance with the Arguana gate, leading to their labeling as Arguana. This suggests that instead of completely removing information about the original dataset, incorporating a weighting factor between the two could further improve performance.



 §.§ Generalization to other datasets


We observe that demonstrates stable performance not only on datasets with corresponding gates but also on those without them. The performance with different numbers of gates is shown in the following tables: Table <ref> (4 gates), Table <ref> (5 gates), Table <ref> (6 gates), Table <ref> (7 gates), and Tables <ref> and <ref> (8 gates).

When using a similar total number of training datasets (Table <ref>), and the MSMARCO-trained model exhibit comparable generalization performance (both at 31.6). However, achieves higher performance on datasets that have corresponding gates (47.5 for MSMARCO-only vs. 49.3 for ). As more gates are added, both generalization ability and performance on datasets with corresponding gates tend to improve (Figure <ref>).




 §.§ Where does the benefit come from?


We hypothesize that the advantage of using multiple expert embedding models with routing, rather than a single embedding model, stems from the influence of the training dataset on a model's parametric knowledge, which in turn affects the extracted embeddings. To test this hypothesis, we experimented to determine whether a model tends to prefer contexts that align with its parametric knowledge over those that conflict with it.

We used a dataset released by <cit.>, which includes instances where each context either aligns with or conflicts with the model's parametric knowledge. For each instance with 5-6 contexts, we evaluated which context the model chose based on the highest similarity. Interestingly, in all instances[We excluded contexts containing extensive length of contexts (tables), as they tend to introduce bias <cit.>.], the models consistently preferred the context that aligned with their parametric knowledge. This suggests that the internal knowledge of the model influences how embeddings are extracted, and that having domain knowledge embedded in the model's parameters enhances performance.

