Duplicate from nomic-ai/colnomic-embed-multimodal-7b

Co-authored-by: Zach Nussbaum <[email protected]>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,176 @@
+---
+base_model:
+- Qwen/Qwen2.5-VL-7B-Instruct
+library_name: peft
+datasets:
+- llamaindex/vdr-multilingual-train
+- nomic-ai/colpali_train_set_split_by_source
+language:
+- en
+- it
+- fr
+- de
+- es
+pipeline_tag: visual-document-retrieval
+tags:
+- vidore
+- colpali
+- multimodal_embedding
+- multilingual_embedding
+- Text-to-Visual Document (T→VD) retrieval
+license: apache-2.0
+---
+
+# ColNomic Embed Multimodal 7B: State-of-the-Art Visual Document Retrieval
+
+`colnomic-embed-multimodal-7b` is a multi-vector state-of-the-art multimodal embedding model that excels at visual document retrieval tasks:
+
+- **High Performance**: Achieves 62.7 NDCG@5 on Vidore-v2, outperforming all other models
+- **Unified Text-Image Encoding**: Directly encodes interleaved text and images without complex preprocessing
+- **Advanced Architecture**: 7B parameter multimodal embedding model
+- **Fully Open-Source**: Model weights, training data, and code available
+
+## Performance
+
+| Model | Avg. | ESG Restaurant Human | Econ Macro Multi. | AXA Multi. | MIT Bio | ESG Restaurant Synth. | ESG Restaurant Synth. Multi. | MIT Bio Multi. | AXA | Econ. Macro |
+|-------|------|----------------------|-------------------|------------|---------|----------------------|----------------------------|---------------|-----|------------|
+| **ColNomic Embed Multimodal 7B** | 62.7 | 73.9 | 54.7 | 61.3 | 66.1 | 57.3 | 56.7 | 64.2 | 68.3 | 61.6 |
+| [ColNomic Embed Multimodal 3B](https://huggingface.co/nomic-ai/colnomic-embed-multimodal-3b) | 61.2 | 65.8 | 55.4 | 61.0 | 63.5 | 56.6 | 57.2 | 62.5 | 68.8 | 60.2 |
+| T-Systems ColQwen2.5-3B | 59.9 | 72.1 | 51.2 | 60.0 | 65.3 | 51.7 | 53.3 | 61.7 | 69.3 | 54.8 |
+| [Nomic Embed Multimodal 7B](https://huggingface.co/nomic-ai/nomic-embed-multimodal-7b) | 59.7 | 65.7 | 57.7 | 59.3 | 64.0 | 49.2 | 51.9 | 61.2 | 66.3 | 63.1 |
+| GME Qwen2 7B | 59.0 | 65.8 | 56.2 | 55.4 | 64.0 | 54.3 | 56.7 | 55.1 | 60.7 | 62.9 |
+| [Nomic Embed Multimodal 3B](https://huggingface.co/nomic-ai/nomic-embed-multimodal-3b) | 58.8 | 59.8 | 57.5 | 58.8 | 62.5 | 49.4 | 49.4 | 58.6 | 69.6 | 63.5 |
+| Llama Index vdr-2b-multi-v1 | 58.4 | 63.1 | 52.8 | 61.0 | 60.6 | 50.3 | 51.2 | 56.9 | 68.8 | 61.2 |
+| Voyage Multimodal 3 | 55.0 | 56.1 | 55.0 | 59.5 | 56.4 | 47.2 | 46.2 | 51.5 | 64.1 | 58.8 |
+
+## Getting Started
+
+To use `colnomic-embed-multimodal-7b`, please install `colpali` from source
+
+```bash
+pip install git+https://github.com/illuin-tech/colpali.git
+```
+
+
+```python
+import torch
+from PIL import Image
+from transformers.utils.import_utils import is_flash_attn_2_available
+
+from colpali_engine.models import ColQwen2_5, ColQwen2_5_Processor
+
+model_name = "nomic-ai/colnomic-embed-multimodal-7b"
+
+model = ColQwen2_5.from_pretrained(
+    model_name,
+    torch_dtype=torch.bfloat16,
+    device_map="cuda:0",  # or "mps" if on Apple Silicon
+    attn_implementation="flash_attention_2" if is_flash_attn_2_available() else None,
+).eval()
+
+processor = ColQwen2_5_Processor.from_pretrained(model_name)
+
+# Your inputs
+images = [
+    Image.new("RGB", (128, 128), color="white"),
+    Image.new("RGB", (64, 32), color="black"),
+]
+queries = [
+    "What is the organizational structure for our R&D department?",
+    "Can you provide a breakdown of last year’s financial performance?",
+]
+
+# Process the inputs
+batch_images = processor.process_images(images).to(model.device)
+batch_queries = processor.process_queries(queries).to(model.device)
+
+# Forward pass
+with torch.no_grad():
+    image_embeddings = model(**batch_images)
+    query_embeddings = model(**batch_queries)
+
+scores = processor.score_multi_vector(query_embeddings, image_embeddings)
+```
+
+## Model Architecture
+
+- **Total Parameters**: 7B
+- **Training Approach**: Fine-tuned from Qwen2.5-VL 7B Instruct
+- **Architecture Type**: Vision-Language Model with unified text and image input processing
+- **Key Innovations**:
+  - Same-source sampling to create harder in-batch negatives
+  - Multi-vector output option for enhanced performance
+
+## Integration with RAG Workflows
+
+Nomic Embed Multimodal 7B seamlessly integrates with Retrieval Augmented Generation (RAG) workflows:
+
+1. **Direct Document Embedding**: Skip OCR and complex processing by directly embedding document page images
+2. **Faster Processing**: Eliminate preprocessing steps for quicker indexing
+3. **More Complete Information**: Capture both textual and visual cues in a single embedding
+4. **Simple Implementation**: Use the same API for both text and images
+
+## Recommended Use Cases
+
+The model excels at handling real-world document retrieval scenarios that challenge traditional text-only systems:
+
+- **Research Papers**: Capture equations, diagrams, and tables
+- **Technical Documentation**: Encode code blocks, flowcharts, and screenshots
+- **Product Catalogs**: Represent images, specifications, and pricing tables
+- **Financial Reports**: Embed charts, graphs, and numerical data
+- **Visually Rich Content**: Where layout and visual information are important
+- **Multilingual Documents**: Where visual context provides important cues
+
+## Training Details
+
+ColNomic Embed Multimodal 7B was developed through several key innovations:
+
+1. **Sampling From the Same Source**: Forcing sampling from the same dataset source creates harder in-batch negatives, preventing the model from learning dataset artifacts.
+
+2. **Multi-Vector Configuration**: Providing a multi-vector variant that achieves higher performance than the dense variant.
+
+## Limitations
+
+- Performance may vary when processing documents with unconventional layouts or unusual visual elements
+- While it handles multiple languages, performance is strongest on English content
+- Processing very large or complex documents may require dividing them into smaller chunks
+- Performance on documents with handwriting or heavily stylized fonts may be reduced
+
+## Join the Nomic Community
+
+- Nomic Embed Ecosystem: [https://www.nomic.ai/embed](https://www.nomic.ai/embed)
+- Website: [https://nomic.ai](https://nomic.ai)
+- Twitter: [https://twitter.com/nomic_ai](https://twitter.com/nomic_ai)
+- Discord: [https://discord.gg/myY5YDR8z8](https://discord.gg/myY5YDR8z8)
+
+## Citation
+
+If you find this model useful in your research or applications, please consider citing:
+
+```bibtex
+@misc{faysse2024colpaliefficientdocumentretrieval,
+  title={ColPali: Efficient Document Retrieval with Vision Language Models}, 
+  author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
+  year={2024},
+  eprint={2407.01449},
+  archivePrefix={arXiv},
+  primaryClass={cs.IR},
+  url={https://arxiv.org/abs/2407.01449}, 
+}
+@misc{ma2024unifyingmultimodalretrievaldocument,
+      title={Unifying Multimodal Retrieval via Document Screenshot Embedding}, 
+      author={Xueguang Ma and Sheng-Chieh Lin and Minghan Li and Wenhu Chen and Jimmy Lin},
+      year={2024},
+      eprint={2406.11251},
+      archivePrefix={arXiv},
+      primaryClass={cs.IR},
+      url={https://arxiv.org/abs/2406.11251}, 
+}
+@misc{nomicembedmultimodal2025,
+  title={Nomic Embed Multimodal: Interleaved Text, Image, and Screenshots for Visual Document Retrieval},
+  author={Nomic Team},
+  year={2025},
+  publisher={Nomic AI},
+  url={https://nomic.ai/blog/posts/nomic-embed-multimodal},
+}
+```

adapter_config.json

@@ -0,0 +1,29 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "nomic-ai/colqwen2.5-7B-base",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": "gaussian",
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "(.*(model).*(down_proj|gate_proj|up_proj|k_proj|q_proj|v_proj|o_proj).*$|.*(custom_text_proj).*$)",
+  "task_type": "FEATURE_EXTRACTION",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa6cba28368342ac6b235f49fcc3e0a5bd891d56df490c12bcbc6dc52c8377f3
+size 323489536

added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
+}

git_hash.txt

@@ -0,0 +1 @@
+d3f780b24ba870601a046b4f35d28d827739446c

preprocessor_config.json

@@ -0,0 +1,29 @@
+{
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "Qwen2VLImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "max_pixels": 602112,
+  "merge_size": 2,
+  "min_pixels": 3136,
+  "patch_size": 14,
+  "processor_class": "ColQwen2_5_Processor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "longest_edge": 12845056,
+    "shortest_edge": 3136
+  },
+  "temporal_patch_size": 2
+}

processor_config.json

@@ -0,0 +1,4 @@
+{
+  "max_num_visual_tokens": 1280,
+  "processor_class": "ColQwen2_5_Processor"
+}

results.json

@@ -0,0 +1,1104 @@
+{
+   "vidore/arxivqa_test_subsampled":{
+      "ndcg_at_1":0.83,
+      "ndcg_at_3":0.87533,
+      "ndcg_at_5":0.88678,
+      "ndcg_at_10":0.89447,
+      "ndcg_at_20":0.89988,
+      "ndcg_at_50":0.90114,
+      "ndcg_at_100":0.90247,
+      "map_at_1":0.83,
+      "map_at_3":0.86467,
+      "map_at_5":0.87097,
+      "map_at_10":0.87409,
+      "map_at_20":0.87551,
+      "map_at_50":0.87574,
+      "map_at_100":0.87587,
+      "recall_at_1":0.83,
+      "recall_at_3":0.906,
+      "recall_at_5":0.934,
+      "recall_at_10":0.958,
+      "recall_at_20":0.98,
+      "recall_at_50":0.986,
+      "recall_at_100":0.994,
+      "precision_at_1":0.83,
+      "precision_at_3":0.302,
+      "precision_at_5":0.1868,
+      "precision_at_10":0.0958,
+      "precision_at_20":0.049,
+      "precision_at_50":0.01972,
+      "precision_at_100":0.00994,
+      "mrr_at_1":0.832,
+      "mrr_at_3":0.8663333333333333,
+      "mrr_at_5":0.8716333333333333,
+      "mrr_at_10":0.8752095238095239,
+      "mrr_at_20":0.8764481089945486,
+      "mrr_at_50":0.8769236116400513,
+      "mrr_at_100":0.8770087474728994,
+      "naucs_at_1_max":0.7799094444766933,
+      "naucs_at_1_std":-0.08580135833284933,
+      "naucs_at_1_diff1":0.9595692807801705,
+      "naucs_at_3_max":0.7955579394878525,
+      "naucs_at_3_std":-0.11155809841667234,
+      "naucs_at_3_diff1":0.9224725351133368,
+      "naucs_at_5_max":0.7705061822708856,
+      "naucs_at_5_std":-0.25026172084995735,
+      "naucs_at_5_diff1":0.9217666864725705,
+      "naucs_at_10_max":0.7739540260548654,
+      "naucs_at_10_std":-0.21324085189631833,
+      "naucs_at_10_diff1":0.9221910986616823,
+      "naucs_at_20_max":0.7708216619981358,
+      "naucs_at_20_std":-0.0996732026143794,
+      "naucs_at_20_diff1":0.8905228758169942,
+      "naucs_at_50_max":0.816593303988261,
+      "naucs_at_50_std":-0.19987995198079136,
+      "naucs_at_50_diff1":0.8832866479925231,
+      "naucs_at_100_max":0.9564270152505304,
+      "naucs_at_100_std":0.04310613134141329,
+      "naucs_at_100_diff1":0.9564270152505304
+   },
+   "vidore/docvqa_test_subsampled":{
+      "ndcg_at_1":0.5388,
+      "ndcg_at_3":0.59542,
+      "ndcg_at_5":0.61269,
+      "ndcg_at_10":0.62919,
+      "ndcg_at_20":0.64259,
+      "ndcg_at_50":0.64929,
+      "ndcg_at_100":0.65683,
+      "map_at_1":0.5388,
+      "map_at_3":0.5813,
+      "map_at_5":0.59084,
+      "map_at_10":0.59764,
+      "map_at_20":0.6013,
+      "map_at_50":0.60241,
+      "map_at_100":0.60307,
+      "recall_at_1":0.5388,
+      "recall_at_3":0.63636,
+      "recall_at_5":0.67849,
+      "recall_at_10":0.72949,
+      "recall_at_20":0.78271,
+      "recall_at_50":0.81596,
+      "recall_at_100":0.86253,
+      "precision_at_1":0.5388,
+      "precision_at_3":0.21212,
+      "precision_at_5":0.1357,
+      "precision_at_10":0.07295,
+      "precision_at_20":0.03914,
+      "precision_at_50":0.01632,
+      "precision_at_100":0.00863,
+      "mrr_at_1":0.5432372505543237,
+      "mrr_at_3":0.5824094604582412,
+      "mrr_at_5":0.5903917220990392,
+      "mrr_at_10":0.5980097138633723,
+      "mrr_at_20":0.601820515422298,
+      "mrr_at_50":0.6029928998761527,
+      "mrr_at_100":0.6035667141572244,
+      "naucs_at_1_max":0.8430735352775856,
+      "naucs_at_1_std":0.5939128313043085,
+      "naucs_at_1_diff1":0.9018027917306838,
+      "naucs_at_3_max":0.8808230124613547,
+      "naucs_at_3_std":0.712681445860455,
+      "naucs_at_3_diff1":0.8646220134947054,
+      "naucs_at_5_max":0.8912698205476521,
+      "naucs_at_5_std":0.7525159249929689,
+      "naucs_at_5_diff1":0.8414611325660849,
+      "naucs_at_10_max":0.8852509111743405,
+      "naucs_at_10_std":0.817204422380503,
+      "naucs_at_10_diff1":0.8264628195305581,
+      "naucs_at_20_max":0.8512017211379178,
+      "naucs_at_20_std":0.8215729110215341,
+      "naucs_at_20_diff1":0.7992889048812711,
+      "naucs_at_50_max":0.8402551716801772,
+      "naucs_at_50_std":0.8733758748271155,
+      "naucs_at_50_diff1":0.7960833079573262,
+      "naucs_at_100_max":0.838262046357413,
+      "naucs_at_100_std":0.904184740422419,
+      "naucs_at_100_diff1":0.7996865824491111
+   },
+   "vidore/infovqa_test_subsampled":{
+      "ndcg_at_1":0.90081,
+      "ndcg_at_3":0.92934,
+      "ndcg_at_5":0.93352,
+      "ndcg_at_10":0.93669,
+      "ndcg_at_20":0.93871,
+      "ndcg_at_50":0.9403,
+      "ndcg_at_100":0.94099,
+      "map_at_1":0.90081,
+      "map_at_3":0.9224,
+      "map_at_5":0.92473,
+      "map_at_10":0.92598,
+      "map_at_20":0.92652,
+      "map_at_50":0.92677,
+      "map_at_100":0.92684,
+      "recall_at_1":0.90081,
+      "recall_at_3":0.94939,
+      "recall_at_5":0.95951,
+      "recall_at_10":0.96964,
+      "recall_at_20":0.97773,
+      "recall_at_50":0.98583,
+      "recall_at_100":0.98988,
+      "precision_at_1":0.90081,
+      "precision_at_3":0.31646,
+      "precision_at_5":0.1919,
+      "precision_at_10":0.09696,
+      "precision_at_20":0.04889,
+      "precision_at_50":0.01972,
+      "precision_at_100":0.0099,
+      "mrr_at_1":0.9008097165991903,
+      "mrr_at_3":0.9210526315789472,
+      "mrr_at_5":0.9248987854251012,
+      "mrr_at_10":0.9258040935672512,
+      "mrr_at_20":0.926239353314252,
+      "mrr_at_50":0.9265337558342984,
+      "mrr_at_100":0.9266429029872886,
+      "naucs_at_1_max":0.666687407456454,
+      "naucs_at_1_std":0.04348917328850024,
+      "naucs_at_1_diff1":0.9460185154989313,
+      "naucs_at_3_max":0.887692938235415,
+      "naucs_at_3_std":0.3724699632820944,
+      "naucs_at_3_diff1":0.9582075346992748,
+      "naucs_at_5_max":0.9121353932948266,
+      "naucs_at_5_std":0.46274982003896187,
+      "naucs_at_5_diff1":0.9608195637805694,
+      "naucs_at_10_max":0.9553640600031791,
+      "naucs_at_10_std":0.6527520622928011,
+      "naucs_at_10_diff1":0.9564661819784096,
+      "naucs_at_20_max":0.9643814216187027,
+      "naucs_at_20_std":0.8342696288850537,
+      "naucs_at_20_diff1":0.9643814216187027,
+      "naucs_at_50_max":1.0,
+      "naucs_at_50_std":0.960323608891707,
+      "naucs_at_50_diff1":0.9626852988386237,
+      "naucs_at_100_max":1.0,
+      "naucs_at_100_std":1.0,
+      "naucs_at_100_diff1":0.9477594183740937
+   },
+   "vidore/tabfquad_test_subsampled":{
+      "ndcg_at_1":0.925,
+      "ndcg_at_3":0.95561,
+      "ndcg_at_5":0.95991,
+      "ndcg_at_10":0.96464,
+      "ndcg_at_20":0.96464,
+      "ndcg_at_50":0.96464,
+      "ndcg_at_100":0.96464,
+      "map_at_1":0.925,
+      "map_at_3":0.94881,
+      "map_at_5":0.95113,
+      "map_at_10":0.95314,
+      "map_at_20":0.95314,
+      "map_at_50":0.95314,
+      "map_at_100":0.95314,
+      "recall_at_1":0.925,
+      "recall_at_3":0.975,
+      "recall_at_5":0.98571,
+      "recall_at_10":1.0,
+      "recall_at_20":1.0,
+      "recall_at_50":1.0,
+      "recall_at_100":1.0,
+      "precision_at_1":0.925,
+      "precision_at_3":0.325,
+      "precision_at_5":0.19714,
+      "precision_at_10":0.1,
+      "precision_at_20":0.05,
+      "precision_at_50":0.02,
+      "precision_at_100":0.01,
+      "mrr_at_1":0.9214285714285714,
+      "mrr_at_3":0.9464285714285714,
+      "mrr_at_5":0.9487499999999999,
+      "mrr_at_10":0.9507624716553289,
+      "mrr_at_20":0.9507624716553289,
+      "mrr_at_50":0.9507624716553289,
+      "mrr_at_100":0.9507624716553289,
+      "naucs_at_1_max":0.5344804588502072,
+      "naucs_at_1_std":0.26817393624116476,
+      "naucs_at_1_diff1":0.9383086567960518,
+      "naucs_at_3_max":0.9416433239962675,
+      "naucs_at_3_std":0.8989595838335351,
+      "naucs_at_3_diff1":0.98132586367881,
+      "naucs_at_5_max":0.967320261437913,
+      "naucs_at_5_std":0.8558590102707857,
+      "naucs_at_5_diff1":0.967320261437913,
+      "naucs_at_10_max":1.0,
+      "naucs_at_10_std":1.0,
+      "naucs_at_10_diff1":1.0,
+      "naucs_at_20_max":1.0,
+      "naucs_at_20_std":1.0,
+      "naucs_at_20_diff1":1.0,
+      "naucs_at_50_max":1.0,
+      "naucs_at_50_std":1.0,
+      "naucs_at_50_diff1":1.0,
+      "naucs_at_100_max":1.0,
+      "naucs_at_100_std":1.0,
+      "naucs_at_100_diff1":1.0
+   },
+   "vidore/tatdqa_test":{
+      "ndcg_at_1":0.70109,
+      "ndcg_at_3":0.79187,
+      "ndcg_at_5":0.81111,
+      "ndcg_at_10":0.82342,
+      "ndcg_at_20":0.8274,
+      "ndcg_at_50":0.83026,
+      "ndcg_at_100":0.83293,
+      "map_at_1":0.70109,
+      "map_at_3":0.77005,
+      "map_at_5":0.78071,
+      "map_at_10":0.78598,
+      "map_at_20":0.78706,
+      "map_at_50":0.78751,
+      "map_at_100":0.78775,
+      "recall_at_1":0.70109,
+      "recall_at_3":0.8548,
+      "recall_at_5":0.90158,
+      "recall_at_10":0.93864,
+      "recall_at_20":0.95443,
+      "recall_at_50":0.96902,
+      "recall_at_100":0.98542,
+      "precision_at_1":0.70109,
+      "precision_at_3":0.28493,
+      "precision_at_5":0.18032,
+      "precision_at_10":0.09386,
+      "precision_at_20":0.04772,
+      "precision_at_50":0.01938,
+      "precision_at_100":0.00985,
+      "mrr_at_1":0.7023086269744836,
+      "mrr_at_3":0.7714661806399354,
+      "mrr_at_5":0.782978938841637,
+      "mrr_at_10":0.7879385330478899,
+      "mrr_at_20":0.7889867733618092,
+      "mrr_at_50":0.7893770561483674,
+      "mrr_at_100":0.7896057919808999,
+      "naucs_at_1_max":0.25184895001768326,
+      "naucs_at_1_std":0.037017257538968965,
+      "naucs_at_1_diff1":0.8508843501202716,
+      "naucs_at_3_max":0.28836101906742934,
+      "naucs_at_3_std":0.0895587034833779,
+      "naucs_at_3_diff1":0.7677154118662606,
+      "naucs_at_5_max":0.33141107758129335,
+      "naucs_at_5_std":0.1402343443047445,
+      "naucs_at_5_diff1":0.726113129605695,
+      "naucs_at_10_max":0.3753484131980775,
+      "naucs_at_10_std":0.21181536222478634,
+      "naucs_at_10_diff1":0.7110698774619854,
+      "naucs_at_20_max":0.398127644970406,
+      "naucs_at_20_std":0.2900002928945751,
+      "naucs_at_20_diff1":0.6944425419244158,
+      "naucs_at_50_max":0.4094042090297014,
+      "naucs_at_50_std":0.3282254760208566,
+      "naucs_at_50_diff1":0.665997592036148,
+      "naucs_at_100_max":0.2601633836764814,
+      "naucs_at_100_std":0.2532897060083396,
+      "naucs_at_100_diff1":0.6808982214080673
+   },
+   "vidore/shiftproject_test":{
+      "ndcg_at_1":0.84,
+      "ndcg_at_3":0.91809,
+      "ndcg_at_5":0.91809,
+      "ndcg_at_10":0.92499,
+      "ndcg_at_20":0.92499,
+      "ndcg_at_50":0.92499,
+      "ndcg_at_100":0.92673,
+      "map_at_1":0.84,
+      "map_at_3":0.9,
+      "map_at_5":0.9,
+      "map_at_10":0.9031,
+      "map_at_20":0.9031,
+      "map_at_50":0.9031,
+      "map_at_100":0.90328,
+      "recall_at_1":0.84,
+      "recall_at_3":0.97,
+      "recall_at_5":0.97,
+      "recall_at_10":0.99,
+      "recall_at_20":0.99,
+      "recall_at_50":0.99,
+      "recall_at_100":1.0,
+      "precision_at_1":0.84,
+      "precision_at_3":0.32333,
+      "precision_at_5":0.194,
+      "precision_at_10":0.099,
+      "precision_at_20":0.0495,
+      "precision_at_50":0.0198,
+      "precision_at_100":0.01,
+      "mrr_at_1":0.84,
+      "mrr_at_3":0.8983333333333333,
+      "mrr_at_5":0.8983333333333333,
+      "mrr_at_10":0.9014285714285714,
+      "mrr_at_20":0.9014285714285714,
+      "mrr_at_50":0.9014285714285714,
+      "mrr_at_100":0.9016208791208791,
+      "naucs_at_1_max":0.1685076092292596,
+      "naucs_at_1_std":-0.2145925380461453,
+      "naucs_at_1_diff1":0.7751595483554242,
+      "naucs_at_3_max":0.6498599439775861,
+      "naucs_at_3_std":0.36834733893557625,
+      "naucs_at_3_diff1":0.664021164021167,
+      "naucs_at_5_max":0.6498599439775937,
+      "naucs_at_5_std":0.36834733893557176,
+      "naucs_at_5_diff1":0.6640211640211615,
+      "naucs_at_10_max":0.7222222222222276,
+      "naucs_at_10_std":0.5541549953314738,
+      "naucs_at_10_diff1":0.8692810457516413,
+      "naucs_at_20_max":0.7222222222222276,
+      "naucs_at_20_std":0.5541549953314738,
+      "naucs_at_20_diff1":0.8692810457516413,
+      "naucs_at_50_max":0.7222222222222041,
+      "naucs_at_50_std":0.554154995331464,
+      "naucs_at_50_diff1":0.8692810457516374,
+      "naucs_at_100_max":null,
+      "naucs_at_100_std":null,
+      "naucs_at_100_diff1":null
+   },
+   "vidore/syntheticDocQA_artificial_intelligence_test":{
+      "ndcg_at_1":0.98,
+      "ndcg_at_3":0.99262,
+      "ndcg_at_5":0.99262,
+      "ndcg_at_10":0.99262,
+      "ndcg_at_20":0.99262,
+      "ndcg_at_50":0.99262,
+      "ndcg_at_100":0.99262,
+      "map_at_1":0.98,
+      "map_at_3":0.99,
+      "map_at_5":0.99,
+      "map_at_10":0.99,
+      "map_at_20":0.99,
+      "map_at_50":0.99,
+      "map_at_100":0.99,
+      "recall_at_1":0.98,
+      "recall_at_3":1.0,
+      "recall_at_5":1.0,
+      "recall_at_10":1.0,
+      "recall_at_20":1.0,
+      "recall_at_50":1.0,
+      "recall_at_100":1.0,
+      "precision_at_1":0.98,
+      "precision_at_3":0.33333,
+      "precision_at_5":0.2,
+      "precision_at_10":0.1,
+      "precision_at_20":0.05,
+      "precision_at_50":0.02,
+      "precision_at_100":0.01,
+      "mrr_at_1":0.99,
+      "mrr_at_3":0.995,
+      "mrr_at_5":0.995,
+      "mrr_at_10":0.995,
+      "mrr_at_20":0.995,
+      "mrr_at_50":0.995,
+      "mrr_at_100":0.995,
+      "naucs_at_1_max":0.540149393090569,
+      "naucs_at_1_std":-0.6909430438842186,
+      "naucs_at_1_diff1":1.0,
+      "naucs_at_3_max":1.0,
+      "naucs_at_3_std":1.0,
+      "naucs_at_3_diff1":1.0,
+      "naucs_at_5_max":1.0,
+      "naucs_at_5_std":1.0,
+      "naucs_at_5_diff1":1.0,
+      "naucs_at_10_max":1.0,
+      "naucs_at_10_std":1.0,
+      "naucs_at_10_diff1":1.0,
+      "naucs_at_20_max":1.0,
+      "naucs_at_20_std":1.0,
+      "naucs_at_20_diff1":1.0,
+      "naucs_at_50_max":null,
+      "naucs_at_50_std":null,
+      "naucs_at_50_diff1":null,
+      "naucs_at_100_max":null,
+      "naucs_at_100_std":null,
+      "naucs_at_100_diff1":null
+   },
+   "vidore/syntheticDocQA_energy_test":{
+      "ndcg_at_1":0.95,
+      "ndcg_at_3":0.95631,
+      "ndcg_at_5":0.96492,
+      "ndcg_at_10":0.96781,
+      "ndcg_at_20":0.96781,
+      "ndcg_at_50":0.96976,
+      "ndcg_at_100":0.96976,
+      "map_at_1":0.95,
+      "map_at_3":0.955,
+      "map_at_5":0.96,
+      "map_at_10":0.961,
+      "map_at_20":0.961,
+      "map_at_50":0.96129,
+      "map_at_100":0.96129,
+      "recall_at_1":0.95,
+      "recall_at_3":0.96,
+      "recall_at_5":0.98,
+      "recall_at_10":0.99,
+      "recall_at_20":0.99,
+      "recall_at_50":1.0,
+      "recall_at_100":1.0,
+      "precision_at_1":0.95,
+      "precision_at_3":0.32,
+      "precision_at_5":0.196,
+      "precision_at_10":0.099,
+      "precision_at_20":0.0495,
+      "precision_at_50":0.02,
+      "precision_at_100":0.01,
+      "mrr_at_1":0.95,
+      "mrr_at_3":0.955,
+      "mrr_at_5":0.96,
+      "mrr_at_10":0.961,
+      "mrr_at_20":0.961,
+      "mrr_at_50":0.9613030303030302,
+      "mrr_at_100":0.9613030303030302,
+      "naucs_at_1_max":0.32362278244631343,
+      "naucs_at_1_std":-0.7713352007469629,
+      "naucs_at_1_diff1":1.0,
+      "naucs_at_3_max":0.8395191409897237,
+      "naucs_at_3_std":-0.6762371615312763,
+      "naucs_at_3_diff1":1.0,
+      "naucs_at_5_max":0.6790382819794609,
+      "naucs_at_5_std":-1.445845004668519,
+      "naucs_at_5_diff1":1.0,
+      "naucs_at_10_max":1.0,
+      "naucs_at_10_std":-1.1517273576097316,
+      "naucs_at_10_diff1":1.0,
+      "naucs_at_20_max":1.0,
+      "naucs_at_20_std":-1.1517273576097316,
+      "naucs_at_20_diff1":1.0,
+      "naucs_at_50_max":null,
+      "naucs_at_50_std":null,
+      "naucs_at_50_diff1":null,
+      "naucs_at_100_max":null,
+      "naucs_at_100_std":null,
+      "naucs_at_100_diff1":null
+   },
+   "vidore/syntheticDocQA_government_reports_test":{
+      "ndcg_at_1":0.91,
+      "ndcg_at_3":0.94655,
+      "ndcg_at_5":0.95085,
+      "ndcg_at_10":0.95743,
+      "ndcg_at_20":0.95743,
+      "ndcg_at_50":0.95743,
+      "ndcg_at_100":0.95743,
+      "map_at_1":0.91,
+      "map_at_3":0.93833,
+      "map_at_5":0.94083,
+      "map_at_10":0.94361,
+      "map_at_20":0.94361,
+      "map_at_50":0.94361,
+      "map_at_100":0.94361,
+      "recall_at_1":0.91,
+      "recall_at_3":0.97,
+      "recall_at_5":0.98,
+      "recall_at_10":1.0,
+      "recall_at_20":1.0,
+      "recall_at_50":1.0,
+      "recall_at_100":1.0,
+      "precision_at_1":0.91,
+      "precision_at_3":0.32333,
+      "precision_at_5":0.196,
+      "precision_at_10":0.1,
+      "precision_at_20":0.05,
+      "precision_at_50":0.02,
+      "precision_at_100":0.01,
+      "mrr_at_1":0.92,
+      "mrr_at_3":0.9433333333333332,
+      "mrr_at_5":0.9483333333333333,
+      "mrr_at_10":0.9494444444444444,
+      "mrr_at_20":0.9494444444444444,
+      "mrr_at_50":0.9494444444444444,
+      "mrr_at_100":0.9494444444444444,
+      "naucs_at_1_max":0.6774561676522453,
+      "naucs_at_1_std":0.14721444133208894,
+      "naucs_at_1_diff1":0.9564270152505436,
+      "naucs_at_3_max":0.8513849984438244,
+      "naucs_at_3_std":-0.24929971988795643,
+      "naucs_at_3_diff1":0.9564270152505466,
+      "naucs_at_5_max":1.0,
+      "naucs_at_5_std":-0.43534080298785716,
+      "naucs_at_5_diff1":0.9346405228758136,
+      "naucs_at_10_max":1.0,
+      "naucs_at_10_std":1.0,
+      "naucs_at_10_diff1":1.0,
+      "naucs_at_20_max":1.0,
+      "naucs_at_20_std":1.0,
+      "naucs_at_20_diff1":1.0,
+      "naucs_at_50_max":null,
+      "naucs_at_50_std":null,
+      "naucs_at_50_diff1":null,
+      "naucs_at_100_max":null,
+      "naucs_at_100_std":null,
+      "naucs_at_100_diff1":null
+   },
+   "vidore/syntheticDocQA_healthcare_industry_test":{
+      "ndcg_at_1":0.98,
+      "ndcg_at_3":0.99262,
+      "ndcg_at_5":0.99262,
+      "ndcg_at_10":0.99262,
+      "ndcg_at_20":0.99262,
+      "ndcg_at_50":0.99262,
+      "ndcg_at_100":0.99262,
+      "map_at_1":0.98,
+      "map_at_3":0.99,
+      "map_at_5":0.99,
+      "map_at_10":0.99,
+      "map_at_20":0.99,
+      "map_at_50":0.99,
+      "map_at_100":0.99,
+      "recall_at_1":0.98,
+      "recall_at_3":1.0,
+      "recall_at_5":1.0,
+      "recall_at_10":1.0,
+      "recall_at_20":1.0,
+      "recall_at_50":1.0,
+      "recall_at_100":1.0,
+      "precision_at_1":0.98,
+      "precision_at_3":0.33333,
+      "precision_at_5":0.2,
+      "precision_at_10":0.1,
+      "precision_at_20":0.05,
+      "precision_at_50":0.02,
+      "precision_at_100":0.01,
+      "mrr_at_1":0.98,
+      "mrr_at_3":0.99,
+      "mrr_at_5":0.99,
+      "mrr_at_10":0.99,
+      "mrr_at_20":0.99,
+      "mrr_at_50":0.99,
+      "mrr_at_100":0.99,
+      "naucs_at_1_max":0.7222222222222248,
+      "naucs_at_1_std":-0.5634920634920563,
+      "naucs_at_1_diff1":1.0,
+      "naucs_at_3_max":1.0,
+      "naucs_at_3_std":1.0,
+      "naucs_at_3_diff1":1.0,
+      "naucs_at_5_max":1.0,
+      "naucs_at_5_std":1.0,
+      "naucs_at_5_diff1":1.0,
+      "naucs_at_10_max":1.0,
+      "naucs_at_10_std":1.0,
+      "naucs_at_10_diff1":1.0,
+      "naucs_at_20_max":1.0,
+      "naucs_at_20_std":1.0,
+      "naucs_at_20_diff1":1.0,
+      "naucs_at_50_max":null,
+      "naucs_at_50_std":null,
+      "naucs_at_50_diff1":null,
+      "naucs_at_100_max":null,
+      "naucs_at_100_std":null,
+      "naucs_at_100_diff1":null
+   },
+   "vidore/synthetic_mit_biomedical_tissue_interactions_unfiltered":{
+      "ndcg_at_1":0.66875,
+      "ndcg_at_3":0.64833,
+      "ndcg_at_5":0.66136,
+      "ndcg_at_10":0.7002,
+      "ndcg_at_20":0.72343,
+      "ndcg_at_50":0.74221,
+      "ndcg_at_100":0.7504,
+      "map_at_1":0.41001,
+      "map_at_3":0.53585,
+      "map_at_5":0.57453,
+      "map_at_10":0.61456,
+      "map_at_20":0.63039,
+      "map_at_50":0.63734,
+      "map_at_100":0.63926,
+      "recall_at_1":0.41001,
+      "recall_at_3":0.59207,
+      "recall_at_5":0.66951,
+      "recall_at_10":0.7842,
+      "recall_at_20":0.84612,
+      "recall_at_50":0.90015,
+      "recall_at_100":0.92697,
+      "precision_at_1":0.66875,
+      "precision_at_3":0.39375,
+      "precision_at_5":0.29125,
+      "precision_at_10":0.18938,
+      "precision_at_20":0.11,
+      "precision_at_50":0.05087,
+      "precision_at_100":0.0275,
+      "mrr_at_1":0.65,
+      "mrr_at_3":0.7364583333333331,
+      "mrr_at_5":0.7480208333333331,
+      "mrr_at_10":0.7537127976190475,
+      "mrr_at_20":0.7549048402255638,
+      "mrr_at_50":0.755591370159045,
+      "mrr_at_100":0.755591370159045,
+      "naucs_at_1_max":0.3941383204398901,
+      "naucs_at_1_std":-0.03446929320007161,
+      "naucs_at_1_diff1":0.4503277010269023,
+      "naucs_at_3_max":0.029009734517562568,
+      "naucs_at_3_std":0.005402681474374929,
+      "naucs_at_3_diff1":-0.048706505991403734,
+      "naucs_at_5_max":-0.08207834225628742,
+      "naucs_at_5_std":-0.0032984001708723115,
+      "naucs_at_5_diff1":-0.17283336776461594,
+      "naucs_at_10_max":-0.17977285443640584,
+      "naucs_at_10_std":-0.04541684253923944,
+      "naucs_at_10_diff1":-0.23295672062273604,
+      "naucs_at_20_max":-0.170593905587224,
+      "naucs_at_20_std":0.015384406511413186,
+      "naucs_at_20_diff1":-0.2551053459454954,
+      "naucs_at_50_max":-0.22984435384481694,
+      "naucs_at_50_std":0.04443493398633732,
+      "naucs_at_50_diff1":-0.27937793023113594,
+      "naucs_at_100_max":-0.25957527630817795,
+      "naucs_at_100_std":0.057081217133354005,
+      "naucs_at_100_diff1":-0.3033742727110607
+   },
+   "vidore/synthetic_economics_macro_economy_2024_filtered_v1.0":{
+      "ndcg_at_1":0.65517,
+      "ndcg_at_3":0.62816,
+      "ndcg_at_5":0.6159,
+      "ndcg_at_10":0.58912,
+      "ndcg_at_20":0.61706,
+      "ndcg_at_50":0.67765,
+      "ndcg_at_100":0.71471,
+      "map_at_1":0.08673,
+      "map_at_3":0.19894,
+      "map_at_5":0.26716,
+      "map_at_10":0.33292,
+      "map_at_20":0.40089,
+      "map_at_50":0.45667,
+      "map_at_100":0.4844,
+      "recall_at_1":0.08673,
+      "recall_at_3":0.24596,
+      "recall_at_5":0.34869,
+      "recall_at_10":0.4683,
+      "recall_at_20":0.62193,
+      "recall_at_50":0.80907,
+      "recall_at_100":0.92119,
+      "precision_at_1":0.65517,
+      "precision_at_3":0.58621,
+      "precision_at_5":0.55172,
+      "precision_at_10":0.42931,
+      "precision_at_20":0.32672,
+      "precision_at_50":0.19759,
+      "precision_at_100":0.1281,
+      "mrr_at_1":0.6896551724137931,
+      "mrr_at_3":0.7902298850574713,
+      "mrr_at_5":0.7971264367816091,
+      "mrr_at_10":0.8015051997810618,
+      "mrr_at_20":0.8015051997810618,
+      "mrr_at_50":0.8023262178434591,
+      "mrr_at_100":0.8023262178434591,
+      "naucs_at_1_max":-0.046307070115366736,
+      "naucs_at_1_std":0.15790688174002224,
+      "naucs_at_1_diff1":-0.14987305341556462,
+      "naucs_at_3_max":0.11671339502888234,
+      "naucs_at_3_std":0.11834362034275059,
+      "naucs_at_3_diff1":-0.17762652230301346,
+      "naucs_at_5_max":0.10679449683484428,
+      "naucs_at_5_std":0.09453878709557821,
+      "naucs_at_5_diff1":-0.1792193824994377,
+      "naucs_at_10_max":0.10651036963192122,
+      "naucs_at_10_std":0.207447480653671,
+      "naucs_at_10_diff1":-0.23752907816911517,
+      "naucs_at_20_max":0.05980777985340397,
+      "naucs_at_20_std":0.23365933479100473,
+      "naucs_at_20_diff1":-0.2025898492844735,
+      "naucs_at_50_max":-0.012623885278852142,
+      "naucs_at_50_std":0.2463485456131626,
+      "naucs_at_50_diff1":-0.18909262376408734,
+      "naucs_at_100_max":-0.08872391317584416,
+      "naucs_at_100_std":0.17948141514389135,
+      "naucs_at_100_diff1":-0.186930857132979
+   },
+   "vidore/synthetic_rse_restaurant_filtered_v1.0":{
+      "ndcg_at_1":0.52632,
+      "ndcg_at_3":0.52504,
+      "ndcg_at_5":0.57289,
+      "ndcg_at_10":0.62003,
+      "ndcg_at_20":0.65505,
+      "ndcg_at_50":0.68979,
+      "ndcg_at_100":0.70204,
+      "map_at_1":0.29474,
+      "map_at_3":0.40396,
+      "map_at_5":0.45972,
+      "map_at_10":0.50866,
+      "map_at_20":0.5276,
+      "map_at_50":0.54534,
+      "map_at_100":0.55161,
+      "recall_at_1":0.29474,
+      "recall_at_3":0.48191,
+      "recall_at_5":0.61543,
+      "recall_at_10":0.75002,
+      "recall_at_20":0.85537,
+      "recall_at_50":0.94841,
+      "recall_at_100":0.98246,
+      "precision_at_1":0.52632,
+      "precision_at_3":0.34503,
+      "precision_at_5":0.28421,
+      "precision_at_10":0.19649,
+      "precision_at_20":0.12281,
+      "precision_at_50":0.06912,
+      "precision_at_100":0.03877,
+      "mrr_at_1":0.543859649122807,
+      "mrr_at_3":0.6403508771929824,
+      "mrr_at_5":0.6526315789473685,
+      "mrr_at_10":0.6602548036758563,
+      "mrr_at_20":0.6651739829371409,
+      "mrr_at_50":0.6655248601301235,
+      "mrr_at_100":0.6655248601301235,
+      "naucs_at_1_max":0.07429395157545386,
+      "naucs_at_1_std":0.03015644215746205,
+      "naucs_at_1_diff1":0.029188519373247488,
+      "naucs_at_3_max":-0.14556834354246306,
+      "naucs_at_3_std":-0.03551874034674307,
+      "naucs_at_3_diff1":-0.03426575331185958,
+      "naucs_at_5_max":-0.132667608690031,
+      "naucs_at_5_std":0.046883073270420496,
+      "naucs_at_5_diff1":0.020789980948042533,
+      "naucs_at_10_max":-0.15350074281217774,
+      "naucs_at_10_std":0.021602996730365333,
+      "naucs_at_10_diff1":0.10610855445931551,
+      "naucs_at_20_max":-0.2870595843942612,
+      "naucs_at_20_std":-0.053200969005437526,
+      "naucs_at_20_diff1":0.009614869287472011,
+      "naucs_at_50_max":-0.3747840510110352,
+      "naucs_at_50_std":-0.1315278407645466,
+      "naucs_at_50_diff1":-0.08682413153737878,
+      "naucs_at_100_max":-0.3797677839881161,
+      "naucs_at_100_std":-0.11609328262961503,
+      "naucs_at_100_diff1":-0.12296867865157128
+   },
+   "vidore/synthetic_axa_filtered_v1.0":{
+      "ndcg_at_1":0.61111,
+      "ndcg_at_3":0.6462,
+      "ndcg_at_5":0.68292,
+      "ndcg_at_10":0.70017,
+      "ndcg_at_20":0.72201,
+      "ndcg_at_50":0.7613,
+      "ndcg_at_100":0.7658,
+      "map_at_1":0.29544,
+      "map_at_3":0.41914,
+      "map_at_5":0.49958,
+      "map_at_10":0.57784,
+      "map_at_20":0.61104,
+      "map_at_50":0.63144,
+      "map_at_100":0.63315,
+      "recall_at_1":0.29544,
+      "recall_at_3":0.51973,
+      "recall_at_5":0.66378,
+      "recall_at_10":0.78472,
+      "recall_at_20":0.85945,
+      "recall_at_50":0.97412,
+      "recall_at_100":0.98765,
+      "precision_at_1":0.61111,
+      "precision_at_3":0.46296,
+      "precision_at_5":0.42222,
+      "precision_at_10":0.31111,
+      "precision_at_20":0.19167,
+      "precision_at_50":0.09111,
+      "precision_at_100":0.04667,
+      "mrr_at_1":0.5555555555555556,
+      "mrr_at_3":0.6759259259259259,
+      "mrr_at_5":0.7037037037037037,
+      "mrr_at_10":0.7037037037037037,
+      "mrr_at_20":0.7037037037037037,
+      "mrr_at_50":0.7060185185185185,
+      "mrr_at_100":0.7060185185185185,
+      "naucs_at_1_max":0.06370640145678692,
+      "naucs_at_1_std":0.42153041887902504,
+      "naucs_at_1_diff1":0.35531111328419196,
+      "naucs_at_3_max":-0.49137089517804067,
+      "naucs_at_3_std":-0.41095639069482337,
+      "naucs_at_3_diff1":0.01707919337550571,
+      "naucs_at_5_max":-0.6120658234932744,
+      "naucs_at_5_std":-0.474389941591449,
+      "naucs_at_5_diff1":-0.165061956148533,
+      "naucs_at_10_max":-0.6883907404624792,
+      "naucs_at_10_std":-0.45513671453258775,
+      "naucs_at_10_diff1":-0.42316346500786917,
+      "naucs_at_20_max":-0.733957515102311,
+      "naucs_at_20_std":-0.4723325528538845,
+      "naucs_at_20_diff1":-0.4807537601912433,
+      "naucs_at_50_max":-0.7013881037720148,
+      "naucs_at_50_std":-0.45324759826825234,
+      "naucs_at_50_diff1":-0.4882986634795409,
+      "naucs_at_100_max":-0.6924248438890106,
+      "naucs_at_100_std":-0.4400487565870318,
+      "naucs_at_100_diff1":-0.4661845036852439
+   },
+   "vidore/synthetic_rse_restaurant_filtered_v1.0_multilingual":{
+      "ndcg_at_1":0.54825,
+      "ndcg_at_3":0.53008,
+      "ndcg_at_5":0.56705,
+      "ndcg_at_10":0.6257,
+      "ndcg_at_20":0.6625,
+      "ndcg_at_50":0.69353,
+      "ndcg_at_100":0.70465,
+      "map_at_1":0.29816,
+      "map_at_3":0.4016,
+      "map_at_5":0.4527,
+      "map_at_10":0.50738,
+      "map_at_20":0.52962,
+      "map_at_50":0.5466,
+      "map_at_100":0.55224,
+      "recall_at_1":0.29816,
+      "recall_at_3":0.48014,
+      "recall_at_5":0.59266,
+      "recall_at_10":0.76588,
+      "recall_at_20":0.87318,
+      "recall_at_50":0.95235,
+      "recall_at_100":0.98465,
+      "precision_at_1":0.54825,
+      "precision_at_3":0.34795,
+      "precision_at_5":0.28158,
+      "precision_at_10":0.19956,
+      "precision_at_20":0.12654,
+      "precision_at_50":0.06991,
+      "precision_at_100":0.03877,
+      "mrr_at_1":0.5394736842105263,
+      "mrr_at_3":0.6381578947368421,
+      "mrr_at_5":0.6528508771929824,
+      "mrr_at_10":0.666278543581175,
+      "mrr_at_20":0.6693822460076329,
+      "mrr_at_50":0.6696454039023697,
+      "mrr_at_100":0.6697038834345334,
+      "naucs_at_1_max":-0.10947777032499051,
+      "naucs_at_1_std":-0.08674361998375714,
+      "naucs_at_1_diff1":0.057249370353071344,
+      "naucs_at_3_max":-0.06687224875668692,
+      "naucs_at_3_std":0.0484364629589946,
+      "naucs_at_3_diff1":-0.05462426569089335,
+      "naucs_at_5_max":-0.08403937224497868,
+      "naucs_at_5_std":0.07293419772827552,
+      "naucs_at_5_diff1":-0.05958666027516894,
+      "naucs_at_10_max":-0.10838130867204888,
+      "naucs_at_10_std":0.05587982504285361,
+      "naucs_at_10_diff1":0.002192750308197299,
+      "naucs_at_20_max":-0.1628596976822229,
+      "naucs_at_20_std":0.06415536794747155,
+      "naucs_at_20_diff1":-0.0901131230050924,
+      "naucs_at_50_max":-0.2471444987505313,
+      "naucs_at_50_std":0.02178464235724526,
+      "naucs_at_50_diff1":-0.16484319783599452,
+      "naucs_at_100_max":-0.2476326734790779,
+      "naucs_at_100_std":0.04921366491713892,
+      "naucs_at_100_diff1":-0.1925485969942229
+   },
+   "vidore/synthetic_axa_filtered_v1.0_multilingual":{
+      "ndcg_at_1":0.56944,
+      "ndcg_at_3":0.60114,
+      "ndcg_at_5":0.61308,
+      "ndcg_at_10":0.63301,
+      "ndcg_at_20":0.65962,
+      "ndcg_at_50":0.70087,
+      "ndcg_at_100":0.71794,
+      "map_at_1":0.26807,
+      "map_at_3":0.38274,
+      "map_at_5":0.44543,
+      "map_at_10":0.51783,
+      "map_at_20":0.55091,
+      "map_at_50":0.57108,
+      "map_at_100":0.57464,
+      "recall_at_1":0.26807,
+      "recall_at_3":0.47345,
+      "recall_at_5":0.56537,
+      "recall_at_10":0.69703,
+      "recall_at_20":0.78343,
+      "recall_at_50":0.90152,
+      "recall_at_100":0.97889,
+      "precision_at_1":0.56944,
+      "precision_at_3":0.43056,
+      "precision_at_5":0.38056,
+      "precision_at_10":0.27917,
+      "precision_at_20":0.17569,
+      "precision_at_50":0.08583,
+      "precision_at_100":0.04569,
+      "mrr_at_1":0.5,
+      "mrr_at_3":0.6481481481481483,
+      "mrr_at_5":0.6550925925925927,
+      "mrr_at_10":0.6566358024691359,
+      "mrr_at_20":0.6577041785375121,
+      "mrr_at_50":0.659896412080552,
+      "mrr_at_100":0.6607831288544433,
+      "naucs_at_1_max":0.16468725740231643,
+      "naucs_at_1_std":0.37919017176717934,
+      "naucs_at_1_diff1":0.30214037198755495,
+      "naucs_at_3_max":-0.2419012234949494,
+      "naucs_at_3_std":-0.11880659562104584,
+      "naucs_at_3_diff1":-0.05983117053027332,
+      "naucs_at_5_max":-0.3435266789444122,
+      "naucs_at_5_std":-0.15122373891433766,
+      "naucs_at_5_diff1":-0.14301810021483025,
+      "naucs_at_10_max":-0.4049299119799346,
+      "naucs_at_10_std":-0.11551373909975436,
+      "naucs_at_10_diff1":-0.3087348082925975,
+      "naucs_at_20_max":-0.4557080881366001,
+      "naucs_at_20_std":-0.1316035057344777,
+      "naucs_at_20_diff1":-0.3767740930238455,
+      "naucs_at_50_max":-0.48221867587284106,
+      "naucs_at_50_std":-0.16520884486012927,
+      "naucs_at_50_diff1":-0.437539802211166,
+      "naucs_at_100_max":-0.5041035001503106,
+      "naucs_at_100_std":-0.19246332353391876,
+      "naucs_at_100_diff1":-0.450877397124322
+   },
+   "vidore/synthetic_mit_biomedical_tissue_interactions_unfiltered_multilingual":{
+      "ndcg_at_1":0.62656,
+      "ndcg_at_3":0.62898,
+      "ndcg_at_5":0.64249,
+      "ndcg_at_10":0.67441,
+      "ndcg_at_20":0.69935,
+      "ndcg_at_50":0.72034,
+      "ndcg_at_100":0.7304,
+      "map_at_1":0.37553,
+      "map_at_3":0.51341,
+      "map_at_5":0.55134,
+      "map_at_10":0.58654,
+      "map_at_20":0.60275,
+      "map_at_50":0.61047,
+      "map_at_100":0.61272,
+      "recall_at_1":0.37553,
+      "recall_at_3":0.58317,
+      "recall_at_5":0.6651,
+      "recall_at_10":0.76267,
+      "recall_at_20":0.83047,
+      "recall_at_50":0.89402,
+      "recall_at_100":0.93005,
+      "precision_at_1":0.62656,
+      "precision_at_3":0.39115,
+      "precision_at_5":0.28812,
+      "precision_at_10":0.18453,
+      "precision_at_20":0.10859,
+      "precision_at_50":0.05059,
+      "precision_at_100":0.02756,
+      "mrr_at_1":0.6203125,
+      "mrr_at_3":0.7114583333333329,
+      "mrr_at_5":0.7251302083333327,
+      "mrr_at_10":0.7312574404761898,
+      "mrr_at_20":0.7330658025942703,
+      "mrr_at_50":0.7337640283895484,
+      "mrr_at_100":0.733871254289439,
+      "naucs_at_1_max":0.2039250412963779,
+      "naucs_at_1_std":-0.07024884868861188,
+      "naucs_at_1_diff1":0.45606770903022215,
+      "naucs_at_3_max":0.058212985355640734,
+      "naucs_at_3_std":-0.05959626421557797,
+      "naucs_at_3_diff1":-0.029625222250297182,
+      "naucs_at_5_max":0.012523889018321476,
+      "naucs_at_5_std":-0.05950139173785889,
+      "naucs_at_5_diff1":-0.12093229795276991,
+      "naucs_at_10_max":-0.07885246896142604,
+      "naucs_at_10_std":-0.07584435766192119,
+      "naucs_at_10_diff1":-0.19868179893511964,
+      "naucs_at_20_max":-0.08104765093094657,
+      "naucs_at_20_std":-0.020633911923048417,
+      "naucs_at_20_diff1":-0.2329474014068329,
+      "naucs_at_50_max":-0.10191575025539294,
+      "naucs_at_50_std":0.006497205696773289,
+      "naucs_at_50_diff1":-0.2654796386435107,
+      "naucs_at_100_max":-0.12912675673565918,
+      "naucs_at_100_std":0.002658413394124888,
+      "naucs_at_100_diff1":-0.286740024806076
+   },
+   "vidore/synthetics_economics_macro_economy_2024_filtered_v1.0_multilingual":{
+      "ndcg_at_1":0.56897,
+      "ndcg_at_3":0.55347,
+      "ndcg_at_5":0.54697,
+      "ndcg_at_10":0.54353,
+      "ndcg_at_20":0.5754,
+      "ndcg_at_50":0.64435,
+      "ndcg_at_100":0.68217,
+      "map_at_1":0.06995,
+      "map_at_3":0.16788,
+      "map_at_5":0.22262,
+      "map_at_10":0.29539,
+      "map_at_20":0.35806,
+      "map_at_50":0.41596,
+      "map_at_100":0.44377,
+      "recall_at_1":0.06995,
+      "recall_at_3":0.22001,
+      "recall_at_5":0.31105,
+      "recall_at_10":0.45156,
+      "recall_at_20":0.60587,
+      "recall_at_50":0.80629,
+      "recall_at_100":0.91617,
+      "precision_at_1":0.56897,
+      "precision_at_3":0.52155,
+      "precision_at_5":0.49483,
+      "precision_at_10":0.4069,
+      "precision_at_20":0.31207,
+      "precision_at_50":0.19595,
+      "precision_at_100":0.1275,
+      "mrr_at_1":0.5689655172413793,
+      "mrr_at_3":0.7040229885057472,
+      "mrr_at_5":0.7173850574712642,
+      "mrr_at_10":0.7212370005473451,
+      "mrr_at_20":0.7221635921046082,
+      "mrr_at_50":0.7223688466202075,
+      "mrr_at_100":0.7223688466202075,
+      "naucs_at_1_max":0.008287845664171043,
+      "naucs_at_1_std":0.1276905868842376,
+      "naucs_at_1_diff1":0.11685328065061831,
+      "naucs_at_3_max":0.04231917732527594,
+      "naucs_at_3_std":0.09097968258247123,
+      "naucs_at_3_diff1":0.07039779015197889,
+      "naucs_at_5_max":0.02450452954672108,
+      "naucs_at_5_std":0.08382631941812721,
+      "naucs_at_5_diff1":0.04903893807286711,
+      "naucs_at_10_max":0.06258885863723426,
+      "naucs_at_10_std":0.1489118063390922,
+      "naucs_at_10_diff1":-0.001772551183085018,
+      "naucs_at_20_max":0.05127122134169797,
+      "naucs_at_20_std":0.16207987797849627,
+      "naucs_at_20_diff1":-0.037084808056486104,
+      "naucs_at_50_max":0.012079926402609917,
+      "naucs_at_50_std":0.16924631618333094,
+      "naucs_at_50_diff1":-0.09727554327201936,
+      "naucs_at_100_max":-0.03658022842524374,
+      "naucs_at_100_std":0.12849977057146159,
+      "naucs_at_100_diff1":-0.10217040873502983
+   },
+   "vidore/restaurant_esg_reports_beir":{
+      "ndcg_at_1":0.71795,
+      "ndcg_at_3":0.71188,
+      "ndcg_at_5":0.73949,
+      "ndcg_at_10":0.77205,
+      "ndcg_at_20":0.78712,
+      "ndcg_at_50":0.80866,
+      "ndcg_at_100":0.81008,
+      "map_at_1":0.50513,
+      "map_at_3":0.62382,
+      "map_at_5":0.6711,
+      "map_at_10":0.70078,
+      "map_at_20":0.7096,
+      "map_at_50":0.71767,
+      "map_at_100":0.71825,
+      "recall_at_1":0.50513,
+      "recall_at_3":0.68668,
+      "recall_at_5":0.76906,
+      "recall_at_10":0.85835,
+      "recall_at_20":0.90725,
+      "recall_at_50":0.97476,
+      "recall_at_100":0.98071,
+      "precision_at_1":0.73077,
+      "precision_at_3":0.38462,
+      "precision_at_5":0.28846,
+      "precision_at_10":0.175,
+      "precision_at_20":0.09712,
+      "precision_at_50":0.045,
+      "precision_at_100":0.02288,
+      "mrr_at_1":0.75,
+      "mrr_at_3":0.8076923076923077,
+      "mrr_at_5":0.8125,
+      "mrr_at_10":0.8226495726495725,
+      "mrr_at_20":0.8226495726495725,
+      "mrr_at_50":0.8233363858363858,
+      "mrr_at_100":0.8233363858363858,
+      "naucs_at_1_max":0.012222350607946745,
+      "naucs_at_1_std":-0.09466371539982948,
+      "naucs_at_1_diff1":0.5022751223737485,
+      "naucs_at_3_max":-0.03148341001971903,
+      "naucs_at_3_std":-0.017366247697995007,
+      "naucs_at_3_diff1":-0.2806074619839212,
+      "naucs_at_5_max":-0.1368097107443235,
+      "naucs_at_5_std":-0.05686309477532154,
+      "naucs_at_5_diff1":-0.34542841772396893,
+      "naucs_at_10_max":-0.11047704445523453,
+      "naucs_at_10_std":0.016760999608655385,
+      "naucs_at_10_diff1":-0.36219386821062893,
+      "naucs_at_20_max":-0.18118046303544919,
+      "naucs_at_20_std":0.00660518472055942,
+      "naucs_at_20_diff1":-0.35322674833881423,
+      "naucs_at_50_max":-0.15496181424390146,
+      "naucs_at_50_std":0.025351601144475155,
+      "naucs_at_50_diff1":-0.3271145822876798,
+      "naucs_at_100_max":-0.14539299033899847,
+      "naucs_at_100_std":0.020343129158802924,
+      "naucs_at_100_diff1":-0.3237772572696072
+   }
+}

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5eee858c5123a4279c3e1f7b81247343f356ac767940b2692a928ad929543214
+size 11422063

tokenizer_config.json

@@ -0,0 +1,210 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "max_num_visual_tokens": 1280,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "processor_class": "ColQwen2_5_Processor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

training_config.yml

@@ -0,0 +1,75 @@
+config:
+  (): colpali_engine.trainer.colmodel_training.ColModelTrainingConfig
+  output_dir: !path ../../../models/colqwen2_5_train_single_source_3ep_r32_512bs_vdr_7b_1e4
+  processor:
+    (): colpali_engine.utils.transformers_wrappers.AllPurposeWrapper
+    class_to_instanciate: !ext colpali_engine.models.ColQwen2_5_Processor
+    pretrained_model_name_or_path:  "Qwen/Qwen2.5-VL-7B-Instruct" # "./models/paligemma-3b-mix-448"
+    max_num_visual_tokens: 1280
+    # num_image_tokens: 2048
+    # max_length: 50
+
+  model:
+    (): colpali_engine.utils.transformers_wrappers.AllPurposeWrapper
+    class_to_instanciate: !ext colpali_engine.models.ColQwen2_5
+    pretrained_model_name_or_path: "Qwen/Qwen2.5-VL-7B-Instruct"
+    torch_dtype:  !ext torch.bfloat16
+    use_cache: false
+    attn_implementation: "flash_attention_2"
+#    device_map: "auto"
+#    quantization_config:
+#      (): transformers.BitsAndBytesConfig
+#      load_in_4bit: true
+#      bnb_4bit_quant_type: "nf4"
+#      bnb_4bit_compute_dtype:  "bfloat16"
+#      bnb_4bit_use_double_quant: true
+
+  dataset_loading_func: !ext colpali_engine.utils.dataset_transformation.load_train_set_colpali_vdr
+  eval_dataset_loader: !import ../data/test_data.yaml
+
+  # max_length: 50
+  run_eval: true
+  loss_func:
+    (): colpali_engine.loss.late_interaction_losses.ColbertPairwiseCELoss
+  tr_args:
+    (): transformers.training_args.TrainingArguments
+    output_dir: null
+    overwrite_output_dir: true
+    num_train_epochs: 3 
+    per_device_train_batch_size: 512
+    ddp_find_unused_parameters: false
+    gradient_checkpointing: true
+    gradient_checkpointing_kwargs: { "use_reentrant": false }
+    # gradient_checkpointing: true
+    # 6 x 8 gpus = 48 batch size
+    # gradient_accumulation_steps: 4
+    per_device_eval_batch_size: 16
+    eval_strategy: "no"
+    dataloader_num_workers: 0 
+    # bf16: true
+    accelerator_config:
+      split_batches: true
+    save_steps: 500
+    logging_steps: 10
+    eval_steps: 100
+    warmup_ratio: 0.01
+    learning_rate: 1e-4
+    save_total_limit: 1
+    # resume_from_checkpoint: true
+    optim: "paged_adamw_8bit"
+    # wandb logging
+    # wandb_project: "mllm"
+    run_name: "colqwen2_5_train_single_source_3ep_r32_512bs_vdr_7b_1e4"
+    report_to: "wandb"
+
+
+  peft_config:
+    (): peft.LoraConfig
+    r: 32
+    lora_alpha: 32
+    lora_dropout: 0.1
+    init_lora_weights: "gaussian"
+    bias: "none"
+    task_type: "FEATURE_EXTRACTION"
+    target_modules: '(.*(model).*(down_proj|gate_proj|up_proj|k_proj|q_proj|v_proj|o_proj).*$|.*(custom_text_proj).*$)'
+    # target_modules: '(.*(language_model).*(down_proj|gate_proj|up_proj|k_proj|q_proj|v_proj|o_proj).*$|.*(custom_text_proj).*$)'