JLee0/rag-embedder-staria-50epochs

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    '하이패스 시스템의 등록을 정상적으로 사용하기 위해 차량과 함께 준비해야 할 것은 무엇인가요?',
    '하이패스 시스템 명의 이전 및 등록 관련 주의사항: • 하이패스 시스템이 장착된 차량을 타인에게 매각 또는 양도하는 경우, 차량의 명의와 함께 하이패스 시스템의 명의를 함께 이전하십시오. 명의 변경이 불가한 경우 하이패스 시스템의 등록을 말소해야 합니다. • 하이패스 시스템이 장착된 차량을 타인에게 매수 또는 양수 받은 경우, 하이패스 시스템의 등록명의를 함께 이전 받아야만 정상적으로 하이패스 기능을 사용할 수 있습니다. 차량과 구비 서류를 준비하여 인근에 위치한 당사 직영 하이테크센터나 블루핸즈에 직접 방문하여 등록하십시오.',
    '시동을 유지하려면 디지털 키를 무선 충전기에 올려놓으십시오: 원격 시동 상태에서 디지털 키로 도어를 잠금 해제하여 도어를 열면 이 경고문이 최대 30초 동안 표시됩니다. 주행 가능한 시동 상태로 전환하려면 경고문이 표시되었을 때 등록된 디지털 키를 무선 충전기 위에 올려놓으십시오.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0266
cosine_accuracy@3	0.6588
cosine_accuracy@5	0.8028
cosine_accuracy@10	0.9132
cosine_precision@1	0.0266
cosine_precision@3	0.2196
cosine_precision@5	0.1606
cosine_precision@10	0.0913
cosine_recall@1	0.0266
cosine_recall@3	0.6588
cosine_recall@5	0.8028
cosine_recall@10	0.9132
cosine_ndcg@10	0.5232
cosine_mrr@10	0.3932
cosine_map@100	0.397

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,634 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 10 tokens mean: 23.78 tokens max: 47 tokens	min: 14 tokens mean: 209.53 tokens max: 2889 tokens

• Samples:

  sentence_0	sentence_1
  동승석 안전벨트 경고등이 켜지지 않을 수 있는 이유는 무엇인가요?	알아두기: 사양에 따라 1열 중앙 좌석이 있는 경우에는 동승석과 동일하게 안전벨트 미착용 경고등 및 경고음이 작동합니다. 경고: 바르지 않은 자세로 앉으면 동승석 안전벨트 경고등이 켜지지 않을 수 있습니다. 안전벨트는 바로 앉은 상태에서 올바른 사용법에 따라 착용하십시오.
  품질이나 성능이 부적합한 카 매트를 사용할 경우 발생할 수 있는 문제는 무엇인가요?	카 매트 후크 주의사항: • 품질이나 성능이 부적합한 카 매트를 사용할 경우에는 사고가 발생할 수 있습니다. 순정 카 매트는 품질과 성능을 당사가 보증하는 부품입니다.
  차량 내 경고음이 작동하지 않을 수 있는 경우는 어떤 상황입니까?	주의: 차량 내 모든 경고음(웰컴/굿바이/능동 소음 제어 포함)은 외장 앰프를 통해 나옵니다. 앰프를 비순정 부품으로 교체하거나 스피커를 비순정 부품으로 교체해서 앰프가 고장이 나면 경고음이 작동하지 않을 수 있으므로 주의하십시오.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 5
• per_device_eval_batch_size: 5
• num_train_epochs: 40
• fp16: True
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 5
• per_device_eval_batch_size: 5
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 40
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Click to expand

Epoch	Step	Training Loss	cosine_ndcg@10
0.1529	50	-	0.5309
0.3058	100	-	0.5310
0.4587	150	-	0.5269
0.6116	200	-	0.5287
0.7645	250	-	0.5292
0.9174	300	-	0.5306
1.0	327	-	0.5279
1.0703	350	-	0.5316
1.2232	400	-	0.5282
1.3761	450	-	0.5199
1.5291	500	0.033	0.5168
1.6820	550	-	0.5201
1.8349	600	-	0.5204
1.9878	650	-	0.5200
2.0	654	-	0.5197
2.1407	700	-	0.5158
2.2936	750	-	0.5261
2.4465	800	-	0.5228
2.5994	850	-	0.5260
2.7523	900	-	0.5241
2.9052	950	-	0.5297
3.0	981	-	0.5337
3.0581	1000	0.0112	0.5299
3.2110	1050	-	0.5155
3.3639	1100	-	0.5189
3.5168	1150	-	0.5021
3.6697	1200	-	0.5119
3.8226	1250	-	0.5092
3.9755	1300	-	0.4996
4.0	1308	-	0.4971
4.1284	1350	-	0.4954
4.2813	1400	-	0.5005
4.4343	1450	-	0.5020
4.5872	1500	0.0142	0.4960
4.7401	1550	-	0.5039
4.8930	1600	-	0.5008
5.0	1635	-	0.5108
5.0459	1650	-	0.5141
5.1988	1700	-	0.5194
5.3517	1750	-	0.5014
5.5046	1800	-	0.5108
5.6575	1850	-	0.5081
5.8104	1900	-	0.5112
5.9633	1950	-	0.5045
6.0	1962	-	0.5082
6.1162	2000	0.0107	0.5056
6.2691	2050	-	0.5015
6.4220	2100	-	0.5111
6.5749	2150	-	0.5107
6.7278	2200	-	0.4879
6.8807	2250	-	0.4988
7.0	2289	-	0.5059
7.0336	2300	-	0.5168
7.1865	2350	-	0.5137
7.3394	2400	-	0.5200
7.4924	2450	-	0.5094
7.6453	2500	0.0104	0.5103
7.7982	2550	-	0.5047
7.9511	2600	-	0.5146
8.0	2616	-	0.5085
8.1040	2650	-	0.5026
8.2569	2700	-	0.5105
8.4098	2750	-	0.5144
8.5627	2800	-	0.5079
8.7156	2850	-	0.5022
8.8685	2900	-	0.5129
9.0	2943	-	0.5049
9.0214	2950	-	0.5041
9.1743	3000	0.0068	0.5044
9.3272	3050	-	0.5096
9.4801	3100	-	0.5098
9.6330	3150	-	0.5138
9.7859	3200	-	0.5119
9.9388	3250	-	0.5079
10.0	3270	-	0.5134
10.0917	3300	-	0.5120
10.2446	3350	-	0.5132
10.3976	3400	-	0.5086
10.5505	3450	-	0.5116
10.7034	3500	0.0069	0.5156
10.8563	3550	-	0.5181
11.0	3597	-	0.5006
11.0092	3600	-	0.5006
11.1621	3650	-	0.5111
11.3150	3700	-	0.5102
11.4679	3750	-	0.5012
11.6208	3800	-	0.4982
11.7737	3850	-	0.4949
11.9266	3900	-	0.4942
12.0	3924	-	0.5053
12.0795	3950	-	0.5015
12.2324	4000	0.0076	0.4873
12.3853	4050	-	0.5006
12.5382	4100	-	0.5102
12.6911	4150	-	0.5045
12.8440	4200	-	0.5018
12.9969	4250	-	0.4979
13.0	4251	-	0.5000
13.1498	4300	-	0.5001
13.3028	4350	-	0.4987
13.4557	4400	-	0.5085
13.6086	4450	-	0.4979
13.7615	4500	0.0082	0.4916
13.9144	4550	-	0.4975
14.0	4578	-	0.4967
14.0673	4600	-	0.5011
14.2202	4650	-	0.5179
14.3731	4700	-	0.5173
14.5260	4750	-	0.5172
14.6789	4800	-	0.5135
14.8318	4850	-	0.5109
14.9847	4900	-	0.5044
15.0	4905	-	0.5056
15.1376	4950	-	0.5098
15.2905	5000	0.0029	0.5101
15.4434	5050	-	0.5065
15.5963	5100	-	0.5045
15.7492	5150	-	0.5049
15.9021	5200	-	0.5109
16.0	5232	-	0.5100
16.0550	5250	-	0.5106
16.2080	5300	-	0.5080
16.3609	5350	-	0.5000
16.5138	5400	-	0.5090
16.6667	5450	-	0.5068
16.8196	5500	0.0076	0.5151
16.9725	5550	-	0.5110
17.0	5559	-	0.5091
17.1254	5600	-	0.5075
17.2783	5650	-	0.5164
17.4312	5700	-	0.5155
17.5841	5750	-	0.5140
17.7370	5800	-	0.5140
17.8899	5850	-	0.5176
18.0	5886	-	0.5170
18.0428	5900	-	0.5164
18.1957	5950	-	0.5121
18.3486	6000	0.0046	0.4981
18.5015	6050	-	0.4994
18.6544	6100	-	0.5125
18.8073	6150	-	0.5011
18.9602	6200	-	0.5043
19.0	6213	-	0.5095
19.1131	6250	-	0.5082
19.2661	6300	-	0.5100
19.4190	6350	-	0.5030
19.5719	6400	-	0.5088
19.7248	6450	-	0.5086
19.8777	6500	0.0064	0.5083
20.0	6540	-	0.5082
20.0306	6550	-	0.5100
20.1835	6600	-	0.5111
20.3364	6650	-	0.5127
20.4893	6700	-	0.5078
20.6422	6750	-	0.5096
20.7951	6800	-	0.5182
20.9480	6850	-	0.5188
21.0	6867	-	0.5197
21.1009	6900	-	0.5162
21.2538	6950	-	0.5135
21.4067	7000	0.0091	0.5099
21.5596	7050	-	0.5080
21.7125	7100	-	0.5091
21.8654	7150	-	0.5087
22.0	7194	-	0.5146
22.0183	7200	-	0.5152
22.1713	7250	-	0.5117
22.3242	7300	-	0.5157
22.4771	7350	-	0.5158
22.6300	7400	-	0.5065
22.7829	7450	-	0.5175
22.9358	7500	0.0044	0.5111
23.0	7521	-	0.5102
23.0887	7550	-	0.5136
23.2416	7600	-	0.5104
23.3945	7650	-	0.5144
23.5474	7700	-	0.5187
23.7003	7750	-	0.5138
23.8532	7800	-	0.5115
24.0	7848	-	0.5175
24.0061	7850	-	0.5178
24.1590	7900	-	0.5116
24.3119	7950	-	0.5130
24.4648	8000	0.0045	0.5190
24.6177	8050	-	0.5199
24.7706	8100	-	0.5168
24.9235	8150	-	0.5154
25.0	8175	-	0.5156
25.0765	8200	-	0.5146
25.2294	8250	-	0.5131
25.3823	8300	-	0.5100
25.5352	8350	-	0.5164
25.6881	8400	-	0.5185
25.8410	8450	-	0.5190
25.9939	8500	0.0051	0.5191
26.0	8502	-	0.5186
26.1468	8550	-	0.5163
26.2997	8600	-	0.5168
26.4526	8650	-	0.5172
26.6055	8700	-	0.5180
26.7584	8750	-	0.5144
26.9113	8800	-	0.5087
27.0	8829	-	0.5038
27.0642	8850	-	0.5074
27.2171	8900	-	0.5238
27.3700	8950	-	0.5232
27.5229	9000	0.0029	0.5237
27.6758	9050	-	0.5240
27.8287	9100	-	0.5214
27.9817	9150	-	0.5222
28.0	9156	-	0.5228
28.1346	9200	-	0.5218
28.2875	9250	-	0.5210
28.4404	9300	-	0.5218
28.5933	9350	-	0.5201
28.7462	9400	-	0.5197
28.8991	9450	-	0.5203
29.0	9483	-	0.5185
29.0520	9500	0.003	0.5135
29.2049	9550	-	0.5164
29.3578	9600	-	0.5161
29.5107	9650	-	0.5152
29.6636	9700	-	0.5153
29.8165	9750	-	0.5122
29.9694	9800	-	0.5134
30.0	9810	-	0.5165
30.1223	9850	-	0.5177
30.2752	9900	-	0.5187
30.4281	9950	-	0.5175
30.5810	10000	0.0027	0.5126
30.7339	10050	-	0.5132
30.8869	10100	-	0.5112
31.0	10137	-	0.5069
31.0398	10150	-	0.5049
31.1927	10200	-	0.5036
31.3456	10250	-	0.5056
31.4985	10300	-	0.5102
31.6514	10350	-	0.5108
31.8043	10400	-	0.5120
31.9572	10450	-	0.5116
32.0	10464	-	0.5118
32.1101	10500	0.0022	0.5144
32.2630	10550	-	0.5165
32.4159	10600	-	0.5151
32.5688	10650	-	0.5154
32.7217	10700	-	0.5161
32.8746	10750	-	0.5168
33.0	10791	-	0.5166
33.0275	10800	-	0.5166
33.1804	10850	-	0.5189
33.3333	10900	-	0.5197
33.4862	10950	-	0.5184
33.6391	11000	0.0029	0.5195
33.7920	11050	-	0.5215
33.9450	11100	-	0.5193
34.0	11118	-	0.5205
34.0979	11150	-	0.5197
34.2508	11200	-	0.5217
34.4037	11250	-	0.5208
34.5566	11300	-	0.5245
34.7095	11350	-	0.5237
34.8624	11400	-	0.5218
35.0	11445	-	0.5225
35.0153	11450	-	0.5224
35.1682	11500	0.004	0.5219
35.3211	11550	-	0.5213
35.4740	11600	-	0.5222
35.6269	11650	-	0.5232
35.7798	11700	-	0.5242
35.9327	11750	-	0.5226
36.0	11772	-	0.5218
36.0856	11800	-	0.5215
36.2385	11850	-	0.5232
36.3914	11900	-	0.5232
36.5443	11950	-	0.5252
36.6972	12000	0.0038	0.5239
36.8502	12050	-	0.5241
37.0	12099	-	0.5219
37.0031	12100	-	0.5220
37.1560	12150	-	0.5222
37.3089	12200	-	0.5224
37.4618	12250	-	0.5224
37.6147	12300	-	0.5234
37.7676	12350	-	0.5217
37.9205	12400	-	0.5229
38.0	12426	-	0.5221
38.0734	12450	-	0.5230
38.2263	12500	0.0032	0.5236
38.3792	12550	-	0.5229
38.5321	12600	-	0.5229
38.6850	12650	-	0.5234
38.8379	12700	-	0.5226
38.9908	12750	-	0.5232
39.0	12753	-	0.5230
39.1437	12800	-	0.5232
39.2966	12850	-	0.5233
39.4495	12900	-	0.5236
39.6024	12950	-	0.5230
39.7554	13000	0.0015	0.5234
39.9083	13050	-	0.5232
40.0	13080	-	0.5232

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 4.0.1
• Transformers: 4.50.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}