iddqd21/fine-tuned-e5-semantic-similarity_lowercase

SentenceTransformer based on intfloat/multilingual-e5-base

This is a sentence-transformers model finetuned from intfloat/multilingual-e5-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: intfloat/multilingual-e5-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("iddqd21/fine-tuned-e5-semantic-similarity_lowercase")
# Run inference
sentences = [
    'homo-gamma linolenate',
    'alpha linolenate',
    'complement factor b',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 109,998 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 3 tokens mean: 10.8 tokens max: 33 tokens	min: 3 tokens mean: 9.36 tokens max: 27 tokens	min: 0.0 mean: 0.44 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  雄烯二酮	androstenedione	1.0
  follitropin^30dk gonadotropin salıcı hormon dozu	follitropin^30m post dose gonadotropin releasing hormone	1.0
  ch50 serpl-mcnc	complement total hemolytic ch50	1.0

• Loss: CosineSimilarityLoss with these parameters:

  {
      "loss_fct": "torch.nn.modules.loss.MSELoss"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 5
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss
0.0727	500	0.1541
0.1455	1000	0.1012
0.2182	1500	0.0972
0.2909	2000	0.0862
0.3636	2500	0.0838
0.4364	3000	0.0818
0.5091	3500	0.0804
0.5818	4000	0.0746
0.6545	4500	0.0726
0.7273	5000	0.0693
0.8	5500	0.0688
0.8727	6000	0.0696
0.9455	6500	0.0661
1.0182	7000	0.0619
1.0909	7500	0.0537
1.1636	8000	0.0536
1.2364	8500	0.0537
1.3091	9000	0.0548
1.3818	9500	0.0526
1.4545	10000	0.0526
1.5273	10500	0.0489
1.6	11000	0.0501
1.6727	11500	0.0498
1.7455	12000	0.0487
1.8182	12500	0.0454
1.8909	13000	0.0454
1.9636	13500	0.0455
2.0364	14000	0.0426
2.1091	14500	0.0382
2.1818	15000	0.039
2.2545	15500	0.0368
2.3273	16000	0.0397
2.4	16500	0.0355
2.4727	17000	0.035
2.5455	17500	0.036
2.6182	18000	0.0348
2.6909	18500	0.0384
2.7636	19000	0.0338
2.8364	19500	0.0331
2.9091	20000	0.0345
2.9818	20500	0.0337
3.0545	21000	0.0293
3.1273	21500	0.0274
3.2	22000	0.0284
3.2727	22500	0.028
3.3455	23000	0.0275
3.4182	23500	0.03
3.4909	24000	0.027
3.5636	24500	0.0279
3.6364	25000	0.0285
3.7091	25500	0.0288
3.7818	26000	0.0263
3.8545	26500	0.0289
3.9273	27000	0.0271
4.0	27500	0.0268
4.0727	28000	0.0234
4.1455	28500	0.0228
4.2182	29000	0.0235
4.2909	29500	0.024
4.3636	30000	0.0234
4.4364	30500	0.0235
4.5091	31000	0.0233
4.5818	31500	0.0239
4.6545	32000	0.0228
4.7273	32500	0.0236
4.8	33000	0.0236
4.8727	33500	0.0223
4.9455	34000	0.022

Framework Versions

• Python: 3.9.20
• Sentence Transformers: 3.3.1
• Transformers: 4.47.1
• PyTorch: 2.5.1+rocm6.2
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}