Hugging Face's logo

am-azadi
/
KaLM-embedding-multilingual-mini-v1_Fine_Tuned_3e

Sentence Similarity
sentence-transformers
Safetensors
qwen2
feature-extraction
Generated from Trainer
dataset_size:21769
loss:MultipleNegativesRankingLoss
text-embeddings-inference
Inference Endpoints
Model card Files Community

SentenceTransformer based on am-azadi/KaLM-embedding-multilingual-mini-v1_Fine_Tuned_1e

This is a sentence-transformers model finetuned from am-azadi/KaLM-embedding-multilingual-mini-v1_Fine_Tuned_1e. It maps sentences & paragraphs to a 896-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: Qwen2Model 
  (1): Pooling({'word_embedding_dimension': 896, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'The moment of the death of President Mohamed Morsi, may God have mercy on him, God willing ',
    'The moment of the death of President Mohamed Morsi This video belongs to the trial of those accused of the Port Said events and does not show the moment of the death of former Egyptian President Mohamed Morsi',
    'Cuba has Interferon Alpha 2B, the cure, the vaccine against the coronavirus The Cuban antiviral Interferon Alfa 2B is used in China to treat patients with the new coronavirus, but it is neither a vaccine nor a cure',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 896]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

  • Size: 21,769 training samples
  • Columns: sentence_0 and sentence_1
  • Approximate statistics based on the first 1000 samples:
    sentence_0 sentence_1
    type string string
    details
    • min: 4 tokens
    • mean: 109.9 tokens
    • max: 512 tokens
    • min: 14 tokens
    • mean: 34.57 tokens
    • max: 132 tokens
  • Samples:
    sentence_0 sentence_1
    Sad, and we are hostages!! If that doesn't make you think about the “measures” that governors and mayors are taking there is nothing that can be made in Brazil Action by a military police officer against a street vendor amid restrictive measures due to the covid-19 pandemic The photo in which a PM seizes products from a street vendor is from 2016, unrelated to the pandemic
    This is why it's important to know your history d4 Rare photo of Queen Elizabeth II and Prince Phillip bowing before the real original African Royalty, Empress Menen Asfaw and her husband Emperor Ras Tafari Makonnen Woldemikael Haile Selassie I of Ethiopia... Queen Elizabeth II bows before Ethiopian Emperor Haile Selassie British monarch's first visit to Ethiopia came 10 years after this photo was taken
    Public Reaction on Hyderabad Priyanka Reddy Case Drabad Encounter . Common people say Photo of suspects killed by police in Hyderabad rape-murder case This photo has circulated online since at least 2015 in connection with an unrelated case in a different Indian state
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 2
  • per_device_eval_batch_size: 2
  • num_train_epochs: 1
  • multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: no
  • prediction_loss_only: True
  • per_device_train_batch_size: 2
  • per_device_eval_batch_size: 2
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1
  • num_train_epochs: 1
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: round_robin

Training Logs

Epoch Step Training Loss
0.0459 500 0.0083
0.0919 1000 0.019
0.1378 1500 0.0255
0.1837 2000 0.0372
0.2297 2500 0.0315
0.2756 3000 0.0258
0.3215 3500 0.0211
0.3675 4000 0.0187
0.4134 4500 0.0264
0.4593 5000 0.0348
0.5053 5500 0.0197
0.5512 6000 0.0102
0.5972 6500 0.0092
0.6431 7000 0.0169
0.6890 7500 0.0109
0.7350 8000 0.0115
0.7809 8500 0.0173
0.8268 9000 0.0196
0.8728 9500 0.028
0.9187 10000 0.0218
0.9646 10500 0.0169
0.0459 500 0.004
0.0919 1000 0.02
0.1378 1500 0.0154
0.1837 2000 0.0141
0.2297 2500 0.014
0.2756 3000 0.0077
0.3215 3500 0.018
0.3675 4000 0.0079
0.4134 4500 0.0238
0.4593 5000 0.0183
0.5053 5500 0.0159
0.5512 6000 0.0043
0.5972 6500 0.0066
0.6431 7000 0.0068
0.6890 7500 0.0035
0.7350 8000 0.0042
0.7809 8500 0.0084
0.8268 9000 0.0049
0.8728 9500 0.0102
0.9187 10000 0.0048
0.9646 10500 0.0045

Framework Versions

  • Python: 3.11.11
  • Sentence Transformers: 3.4.1
  • Transformers: 4.48.3
  • PyTorch: 2.5.1+cu124
  • Accelerate: 1.3.0
  • Datasets: 3.3.2
  • Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss 

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
15
Safetensors
Model size
494M params
Tensor type
F32
·
Inference Providers NEW
Sentence Similarity
This model is not currently available via any of the supported Inference Providers.

Model tree for am-azadi/KaLM-embedding-multilingual-mini-v1_Fine_Tuned_3e

Base model

HIT-TMG/KaLM-embedding-multilingual-mini-v1
Finetuned
am-azadi/KaLM-embedding-multilingual-mini-v1_Fine_Tuned_1e
Finetuned
(2)
this model