abdulmunimjemal/xlm-r-retrieval-am-v1

SentenceTransformer Fine-Tuned for Amharic Retrieval

This model is a sentence-transformers model finetuned on Amharic QA triplets. It maps sentences and paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

• Model Type: Sentence Transformer
• Base Model: sentence-transformers/paraphrase-xlm-r-multilingual-v1
• Training Task: Triplet loss with Matryoshka loss
• Language: Amharic
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Training Overview

• Training Data: Custom Amharic QA triplets (with positive and negative examples)
• Training Strategy:
  The model was finetuned using a combination of triplet loss and a Matryoshka loss, with evaluation performed using a TripletEvaluator.
• Hyperparameters:
  • Epochs: 3
  • Batch Size: 16
  • Learning Rate: 1e-6
  • Warmup Ratio: 0.08
  • Weight Decay: 0.05

Evaluation

The model was evaluated on a held-out test set using cosine similarity as the metric:

Metric	Value
Cosine Accuracy	0.875

Usage

To use the model in your own project:

1. Install Sentence Transformers:

   pip install -U sentence-transformers
   

2. Load the Model:

   from sentence_transformers import SentenceTransformer
   
   model = SentenceTransformer("abdulmunimjemal/xlm-r-retrieval-am-v5")
   sentences = [
       "ሰማይ ምን አይነት ቀለም ነው?",
       "ሰማይ ሰማያዊ ቀለም አለው።" ,
       "እኔ ምሳ እንጀራ በላሁ።" ,
       "ባሕር ምን አይነት ቀለም ነው?",
       "አየር በምድር ዙሪያ ያለ ነው።"
   ]
   embeddings = model.encode(sentences)
   print(embeddings.shape)  # Expected output: (5, 768)
   

3. Compute Similarity:

   from sklearn.metrics.pairwise import cosine_similarity
   similarities = cosine_similarity(embeddings, embeddings)
   print(similarities.shape)  # Expected output: (5, 5)
   

Model Architecture

Below is an outline of the model architecture:

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_mean_tokens': True, ...})
)

Training Environment

• Python: 3.11.11
• Sentence Transformers: 3.3.1
• Transformers: 4.47.1
• PyTorch: 2.5.1+cu124
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

If you use this model in your research, please cite it appropriately.

@misc{your_model,
  title = {SentenceTransformer Fine-Tuned for Amharic Retrieval},
  author = {Abdulmunim J. Jemal},
  year = {2025},
  howpublished = {Hugging Face Model Hub, \url{https://huggingface.co/abdulmunimjemal/xlm-r-retrieval-am-v1}}
}