metadata
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- mteb
model-index:
- name: sentence_croissant_alpha_v0.2
results:
- task:
type: Clustering
dataset:
type: lyon-nlp/alloprof
name: MTEB AlloProfClusteringP2P
config: default
split: test
revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b
metrics:
- type: v_measure
value: 59.14629497199997
- task:
type: Clustering
dataset:
type: lyon-nlp/alloprof
name: MTEB AlloProfClusteringS2S
config: default
split: test
revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b
metrics:
- type: v_measure
value: 36.450870830351036
- task:
type: Reranking
dataset:
type: lyon-nlp/mteb-fr-reranking-alloprof-s2p
name: MTEB AlloprofReranking
config: default
split: test
revision: e40c8a63ce02da43200eccb5b0846fcaa888f562
metrics:
- type: map
value: 67.23549444979429
- type: mrr
value: 68.49382830276612
- task:
type: Retrieval
dataset:
type: lyon-nlp/alloprof
name: MTEB AlloprofRetrieval
config: default
split: test
revision: 2df7bee4080bedf2e97de3da6bd5c7bc9fc9c4d2
metrics:
- type: map_at_1
value: 30.285
- type: map_at_10
value: 41.724
- type: map_at_100
value: 42.696
- type: map_at_1000
value: 42.739
- type: map_at_3
value: 38.68
- type: map_at_5
value: 40.474
- type: mrr_at_1
value: 30.285
- type: mrr_at_10
value: 41.724
- type: mrr_at_100
value: 42.696
- type: mrr_at_1000
value: 42.739
- type: mrr_at_3
value: 38.68
- type: mrr_at_5
value: 40.474
- type: ndcg_at_1
value: 30.285
- type: ndcg_at_10
value: 47.687000000000005
- type: ndcg_at_100
value: 52.580000000000005
- type: ndcg_at_1000
value: 53.738
- type: ndcg_at_3
value: 41.439
- type: ndcg_at_5
value: 44.67
- type: precision_at_1
value: 30.285
- type: precision_at_10
value: 6.657
- type: precision_at_100
value: 0.898
- type: precision_at_1000
value: 0.099
- type: precision_at_3
value: 16.477
- type: precision_at_5
value: 11.454
- type: recall_at_1
value: 30.285
- type: recall_at_10
value: 66.572
- type: recall_at_100
value: 89.819
- type: recall_at_1000
value: 98.955
- type: recall_at_3
value: 49.43
- type: recall_at_5
value: 57.27100000000001
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (fr)
config: fr
split: test
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
metrics:
- type: accuracy
value: 36.484
- type: f1
value: 36.358267416839176
- task:
type: Retrieval
dataset:
type: maastrichtlawtech/bsard
name: MTEB BSARDRetrieval
config: default
split: test
revision: 5effa1b9b5fa3b0f9e12523e6e43e5f86a6e6d59
metrics:
- type: map_at_1
value: 0.44999999999999996
- type: map_at_10
value: 1.184
- type: map_at_100
value: 1.5939999999999999
- type: map_at_1000
value: 1.6680000000000001
- type: map_at_3
value: 0.901
- type: map_at_5
value: 1.014
- type: mrr_at_1
value: 0.44999999999999996
- type: mrr_at_10
value: 1.184
- type: mrr_at_100
value: 1.5939999999999999
- type: mrr_at_1000
value: 1.6680000000000001
- type: mrr_at_3
value: 0.901
- type: mrr_at_5
value: 1.014
- type: ndcg_at_1
value: 0.44999999999999996
- type: ndcg_at_10
value: 1.746
- type: ndcg_at_100
value: 4.271
- type: ndcg_at_1000
value: 6.662
- type: ndcg_at_3
value: 1.126
- type: ndcg_at_5
value: 1.32
- type: precision_at_1
value: 0.44999999999999996
- type: precision_at_10
value: 0.36
- type: precision_at_100
value: 0.167
- type: precision_at_1000
value: 0.036000000000000004
- type: precision_at_3
value: 0.601
- type: precision_at_5
value: 0.44999999999999996
- type: recall_at_1
value: 0.44999999999999996
- type: recall_at_10
value: 3.604
- type: recall_at_100
value: 16.667
- type: recall_at_1000
value: 36.486000000000004
- type: recall_at_3
value: 1.802
- type: recall_at_5
value: 2.252
- task:
type: Clustering
dataset:
type: lyon-nlp/clustering-hal-s2s
name: MTEB HALClusteringS2S
config: default
split: test
revision: e06ebbbb123f8144bef1a5d18796f3dec9ae2915
metrics:
- type: v_measure
value: 24.970553942854256
- task:
type: Clustering
dataset:
type: mlsum
name: MTEB MLSUMClusteringP2P
config: default
split: test
revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7
metrics:
- type: v_measure
value: 42.48794423025542
- task:
type: Clustering
dataset:
type: mlsum
name: MTEB MLSUMClusteringS2S
config: default
split: test
revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7
metrics:
- type: v_measure
value: 34.44830504100088
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (fr)
config: fr
split: test
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
metrics:
- type: accuracy
value: 88.96335734419041
- type: f1
value: 88.77543132157024
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (fr)
config: fr
split: test
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
metrics:
- type: accuracy
value: 67.76072658941435
- type: f1
value: 47.98533031010631
- task:
type: Classification
dataset:
type: masakhane/masakhanews
name: MTEB MasakhaNEWSClassification (fra)
config: fra
split: test
revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60
metrics:
- type: accuracy
value: 73.17535545023696
- type: f1
value: 69.07397342867827
- task:
type: Clustering
dataset:
type: masakhane/masakhanews
name: MTEB MasakhaNEWSClusteringP2P (fra)
config: fra
split: test
revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60
metrics:
- type: v_measure
value: 47.584542055968335
- task:
type: Clustering
dataset:
type: masakhane/masakhanews
name: MTEB MasakhaNEWSClusteringS2S (fra)
config: fra
split: test
revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60
metrics:
- type: v_measure
value: 33.58141573894578
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (fr)
config: fr
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 66.29791526563551
- type: f1
value: 64.11383858035595
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (fr)
config: fr
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 71.7014122394082
- type: f1
value: 71.28396788755553
- task:
type: Retrieval
dataset:
type: jinaai/mintakaqa
name: MTEB MintakaRetrieval (fr)
config: fr
split: test
revision: efa78cc2f74bbcd21eff2261f9e13aebe40b814e
metrics:
- type: map_at_1
value: 14.824000000000002
- type: map_at_10
value: 23.217
- type: map_at_100
value: 24.484
- type: map_at_1000
value: 24.571
- type: map_at_3
value: 20.762
- type: map_at_5
value: 22.121
- type: mrr_at_1
value: 14.824000000000002
- type: mrr_at_10
value: 23.217
- type: mrr_at_100
value: 24.484
- type: mrr_at_1000
value: 24.571
- type: mrr_at_3
value: 20.762
- type: mrr_at_5
value: 22.121
- type: ndcg_at_1
value: 14.824000000000002
- type: ndcg_at_10
value: 27.876
- type: ndcg_at_100
value: 34.53
- type: ndcg_at_1000
value: 37.153999999999996
- type: ndcg_at_3
value: 22.746
- type: ndcg_at_5
value: 25.192999999999998
- type: precision_at_1
value: 14.824000000000002
- type: precision_at_10
value: 4.279
- type: precision_at_100
value: 0.75
- type: precision_at_1000
value: 0.096
- type: precision_at_3
value: 9.5
- type: precision_at_5
value: 6.888
- type: recall_at_1
value: 14.824000000000002
- type: recall_at_10
value: 42.793
- type: recall_at_100
value: 75.02
- type: recall_at_1000
value: 96.274
- type: recall_at_3
value: 28.500999999999998
- type: recall_at_5
value: 34.439
- task:
type: PairClassification
dataset:
type: GEM/opusparcus
name: MTEB OpusparcusPC (fr)
config: fr
split: test
revision: 9e9b1f8ef51616073f47f306f7f47dd91663f86a
metrics:
- type: cos_sim_accuracy
value: 82.56130790190736
- type: cos_sim_ap
value: 93.47537508242819
- type: cos_sim_f1
value: 87.60250844187169
- type: cos_sim_precision
value: 85.17823639774859
- type: cos_sim_recall
value: 90.16881827209534
- type: dot_accuracy
value: 81.06267029972753
- type: dot_ap
value: 91.67254760894009
- type: dot_f1
value: 87.07172224760164
- type: dot_precision
value: 80.62605752961083
- type: dot_recall
value: 94.63753723932473
- type: euclidean_accuracy
value: 81.19891008174388
- type: euclidean_ap
value: 93.11746326702661
- type: euclidean_f1
value: 86.52278177458035
- type: euclidean_precision
value: 83.6734693877551
- type: euclidean_recall
value: 89.57298907646475
- type: manhattan_accuracy
value: 81.06267029972753
- type: manhattan_ap
value: 93.10511956552851
- type: manhattan_f1
value: 86.62175168431185
- type: manhattan_precision
value: 84.03361344537815
- type: manhattan_recall
value: 89.37437934458788
- type: max_accuracy
value: 82.56130790190736
- type: max_ap
value: 93.47537508242819
- type: max_f1
value: 87.60250844187169
- task:
type: PairClassification
dataset:
type: paws-x
name: MTEB PawsX (fr)
config: fr
split: test
revision: 8a04d940a42cd40658986fdd8e3da561533a3646
metrics:
- type: cos_sim_accuracy
value: 64.7
- type: cos_sim_ap
value: 66.97936856243149
- type: cos_sim_f1
value: 64.10698878343399
- type: cos_sim_precision
value: 52.50883392226149
- type: cos_sim_recall
value: 82.281284606866
- type: dot_accuracy
value: 55.7
- type: dot_ap
value: 49.248259184437195
- type: dot_f1
value: 62.51298026998961
- type: dot_precision
value: 45.468277945619334
- type: dot_recall
value: 100
- type: euclidean_accuracy
value: 65.14999999999999
- type: euclidean_ap
value: 67.67376405881289
- type: euclidean_f1
value: 64.10034602076125
- type: euclidean_precision
value: 52.59048970901349
- type: euclidean_recall
value: 82.05980066445183
- type: manhattan_accuracy
value: 65.2
- type: manhattan_ap
value: 67.68415171194316
- type: manhattan_f1
value: 64.16899163013153
- type: manhattan_precision
value: 50.12453300124533
- type: manhattan_recall
value: 89.14728682170544
- type: max_accuracy
value: 65.2
- type: max_ap
value: 67.68415171194316
- type: max_f1
value: 64.16899163013153
- task:
type: STS
dataset:
type: Lajavaness/SICK-fr
name: MTEB SICKFr
config: default
split: test
revision: e077ab4cf4774a1e36d86d593b150422fafd8e8a
metrics:
- type: cos_sim_pearson
value: 77.68761269197373
- type: cos_sim_spearman
value: 69.66744624141576
- type: euclidean_pearson
value: 72.05200050489465
- type: euclidean_spearman
value: 68.04895470259305
- type: manhattan_pearson
value: 72.16693522711834
- type: manhattan_spearman
value: 68.12086601967899
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (fr)
config: fr
split: test
revision: eea2b4fe26a775864c896887d910b76a8098ad3f
metrics:
- type: cos_sim_pearson
value: 75.11874053715779
- type: cos_sim_spearman
value: 78.68085137779333
- type: euclidean_pearson
value: 68.83921367763453
- type: euclidean_spearman
value: 71.35148956255736
- type: manhattan_pearson
value: 69.46950072200525
- type: manhattan_spearman
value: 71.66493261411941
- task:
type: STS
dataset:
type: PhilipMay/stsb_multi_mt
name: MTEB STSBenchmarkMultilingualSTS (fr)
config: fr
split: test
revision: 93d57ef91790589e3ce9c365164337a8a78b7632
metrics:
- type: cos_sim_pearson
value: 78.09242108846412
- type: cos_sim_spearman
value: 76.38442769094321
- type: euclidean_pearson
value: 76.19649405196662
- type: euclidean_spearman
value: 75.95441973818816
- type: manhattan_pearson
value: 76.13548797312832
- type: manhattan_spearman
value: 75.93264073187262
- task:
type: Summarization
dataset:
type: lyon-nlp/summarization-summeval-fr-p2p
name: MTEB SummEvalFr
config: default
split: test
revision: b385812de6a9577b6f4d0f88c6a6e35395a94054
metrics:
- type: cos_sim_pearson
value: 30.511451950181858
- type: cos_sim_spearman
value: 30.267871792007288
- type: dot_pearson
value: 27.428950856263114
- type: dot_spearman
value: 26.895658072972395
- task:
type: Reranking
dataset:
type: lyon-nlp/mteb-fr-reranking-syntec-s2p
name: MTEB SyntecReranking
config: default
split: test
revision: b205c5084a0934ce8af14338bf03feb19499c84d
metrics:
- type: map
value: 83.16666666666667
- type: mrr
value: 83.16666666666667
- task:
type: Retrieval
dataset:
type: lyon-nlp/mteb-fr-retrieval-syntec-s2p
name: MTEB SyntecRetrieval
config: default
split: test
revision: aa460cd4d177e6a3c04fcd2affd95e8243289033
metrics:
- type: map_at_1
value: 61
- type: map_at_10
value: 71.863
- type: map_at_100
value: 72.115
- type: map_at_1000
value: 72.115
- type: map_at_3
value: 69
- type: map_at_5
value: 70.95
- type: mrr_at_1
value: 61
- type: mrr_at_10
value: 71.863
- type: mrr_at_100
value: 72.115
- type: mrr_at_1000
value: 72.115
- type: mrr_at_3
value: 69
- type: mrr_at_5
value: 70.95
- type: ndcg_at_1
value: 61
- type: ndcg_at_10
value: 77.666
- type: ndcg_at_100
value: 78.63900000000001
- type: ndcg_at_1000
value: 78.63900000000001
- type: ndcg_at_3
value: 71.809
- type: ndcg_at_5
value: 75.422
- type: precision_at_1
value: 61
- type: precision_at_10
value: 9.6
- type: precision_at_100
value: 1
- type: precision_at_1000
value: 0.1
- type: precision_at_3
value: 26.667
- type: precision_at_5
value: 17.8
- type: recall_at_1
value: 61
- type: recall_at_10
value: 96
- type: recall_at_100
value: 100
- type: recall_at_1000
value: 100
- type: recall_at_3
value: 80
- type: recall_at_5
value: 89
- task:
type: Retrieval
dataset:
type: jinaai/xpqa
name: MTEB XPQARetrieval (fr)
config: fr
split: test
revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f
metrics:
- type: map_at_1
value: 37.736999999999995
- type: map_at_10
value: 57.842000000000006
- type: map_at_100
value: 59.373
- type: map_at_1000
value: 59.426
- type: map_at_3
value: 51.598
- type: map_at_5
value: 55.279999999999994
- type: mrr_at_1
value: 59.68
- type: mrr_at_10
value: 66.71000000000001
- type: mrr_at_100
value: 67.28699999999999
- type: mrr_at_1000
value: 67.301
- type: mrr_at_3
value: 64.486
- type: mrr_at_5
value: 65.888
- type: ndcg_at_1
value: 59.68
- type: ndcg_at_10
value: 64.27199999999999
- type: ndcg_at_100
value: 69.429
- type: ndcg_at_1000
value: 70.314
- type: ndcg_at_3
value: 58.569
- type: ndcg_at_5
value: 60.272999999999996
- type: precision_at_1
value: 59.68
- type: precision_at_10
value: 15.113
- type: precision_at_100
value: 1.941
- type: precision_at_1000
value: 0.20600000000000002
- type: precision_at_3
value: 35.514
- type: precision_at_5
value: 25.367
- type: recall_at_1
value: 37.736999999999995
- type: recall_at_10
value: 73.458
- type: recall_at_100
value: 93.554
- type: recall_at_1000
value: 99.346
- type: recall_at_3
value: 55.774
- type: recall_at_5
value: 63.836000000000006
{MODEL_NAME}
This is a sentence-transformers model: It maps sentences & paragraphs to a 2048 dimensional dense vector space and can be used for tasks like clustering or semantic search.
Usage (Sentence-Transformers)
Using this model becomes easy when you have sentence-transformers installed:
pip install -U sentence-transformers
Then you can use the model like this:
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SentenceTransformer('{MODEL_NAME}')
embeddings = model.encode(sentences)
print(embeddings)
Evaluation Results
For an automated evaluation of this model, see the Sentence Embeddings Benchmark: https://seb.sbert.net
Training
The model was trained with the parameters:
DataLoader:
torch.utils.data.dataloader.DataLoader of length 350 with parameters:
{'batch_size': 512, 'sampler': 'torch.utils.data.sampler.SequentialSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
Loss:
__main__.MultipleNegativesRankingLoss_with_logging
Parameters of the fit()-Method:
{
"epochs": 1,
"evaluation_steps": 0,
"evaluator": "NoneType",
"max_grad_norm": 1,
"optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
"optimizer_params": {
"lr": 2e-05
},
"scheduler": "WarmupLinear",
"steps_per_epoch": null,
"warmup_steps": 50,
"weight_decay": 0.01
}
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: LlamaModel
(1): Pooling({'word_embedding_dimension': 2048, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)