Hugging Face's logo

sergeyzh
/
rubert-tiny-sts

Sentence Similarity
sentence-transformers
Safetensors
Transformers
Russian
bert
pretraining
russian
embeddings
tiny
feature-extraction
text-embeddings-inference
Model card Files Community
sergeyzh commited on
Commit
91435a1
·
verified ·
1 Parent(s): 94de706

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +96 -135
README.md CHANGED
@@ -1,135 +1,96 @@
1
- ---
2
- language:
3
- - ru
4
-
5
- pipeline_tag: sentence-similarity
6
-
7
- tags:
8
- - russian
9
- - pretraining
10
- - embeddings
11
- - tiny
12
- - feature-extraction
13
- - sentence-similarity
14
- - sentence-transformers
15
- - transformers
16
-
17
- license: mit
18
- base_model: cointegrated/rubert-tiny2
19
-
20
- ---
21
-
22
- ## Быстрый Bert для Semantic text similarity (STS) на CPU
23
-
24
- Быстрая модель BERT для расчетов компактных эмбедингов предложений на русском языке. Модель основана на [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) - имеет аналогичные размеры контекста (2048), ембединга (312) и быстродействие. Является первой и самой быстрой моделью в серии BERT-STS.
25
-
26
- На STS и близких задачах (PI, NLI, SA, TI) для русского языка превосходит по качеству LaBSE. Для работы с контекстом свыше 512 токенов требует дообучения под целевой домен.
27
-
28
- ## Выбор модели из серии BERT-STS (качество/скорость)
29
- | Рекомендуемая модель | CPU <br> (STS; snt/s) | GPU <br> (STS; snt/s) |
30
- |:---------------------------------|:---------:|:---------:|
31
- | Быстрая модель (скорость) | **rubert-tiny-sts <br> (0.797; 1190)** | - |
32
- | Базовая модель (качество) | [rubert-mini-sts](https://huggingface.co/sergeyzh/rubert-mini-sts) <br> (0.815; 539) | [LaBSE-ru-sts](https://huggingface.co/sergeyzh/LaBSE-ru-sts) <br> (0.845; 1894) |
33
-
34
- ## Быстрая модель для использования в составе RAG LLMs при инференсе на CPU:
35
- - высокое качество при нечетких запросах (отличный метрики на задачах STS, PI, NLI);
36
- - низкое влияение эмоциональной окраски текста на ембединг (средние показатели на задачах SA, TI);
37
- - легкое расширение базы текстовых документов (скорость работы на CPU > 1k предложений в секунду);
38
- - ускорение алгоритмов knn при поиске соответствий (низкая размерность эмбединга 312);
39
- - простота использования (совместимость с [SentenceTransformer](https://github.com/UKPLab/sentence-transformers)).
40
-
41
- ## Использование модели с библиотекой `transformers`:
42
-
43
- ```python
44
- # pip install transformers sentencepiece
45
- import torch
46
- from transformers import AutoTokenizer, AutoModel
47
- tokenizer = AutoTokenizer.from_pretrained("sergeyzh/rubert-tiny-sts")
48
- model = AutoModel.from_pretrained("sergeyzh/rubert-tiny-sts")
49
- # model.cuda() # uncomment it if you have a GPU
50
-
51
- def embed_bert_cls(text, model, tokenizer):
52
- t = tokenizer(text, padding=True, truncation=True, return_tensors='pt')
53
- with torch.no_grad():
54
- model_output = model(**{k: v.to(model.device) for k, v in t.items()})
55
- embeddings = model_output.last_hidden_state[:, 0, :]
56
- embeddings = torch.nn.functional.normalize(embeddings)
57
- return embeddings[0].cpu().numpy()
58
-
59
- print(embed_bert_cls('привет мир', model, tokenizer).shape)
60
- # (312,)
61
- ```
62
-
63
- ## Использование с `sentence_transformers`:
64
- ```Python
65
- from sentence_transformers import SentenceTransformer, util
66
-
67
- model = SentenceTransformer('sergeyzh/rubert-tiny-sts')
68
-
69
- sentences = ["привет мир", "hello world", "здравствуй вселенная"]
70
- embeddings = model.encode(sentences)
71
- print(util.dot_score(embeddings, embeddings))
72
- ```
73
-
74
- ## Метрики
75
- Оценки модели на бенчмарке [encodechka](https://github.com/avidale/encodechka):
76
-
77
- | Модель | STS | PI | NLI | SA | TI |
78
- |:---------------------------------|:---------:|:---------:|:---------:|:---------:|:---------:|
79
- | [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 0.862 | 0.727 | 0.473 | 0.810 | 0.979 |
80
- | [sergeyzh/LaBSE-ru-sts](https://huggingface.co/sergeyzh/LaBSE-ru-sts) | 0.845 | 0.737 | 0.481 | 0.805 | 0.957 |
81
- | [sergeyzh/rubert-mini-sts](https://huggingface.co/sergeyzh/rubert-mini-sts) | 0.815 | 0.723 | 0.477 | 0.791 | 0.949 |
82
- | **sergeyzh/rubert-tiny-sts** | **0.797** | **0.702** | **0.453** | **0.778** | **0.946** |
83
- | [Tochka-AI/ruRoPEBert-e5-base-512](https://huggingface.co/Tochka-AI/ruRoPEBert-e5-base-512) | 0.793 | 0.704 | 0.457 | 0.803 | 0.970 |
84
- | [cointegrated/LaBSE-en-ru](https://huggingface.co/cointegrated/LaBSE-en-ru) | 0.794 | 0.659 | 0.431 | 0.761 | 0.946 |
85
- | [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) | 0.750 | 0.651 | 0.417 | 0.737 | 0.937 |
86
-
87
- **Задачи:**
88
-
89
- - Semantic text similarity (**STS**);
90
- - Paraphrase identification (**PI**);
91
- - Natural language inference (**NLI**);
92
- - Sentiment analysis (**SA**);
93
- - Toxicity identification (**TI**).
94
-
95
- ## Быстродействие и размеры
96
-
97
- На бенчмарке [encodechka](https://github.com/avidale/encodechka):
98
-
99
- | Модель | CPU | GPU | size | dim | n_ctx | n_vocab |
100
- |:---------------------------------|----------:|----------:|----------:|----------:|----------:|----------:|
101
- | [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 149.026 | 15.629 | 2136 | 1024 | 514 | 250002 |
102
- | [sergeyzh/LaBSE-ru-sts](https://huggingface.co/sergeyzh/LaBSE-ru-sts) | 42.835 | 8.561 | 490 | 768 | 512 | 55083 |
103
- | [sergeyzh/rubert-mini-sts](https://huggingface.co/sergeyzh/rubert-mini-sts) | 6.417 | 5.517 | 123 | 312 | 2048 | 83828 |
104
- | **sergeyzh/rubert-tiny-sts** | **3.208** | **3.379** | **111** | **312** | **2048** | **83828** |
105
- | [Tochka-AI/ruRoPEBert-e5-base-512](https://huggingface.co/Tochka-AI/ruRoPEBert-e5-base-512) | 43.314 | 9.338 | 532 | 768 | 512 | 69382 |
106
- | [cointegrated/LaBSE-en-ru](https://huggingface.co/cointegrated/LaBSE-en-ru) | 42.867 | 8.549 | 490 | 768 | 512 | 55083 |
107
- | [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) | 3.212 | 3.384 | 111 | 312 | 2048 | 83828 |
108
-
109
-
110
-
111
- При использовании батчей с `sentence_transformers`:
112
-
113
- ```python
114
- from sentence_transformers import SentenceTransformer
115
-
116
- model_name = 'sergeyzh/rubert-tiny-sts'
117
- model = SentenceTransformer(model_name, device='cpu')
118
- sentences = ["Тест быстродействия на CPU Ryzen 7 3800X: batch = 1000"] * 1000
119
- %timeit -n 5 -r 3 model.encode(sentences)
120
-
121
- # 840 ms ± 8.08 ms per loop (mean ± std. dev. of 3 runs, 5 loops each)
122
- # 1000/0.840 = 1190 snt/s
123
-
124
- model = SentenceTransformer(model_name, device='cuda')
125
- sentences = ["Тест быстродействия на GPU RTX 3060: batch = 8000"] * 8000
126
- %timeit -n 5 -r 3 model.encode(sentences)
127
-
128
- # 922 ms ± 29.5 ms per loop (mean ± std. dev. of 3 runs, 5 loops each)
129
- # 8000/0.922 = 8677 snt/s
130
- ```
131
-
132
-
133
- ## Связанные ресурсы
134
- Вопросы использования модели обсуждаются в [русскоязычном чате NLP](https://t.me/natural_language_processing).
135
-
 
1
+ ---
2
+ language:
3
+ - ru
4
+
5
+ pipeline_tag: sentence-similarity
6
+
7
+ tags:
8
+ - russian
9
+ - pretraining
10
+ - embeddings
11
+ - tiny
12
+ - feature-extraction
13
+ - sentence-similarity
14
+ - sentence-transformers
15
+ - transformers
16
+
17
+ license: mit
18
+ base_model: cointegrated/rubert-tiny2
19
+
20
+ ---
21
+
22
+ ## Быстрый Bert для Semantic text similarity (STS) на CPU
23
+
24
+ Быстрая модель BERT для расчетов компактных эмбеддингов предложений на русском языке. Модель основана на [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) - имеет аналогичные размеры контекста (2048), ембеддинга (312) и быстродействие.
25
+
26
+
27
+ ## Использование модели с библиотекой `transformers`:
28
+
29
+ ```python
30
+ # pip install transformers sentencepiece
31
+ import torch
32
+ from transformers import AutoTokenizer, AutoModel
33
+ tokenizer = AutoTokenizer.from_pretrained("sergeyzh/rubert-tiny-sts")
34
+ model = AutoModel.from_pretrained("sergeyzh/rubert-tiny-sts")
35
+ # model.cuda() # uncomment it if you have a GPU
36
+
37
+ def embed_bert_cls(text, model, tokenizer):
38
+ t = tokenizer(text, padding=True, truncation=True, return_tensors='pt')
39
+ with torch.no_grad():
40
+ model_output = model(**{k: v.to(model.device) for k, v in t.items()})
41
+ embeddings = model_output.last_hidden_state[:, 0, :]
42
+ embeddings = torch.nn.functional.normalize(embeddings)
43
+ return embeddings[0].cpu().numpy()
44
+
45
+ print(embed_bert_cls('привет мир', model, tokenizer).shape)
46
+ # (312,)
47
+ ```
48
+
49
+ ## Использование с `sentence_transformers`:
50
+ ```Python
51
+ from sentence_transformers import SentenceTransformer, util
52
+
53
+ model = SentenceTransformer('sergeyzh/rubert-tiny-sts')
54
+
55
+ sentences = ["привет мир", "hello world", "здравствуй вселенная"]
56
+ embeddings = model.encode(sentences)
57
+ print(util.dot_score(embeddings, embeddings))
58
+ ```
59
+
60
+ ## Метрики
61
+ Оценки модели на бенчмарке [encodechka](https://github.com/avidale/encodechka):
62
+
63
+ | Модель | STS | PI | NLI | SA | TI |
64
+ |:---------------------------------|:---------:|:---------:|:---------:|:---------:|:---------:|
65
+ | [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 0.862 | 0.727 | 0.473 | 0.810 | 0.979 |
66
+ | [sergeyzh/LaBSE-ru-sts](https://huggingface.co/sergeyzh/LaBSE-ru-sts) | 0.845 | 0.737 | 0.481 | 0.805 | 0.957 |
67
+ | [sergeyzh/rubert-mini-sts](https://huggingface.co/sergeyzh/rubert-mini-sts) | 0.815 | 0.723 | 0.477 | 0.791 | 0.949 |
68
+ | **sergeyzh/rubert-tiny-sts** | 0.797 | 0.702 | 0.453 | 0.778 | 0.946 |
69
+ | [Tochka-AI/ruRoPEBert-e5-base-512](https://huggingface.co/Tochka-AI/ruRoPEBert-e5-base-512) | 0.793 | 0.704 | 0.457 | 0.803 | 0.970 |
70
+ | [cointegrated/LaBSE-en-ru](https://huggingface.co/cointegrated/LaBSE-en-ru) | 0.794 | 0.659 | 0.431 | 0.761 | 0.946 |
71
+ | [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) | 0.750 | 0.651 | 0.417 | 0.737 | 0.937 |
72
+
73
+ **Задачи:**
74
+
75
+ - Semantic text similarity (**STS**);
76
+ - Paraphrase identification (**PI**);
77
+ - Natural language inference (**NLI**);
78
+ - Sentiment analysis (**SA**);
79
+ - Toxicity identification (**TI**).
80
+
81
+ ## Быстродействие и размеры
82
+
83
+ На бенчмарке [encodechka](https://github.com/avidale/encodechka):
84
+
85
+ | Модель | CPU | GPU | size | dim | n_ctx | n_vocab |
86
+ |:---------------------------------|----------:|----------:|----------:|----------:|----------:|----------:|
87
+ | [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 149.026 | 15.629 | 2136 | 1024 | 514 | 250002 |
88
+ | [sergeyzh/LaBSE-ru-sts](https://huggingface.co/sergeyzh/LaBSE-ru-sts) | 42.835 | 8.561 | 490 | 768 | 512 | 55083 |
89
+ | [sergeyzh/rubert-mini-sts](https://huggingface.co/sergeyzh/rubert-mini-sts) | 6.417 | 5.517 | 123 | 312 | 2048 | 83828 |
90
+ | **sergeyzh/rubert-tiny-sts** | 3.208 | 3.379 | 111 | 312 | 2048 | 83828 |
91
+ | [Tochka-AI/ruRoPEBert-e5-base-512](https://huggingface.co/Tochka-AI/ruRoPEBert-e5-base-512) | 43.314 | 9.338 | 532 | 768 | 512 | 69382 |
92
+ | [cointegrated/LaBSE-en-ru](https://huggingface.co/cointegrated/LaBSE-en-ru) | 42.867 | 8.549 | 490 | 768 | 512 | 55083 |
93
+ | [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) | 3.212 | 3.384 | 111 | 312 | 2048 | 83828 |
94
+
95
+
96
+