Upload 14 files

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false,
+  "include_prompt": true
+}

README.md

@@ -0,0 +1,180 @@
+---
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+tags:
+  - sentence-transformers
+  - feature-extraction
+  - sentence-similarity
+  - transformers
+---
+
+# Enbeddrus v0.1 PC - English and Russian embedder
+
+> This is the model trained on Parallel Corpora only 
+
+This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional
+dense vector space and can be used for tasks like clustering or semantic search.
+
+- **Parameters**: 168 million
+- **Layers**: 12
+- **Hidden Size**: 768
+- **Attention Heads**: 12
+- **Vocabulary Size**: 119,547
+- **Maximum Sequence Length**: 512 tokens
+
+The Enbeddrus model is designed to extract similar embeddings for comparable English and Russian phrases. It is based on
+the [bert-base-multilingual-uncased](https://huggingface.co/google-bert/bert-base-multilingual-cased) model and was
+trained over 20 epochs on the following datasets:
+
+- [evilfreelancer/opus-php-en-ru-cleaned](https://huggingface.co/datasets/evilfreelancer/opus-php-en-ru-cleaned) (
+  train): 1.6k lines
+- [Helsinki-NLP/opus_books](https://huggingface.co/datasets/Helsinki-NLP/opus_books/viewer/en-ru) (en-ru, train): 17.5k
+  lines
+
+The goal of this model is to generate identical or very similar embeddings regardless of whether the text is written in
+English or Russian.
+
+## Usage (Sentence-Transformers)
+
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+
+```
+pip install -U sentence-transformers
+```
+
+Then you can use the model like this:
+
+```python
+from sentence_transformers import SentenceTransformer
+
+sentences = [
+    "PHP является скриптовым языком программирования, широко используемым для веб-разработки.",
+    "PHP is a scripting language widely used for web development.",
+    "PHP поддерживает множество баз данных, таких как MySQL, PostgreSQL и SQLite.",
+    "PHP supports many databases like MySQL, PostgreSQL, and SQLite.",
+    "Функция echo в PHP используется для вывода текста на экран.",
+    "The echo function in PHP is used to output text to the screen.",
+    "Машинное обучение помогает создавать интеллектуальные системы.",
+    "Machine learning helps to create intelligent systems.",
+]
+
+model = SentenceTransformer('evilfreelancer/enbeddrus')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
+
+## Usage (HuggingFace Transformers)
+
+Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input
+through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word
+embeddings.
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+
+# Mean Pooling - Take attention mask into account for correct averaging
+def mean_pooling(model_output, attention_mask):
+    token_embeddings = model_output[0]  # First element of model_output contains all token embeddings
+    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+
+
+# Sentences we want sentence embeddings for
+sentences = [
+    "PHP является скриптовым языком программирования, широко используемым для веб-разработки.",
+    "PHP is a scripting language widely used for web development.",
+    "PHP поддерживает множество баз данных, таких как MySQL, PostgreSQL и SQLite.",
+    "PHP supports many databases like MySQL, PostgreSQL, and SQLite.",
+    "Функция echo в PHP используется для вывода текста на экран.",
+    "The echo function in PHP is used to output text to the screen.",
+    "Машинное обучение помогает создавать интеллектуальные системы.",
+    "Machine learning helps to create intelligent systems.",
+]
+
+# Load model from HuggingFace Hub
+tokenizer = AutoTokenizer.from_pretrained('evilfreelancer/enbeddrus')
+model = AutoModel.from_pretrained('evilfreelancer/enbeddrus')
+
+# Tokenize sentences
+encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+
+# Compute token embeddings
+with torch.no_grad():
+    model_output = model(**encoded_input)
+
+# Perform pooling. In this case, mean pooling.
+sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
+
+print("Sentence embeddings:")
+print(sentence_embeddings)
+```
+
+## Evaluation Results
+
+The model was tested on the `eval` split of the
+dataset [evilfreelancer/opus-php-en-ru-cleaned](https://huggingface.co/datasets/evilfreelancer/opus-php-en-ru-cleaned),
+which contains 100 pairs of sentences in Russian and English on the topic of PHP. The results of the testing are
+presented in the image below.
+
+![Evaluation Results](./eval.png)
+
+* **Left**: Embedding similarity in Russian and English before training
+  (the points are spread out into two distinct clusters).
+* **Center**: Embedding similarity after training
+  (the points representing similar phrases are very close to each other).
+* **Right**: Cosine distance before and after training.
+
+## Training
+
+The model was trained with the parameters:
+
+**DataLoader**:
+
+`torch.utils.data.dataloader.DataLoader` of length 556 with parameters:
+
+```python
+{
+    'batch_size': 64,
+    'sampler': 'torch.utils.data.sampler.RandomSampler',
+    'batch_sampler': 'torch.utils.data.sampler.BatchSampler'
+}
+```
+
+**Loss**:
+
+`sentence_transformers.losses.MSELoss.MSELoss`
+
+Parameters of the fit()-Method:
+
+```
+{
+    "epochs": 20,
+    "evaluation_steps": 100,
+    "evaluator": "sentence_transformers.evaluation.SequentialEvaluator.SequentialEvaluator",
+    "max_grad_norm": 1,
+    "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
+    "optimizer_params": {
+        "eps": 1e-06,
+        "lr": 2e-05
+    },
+    "scheduler": "WarmupLinear",
+    "steps_per_epoch": null,
+    "warmup_steps": 10000,
+    "weight_decay": 0.01
+}
+```
+
+## Full Model Architecture
+
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
+)
+```
+
+## Citing & Authors
+
+<!--- Describe where people can find more information -->

config.json

@@ -0,0 +1,31 @@
+{
+  "_name_or_path": "./output/enbeddrus_domain",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "directionality": "bidi",
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "pooler_fc_size": 768,
+  "pooler_num_attention_heads": 12,
+  "pooler_num_fc_layers": 3,
+  "pooler_size_per_head": 128,
+  "pooler_type": "first_token_transform",
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.40.2",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 105879
+}

config_sentence_transformers.json

@@ -0,0 +1,9 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.7.0",
+    "transformers": "4.40.2",
+    "pytorch": "2.3.0+cu121"
+  },
+  "prompts": {},
+  "default_prompt_name": null
+}

eval/mse_evaluation_talks-en-ru-dev.tsv.gz_results.csv

@@ -0,0 +1,121 @@
+epoch,steps,MSE
+0,100,10.387847572565079
+0,200,9.86761674284935
+0,300,9.120597690343857
+0,400,8.35539773106575
+0,500,7.655713707208633
+0,-1,7.338091731071472
+1,100,6.162974983453751
+1,200,5.53399957716465
+1,300,5.004849284887314
+1,400,4.520170763134956
+1,500,4.106686636805534
+1,-1,3.9841126650571823
+2,100,3.6872930824756622
+2,200,3.44105027616024
+2,300,3.2708097249269485
+2,400,3.0513660982251167
+2,500,2.870635688304901
+2,-1,2.7876438573002815
+3,100,2.6700804010033607
+3,200,2.546677738428116
+3,300,2.4392174556851387
+3,400,2.3880528286099434
+3,500,2.31433417648077
+3,-1,2.2802533581852913
+4,100,2.2168152034282684
+4,200,2.1714042872190475
+4,300,2.1294882521033287
+4,400,2.094305120408535
+4,500,2.0370664075016975
+4,-1,2.0246170461177826
+5,100,2.0074699074029922
+5,200,1.9568240270018578
+5,300,1.9343534484505653
+5,400,1.898401603102684
+5,500,1.8790734931826591
+5,-1,1.8960969522595406
+6,100,1.8495647236704826
+6,200,1.8361380323767662
+6,300,1.8119128420948982
+6,400,1.808309368789196
+6,500,1.7959684133529663
+6,-1,1.7735589295625687
+7,100,1.7616048455238342
+7,200,1.7485374584794044
+7,300,1.7368398606777191
+7,400,1.7239836975932121
+7,500,1.6941288486123085
+7,-1,1.6788296401500702
+8,100,1.6896054148674011
+8,200,1.6750071197748184
+8,300,1.6543004661798477
+8,400,1.6510825604200363
+8,500,1.6183236613869667
+8,-1,1.6258953139185905
+9,100,1.6203954815864563
+9,200,1.6064176335930824
+9,300,1.5877032652497292
+9,400,1.594270020723343
+9,500,1.5924764797091484
+9,-1,1.5786411240696907
+10,100,1.5770653262734413
+10,200,1.550190057605505
+10,300,1.5557215549051762
+10,400,1.5478377230465412
+10,500,1.53389573097229
+10,-1,1.5464226715266705
+11,100,1.5201290138065815
+11,200,1.521117053925991
+11,300,1.5080863609910011
+11,400,1.5010199509561062
+11,500,1.501933578401804
+11,-1,1.494067721068859
+12,100,1.4922548085451126
+12,200,1.5047299675643444
+12,300,1.4688256196677685
+12,400,1.4845688827335835
+12,500,1.4618145301938057
+12,-1,1.4620558358728886
+13,100,1.462116278707981
+13,200,1.4583625830709934
+13,300,1.4507200568914413
+13,400,1.4375234954059124
+13,500,1.4326025731861591
+13,-1,1.4305923134088516
+14,100,1.419608574360609
+14,200,1.4264494180679321
+14,300,1.4211633242666721
+14,400,1.4133354648947716
+14,500,1.4246515929698944
+14,-1,1.4119409956037998
+15,100,1.4064767397940159
+15,200,1.4039582572877407
+15,300,1.3883032836019993
+15,400,1.3942951336503029
+15,500,1.385231874883175
+15,-1,1.3819151557981968
+16,100,1.3912687078118324
+16,200,1.3699814677238464
+16,300,1.413984689861536
+16,400,1.3735463842749596
+16,500,1.3753779232501984
+16,-1,1.3813098892569542
+17,100,1.3768729753792286
+17,200,1.3731478713452816
+17,300,1.3841950334608555
+17,400,1.3581447303295135
+17,500,1.3539798557758331
+17,-1,1.3513457030057907
+18,100,1.3624225743114948
+18,200,1.3440818525850773
+18,300,1.3474151492118835
+18,400,1.3462526723742485
+18,500,1.348408032208681
+18,-1,1.343551930040121
+19,100,1.3368207029998302
+19,200,1.3246512971818447
+19,300,1.3250414282083511
+19,400,1.3333231210708618
+19,500,1.3302416540682316
+19,-1,1.3309257104992867

eval/translation_evaluation_talks-en-ru-dev.tsv.gz_results.csv

@@ -0,0 +1,121 @@
+epoch,steps,src2trg,trg2src
+0,100,0.797,0.683
+0,200,0.803,0.697
+0,300,0.814,0.698
+0,400,0.82,0.693
+0,500,0.825,0.677
+0,-1,0.823,0.666
+1,100,0.823,0.635
+1,200,0.797,0.639
+1,300,0.763,0.608
+1,400,0.724,0.55
+1,500,0.696,0.524
+1,-1,0.698,0.527
+2,100,0.698,0.523
+2,200,0.705,0.531
+2,300,0.718,0.54
+2,400,0.741,0.558
+2,500,0.754,0.559
+2,-1,0.766,0.57
+3,100,0.771,0.581
+3,200,0.776,0.591
+3,300,0.78,0.606
+3,400,0.784,0.608
+3,500,0.793,0.619
+3,-1,0.794,0.631
+4,100,0.801,0.64
+4,200,0.805,0.644
+4,300,0.817,0.66
+4,400,0.815,0.658
+4,500,0.821,0.666
+4,-1,0.823,0.666
+5,100,0.825,0.67
+5,200,0.826,0.681
+5,300,0.832,0.693
+5,400,0.831,0.693
+5,500,0.832,0.686
+5,-1,0.829,0.691
+6,100,0.828,0.7
+6,200,0.836,0.701
+6,300,0.837,0.709
+6,400,0.831,0.712
+6,500,0.836,0.718
+6,-1,0.842,0.725
+7,100,0.838,0.735
+7,200,0.836,0.735
+7,300,0.835,0.736
+7,400,0.841,0.743
+7,500,0.845,0.754
+7,-1,0.846,0.751
+8,100,0.847,0.757
+8,200,0.851,0.76
+8,300,0.852,0.768
+8,400,0.855,0.763
+8,500,0.853,0.775
+8,-1,0.847,0.774
+9,100,0.852,0.772
+9,200,0.854,0.78
+9,300,0.855,0.779
+9,400,0.86,0.778
+9,500,0.853,0.78
+9,-1,0.857,0.781
+10,100,0.855,0.781
+10,200,0.861,0.781
+10,300,0.856,0.786
+10,400,0.862,0.793
+10,500,0.864,0.79
+10,-1,0.868,0.789
+11,100,0.864,0.798
+11,200,0.867,0.789
+11,300,0.863,0.791
+11,400,0.871,0.798
+11,500,0.868,0.794
+11,-1,0.87,0.799
+12,100,0.868,0.801
+12,200,0.869,0.798
+12,300,0.873,0.799
+12,400,0.878,0.796
+12,500,0.879,0.801
+12,-1,0.876,0.8
+13,100,0.876,0.808
+13,200,0.877,0.802
+13,300,0.879,0.813
+13,400,0.876,0.813
+13,500,0.875,0.813
+13,-1,0.878,0.813
+14,100,0.88,0.817
+14,200,0.882,0.814
+14,300,0.885,0.813
+14,400,0.883,0.82
+14,500,0.884,0.822
+14,-1,0.884,0.822
+15,100,0.888,0.826
+15,200,0.886,0.829
+15,300,0.889,0.828
+15,400,0.89,0.831
+15,500,0.887,0.83
+15,-1,0.887,0.831
+16,100,0.89,0.834
+16,200,0.894,0.83
+16,300,0.893,0.835
+16,400,0.895,0.826
+16,500,0.895,0.837
+16,-1,0.893,0.835
+17,100,0.897,0.835
+17,200,0.897,0.838
+17,300,0.899,0.844
+17,400,0.899,0.841
+17,500,0.9,0.84
+17,-1,0.893,0.841
+18,100,0.897,0.843
+18,200,0.897,0.839
+18,300,0.903,0.844
+18,400,0.903,0.843
+18,500,0.897,0.84
+18,-1,0.896,0.842
+19,100,0.901,0.844
+19,200,0.902,0.841
+19,300,0.9,0.847
+19,400,0.902,0.845
+19,500,0.902,0.849
+19,-1,0.902,0.848

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83c3698a48afbd81bd2adc14c418ca3531094d1da72313b9f9aefb74619b4a5b
+size 669448040

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,62 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "max_length": 350,
+  "model_max_length": 512,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}