Update README.md

README.md

@@ -1,42 +1,51 @@
 ---
 license: mit
+widget:
+- text: "привет[SEP]привет![SEP]как дела?[RESPONSE_TOKEN]супер, вот только проснулся, у тебя как?"
+  example_title: "Dialog example 1"
+- text: "привет[SEP]привет![SEP]как дела?[RESPONSE_TOKEN]норм"
+  example_title: "Dialog example 2"
+- text: "привет[SEP]привет![SEP]как дела?[RESPONSE_TOKEN]норм, у тя как?"
+  example_title: "Dialog example 3"
 ---
 
 This classification model is based on [sberbank-ai/ruRoberta-large](https://huggingface.co/sberbank-ai/ruRoberta-large).
-The model should be used to produce relevance and specificity of the last message in the context of a dialog.
-
-It is pretrained on corpus of dialog data from social networks and finetuned on [tinkoff-ai/context_similarity](https://huggingface.co/tinkoff-ai/context_similarity). 
-The performance of the model on validation split [tinkoff-ai/context_similarity](https://huggingface.co/tinkoff-ai/context_similarity) (with the best thresholds for validation samples):
-
-<table>
-    <thead>
-        <tr>
-            <td colspan="2">relevance</td>
-            <td colspan="2">specificity</td>
-        </tr>
-    </thead>
-    <tbody>
-        <tr>
-            <td>f0.5</td>
-            <td>roc-auc</td>
-            <td>f0.5</td>
-            <td>roc-auc</td>
-        </tr>
-        <tr>
-            <td>0.86</td>
-            <td>0.83</td>
-            <td>0.85</td>
-            <td>0.86</td>
-        </tr>
-    </tbody>
-</table>
-
-The model can be loaded as follows:
+The model should be used to produce relevance and specificity of the last message in the context of a dialogue.
+
+The labels explanation:
+- `relevance`: is the last message in the dialogue relevant in the context of the full dialogue.
+- `specificity`: is the last message in the dialogue interesting and promotes the continuation of the dialogue.
+
+It is pretrained on a large corpus of dialog data in unsupervised manner: the model is trained to predict whether last response was in a real dialog, or it was pulled from some other dialog at random.
+Then it was finetuned on manually labelled examples (dataset will be posted soon).
+
+The model was trained with three messages in the context and one response. Each message was tokenized separately with ```  max_length = 32 ```.
+
+The performance of the model on validation split (dataset will be posted soon) (with the best thresholds for validation samples):
+
+
+|             |   threshold |   f0.5 |   ROC AUC |
+|:------------|------------:|-------:|----------:|
+| relevance   |        0.59 |   0.86 |      0.83 |
+| specificity |        0.61 |   0.85 |      0.86 |
+
+
+How to use:
 
 ```python
-# pip install transformers
-from transformers import AutoTokenizer, AutoModel
-tokenizer = AutoTokenizer.from_pretrained("tinkoff-ai/context_similarity")
-model = AutoModel.from_pretrained("tinkoff-ai/context_similarity")
-# model.cuda()
-```
+pip install transformers
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+import torch
+tokenizer = AutoTokenizer.from_pretrained('tinkoff-ai/response-quality-classifier-large')
+model = AutoModelForSequenceClassification.from_pretrained('tinkoff-ai/response-quality-classifier-large')
+model.cuda()
+inputs = tokenizer('[CLS]привет[SEP]привет![SEP]как дела?[RESPONSE_TOKEN]норм, у тя как?', max_length=128, add_special_tokens=False, return_tensors='pt')
+with torch.inference_mode():
+    logits = model(**inputs).logits
+    probas = torch.sigmoid(logits)[0].cpu().detach().numpy()
+relevance, specificity = probas
+```
+
+The [app](https://huggingface.co/spaces/tinkoff-ai/response-quality-classifiers) where you can easily interact with this model.
+
+The work was done during internship at Tinkoff by [egoriyaa](https://github.com/egoriyaa), mentored by [solemn-leader](https://huggingface.co/solemn-leader).