Update README.md
Browse files
README.md
CHANGED
|
@@ -28,7 +28,15 @@ widget:
|
|
| 28 |
pipeline_tag: text-classification
|
| 29 |
---
|
| 30 |
|
| 31 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
- Install `pyvi` to word segment:
|
| 33 |
|
| 34 |
- `pip install pyvi`
|
|
@@ -41,7 +49,7 @@ pipeline_tag: text-classification
|
|
| 41 |
|
| 42 |
- `pip install transformers`
|
| 43 |
|
| 44 |
-
## Pre-processing
|
| 45 |
|
| 46 |
```python
|
| 47 |
from pyvi import ViTokenizer
|
|
@@ -59,7 +67,7 @@ tokenized_sentences = [ViTokenizer.tokenize(sent) for sent in sentences]
|
|
| 59 |
tokenized_pairs = [[tokenized_query, sent] for sent in tokenized_sentences]
|
| 60 |
```
|
| 61 |
|
| 62 |
-
## Usage with sentence-transformers
|
| 63 |
|
| 64 |
```python
|
| 65 |
from sentence_transformers import CrossEncoder
|
|
@@ -67,7 +75,7 @@ model = CrossEncoder('itdainb/vietnamese-cross-encoder', max_length=256)
|
|
| 67 |
scores = model.predict(tokenized_pairs)
|
| 68 |
```
|
| 69 |
|
| 70 |
-
## Usage with transformers
|
| 71 |
|
| 72 |
```python
|
| 73 |
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
|
@@ -86,7 +94,7 @@ with torch.no_grad():
|
|
| 86 |
print(scores)
|
| 87 |
```
|
| 88 |
|
| 89 |
-
## Performance
|
| 90 |
In the following table, we provide various pre-trained Cross-Encoders together with their performance on the [MS MMarco Passage Reranking - Vi - Dev](https://huggingface.co/datasets/unicamp-dl/mmarco) dataset.
|
| 91 |
|
| 92 |
| Model-Name | NDCG@3 | MRR@3 | NDCG@5 | MRR@5 | NDCG@10 | MRR@10 | Docs / Sec |
|
|
|
|
| 28 |
pipeline_tag: text-classification
|
| 29 |
---
|
| 30 |
|
| 31 |
+
#### Table of contents
|
| 32 |
+
1. [Installation](#install)
|
| 33 |
+
2. [Pre-processing](#preprocess)
|
| 34 |
+
3. [Usage with `sentence-transformers`](#sentence)
|
| 35 |
+
4. [Usage with `transformers`](#transformers)
|
| 36 |
+
5. [Performance](#performance)
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
## Installation<a name="install"></a>
|
| 40 |
- Install `pyvi` to word segment:
|
| 41 |
|
| 42 |
- `pip install pyvi`
|
|
|
|
| 49 |
|
| 50 |
- `pip install transformers`
|
| 51 |
|
| 52 |
+
## Pre-processing<a name="preprocess"></a>
|
| 53 |
|
| 54 |
```python
|
| 55 |
from pyvi import ViTokenizer
|
|
|
|
| 67 |
tokenized_pairs = [[tokenized_query, sent] for sent in tokenized_sentences]
|
| 68 |
```
|
| 69 |
|
| 70 |
+
## Usage with sentence-transformers<a name="sentence"></a>
|
| 71 |
|
| 72 |
```python
|
| 73 |
from sentence_transformers import CrossEncoder
|
|
|
|
| 75 |
scores = model.predict(tokenized_pairs)
|
| 76 |
```
|
| 77 |
|
| 78 |
+
## Usage with transformers<a name="transformers"></a>
|
| 79 |
|
| 80 |
```python
|
| 81 |
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
|
|
|
| 94 |
print(scores)
|
| 95 |
```
|
| 96 |
|
| 97 |
+
## Performance<a name="performance"></a>
|
| 98 |
In the following table, we provide various pre-trained Cross-Encoders together with their performance on the [MS MMarco Passage Reranking - Vi - Dev](https://huggingface.co/datasets/unicamp-dl/mmarco) dataset.
|
| 99 |
|
| 100 |
| Model-Name | NDCG@3 | MRR@3 | NDCG@5 | MRR@5 | NDCG@10 | MRR@10 | Docs / Sec |
|