Update README.md
Browse files
README.md
CHANGED
@@ -29,14 +29,14 @@ pipeline_tag: text-classification
|
|
29 |
---
|
30 |
|
31 |
#### Table of contents
|
32 |
-
1. [Installation](#
|
33 |
2. [Pre-processing](#preprocess)
|
34 |
3. [Usage with `sentence-transformers`](#sentence)
|
35 |
4. [Usage with `transformers`](#transformers)
|
36 |
-
5. [Performance](#
|
37 |
|
38 |
|
39 |
-
## Installation
|
40 |
- Install `pyvi` to word segment:
|
41 |
|
42 |
- `pip install pyvi`
|
@@ -49,7 +49,7 @@ pipeline_tag: text-classification
|
|
49 |
|
50 |
- `pip install transformers`
|
51 |
|
52 |
-
## Pre-processing
|
53 |
|
54 |
```python
|
55 |
from pyvi import ViTokenizer
|
@@ -67,7 +67,7 @@ tokenized_sentences = [ViTokenizer.tokenize(sent) for sent in sentences]
|
|
67 |
tokenized_pairs = [[tokenized_query, sent] for sent in tokenized_sentences]
|
68 |
```
|
69 |
|
70 |
-
## Usage with sentence-transformers
|
71 |
|
72 |
```python
|
73 |
from sentence_transformers import CrossEncoder
|
@@ -75,7 +75,7 @@ model = CrossEncoder('itdainb/vietnamese-cross-encoder', max_length=256)
|
|
75 |
scores = model.predict(tokenized_pairs)
|
76 |
```
|
77 |
|
78 |
-
## Usage with transformers
|
79 |
|
80 |
```python
|
81 |
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
@@ -94,7 +94,7 @@ with torch.no_grad():
|
|
94 |
print(scores)
|
95 |
```
|
96 |
|
97 |
-
## Performance
|
98 |
In the following table, we provide various pre-trained Cross-Encoders together with their performance on the [MS MMarco Passage Reranking - Vi - Dev](https://huggingface.co/datasets/unicamp-dl/mmarco) dataset.
|
99 |
|
100 |
| Model-Name | NDCG@3 | MRR@3 | NDCG@5 | MRR@5 | NDCG@10 | MRR@10 | Docs / Sec |
|
|
|
29 |
---
|
30 |
|
31 |
#### Table of contents
|
32 |
+
1. [Installation](#Installation)
|
33 |
2. [Pre-processing](#preprocess)
|
34 |
3. [Usage with `sentence-transformers`](#sentence)
|
35 |
4. [Usage with `transformers`](#transformers)
|
36 |
+
5. [Performance](#Performance)
|
37 |
|
38 |
|
39 |
+
## Installation
|
40 |
- Install `pyvi` to word segment:
|
41 |
|
42 |
- `pip install pyvi`
|
|
|
49 |
|
50 |
- `pip install transformers`
|
51 |
|
52 |
+
## Pre-processing
|
53 |
|
54 |
```python
|
55 |
from pyvi import ViTokenizer
|
|
|
67 |
tokenized_pairs = [[tokenized_query, sent] for sent in tokenized_sentences]
|
68 |
```
|
69 |
|
70 |
+
## Usage with sentence-transformers
|
71 |
|
72 |
```python
|
73 |
from sentence_transformers import CrossEncoder
|
|
|
75 |
scores = model.predict(tokenized_pairs)
|
76 |
```
|
77 |
|
78 |
+
## Usage with transformers
|
79 |
|
80 |
```python
|
81 |
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
|
|
94 |
print(scores)
|
95 |
```
|
96 |
|
97 |
+
## Performance
|
98 |
In the following table, we provide various pre-trained Cross-Encoders together with their performance on the [MS MMarco Passage Reranking - Vi - Dev](https://huggingface.co/datasets/unicamp-dl/mmarco) dataset.
|
99 |
|
100 |
| Model-Name | NDCG@3 | MRR@3 | NDCG@5 | MRR@5 | NDCG@10 | MRR@10 | Docs / Sec |
|