Update README.md
Browse files
README.md
CHANGED
@@ -96,7 +96,7 @@ output = model(encoded_input)
|
|
96 |
|
97 |
## Training data
|
98 |
|
99 |
-
CLUECorpusSmall is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.
|
100 |
|
101 |
## Training procedure
|
102 |
|
@@ -142,6 +142,13 @@ python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
|
|
142 |
--tie_weights --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm
|
143 |
```
|
144 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
145 |
### BibTeX entry and citation info
|
146 |
|
147 |
```
|
|
|
96 |
|
97 |
## Training data
|
98 |
|
99 |
+
[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.
|
100 |
|
101 |
## Training procedure
|
102 |
|
|
|
142 |
--tie_weights --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm
|
143 |
```
|
144 |
|
145 |
+
Finally, we convert the pre-trained model into Huggngface's format:
|
146 |
+
```
|
147 |
+
python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path pytorch_model.bin
|
148 |
+
--output_model_path huggingface_model.bin
|
149 |
+
--layers_num 12 --target mlm
|
150 |
+
```
|
151 |
+
|
152 |
### BibTeX entry and citation info
|
153 |
|
154 |
```
|