metadata
license: apache-2.0
datasets:
- mozilla-foundation/common_voice_17_0
language:
- uz
base_model:
- FacebookAI/xlm-roberta-base
Tokenizer for Uzbek Language
Introduction
Ushbu tokenizer Mozilla Common Voice dataset ma'lumotlariga asoslangan. train+validated 130.000 sentences
Features
- Matnlarni tokenlarga ajratadi.
- Ko'p bo'lmagan talaffuz va aksentlarni qo'llab-quvvatlaydi.
Installation
Python va kerakli kutubxonalar:
pip install transformers datasets
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("jamshidahmadov/uz_tokenizer")
text = "O'zbekistonda turli xil NLP loyihalari qurilmoqda"
tokens = tokenizer.tokenize(text)
print(tokens)
Dataset Description
Common Voice 17.0 dataseti multilangual ya'ni ko'p tilli bo'lib o'zbek tilini ham qo'llab quvvatlaydi.