---
license: apache-2.0
datasets:
- mozilla-foundation/common_voice_17_0
language:
- uz
base_model:
- FacebookAI/xlm-roberta-base
---

# Tokenizer for Uzbek Language

## Introduction
Ushbu tokenizer Mozilla Common Voice dataset ma'lumotlariga asoslangan. train+validated 130.000 sentences

## Features
- Matnlarni tokenlarga ajratadi.
- Ko'p bo'lmagan talaffuz va aksentlarni qo'llab-quvvatlaydi.

## Installation
Python va kerakli kutubxonalar:
```
pip install transformers datasets
```

## Usage
```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("jamshidahmadov/uz_tokenizer")

text = "O'zbekistonda turli xil NLP loyihalari qurilmoqda"
tokens = tokenizer.tokenize(text)
print(tokens)
```

## Dataset Description
Common Voice 17.0 dataseti multilangual ya'ni ko'p tilli bo'lib o'zbek tilini ham qo'llab quvvatlaydi.

## Contact 
[Jamshid Ahmadov](https://www.linkedin.com/in/jamshid-ds)