|
--- |
|
language: hi |
|
tags: |
|
- hindi |
|
- tokenizer |
|
- bpe |
|
- subword |
|
- text-processing |
|
pipeline_tag: text2text-generation |
|
inference: true |
|
license: mit |
|
spaces: |
|
- aayushraina/bpe-hindi |
|
--- |
|
|
|
# Hindi Byte Pair Encoding (BPE) Tokenizer |
|
|
|
A specialized BPE tokenizer for Hindi text that achieves efficient compression while maintaining linguistic coherence. |
|
|
|
## Online Demo |
|
|
|
Try the tokenizer in your browser: [Hindi BPE Tokenizer Demo](https://huggingface.co/spaces/aayushraina/bpe-hindi) |
|
|
|
## Project Overview |
|
|
|
This project implements a Byte Pair Encoding (BPE) tokenizer specifically designed for Hindi text. It features: |
|
- Efficient trie-based tokenization |
|
- Visualization of training progress |
|
- Compression ratio optimization |
|
- Support for large Hindi text datasets |
|
- Hugging Face compatibility |
|
|
|
## Project Structure |
|
hindi-bpe/ |
|
├── data/ # Dataset directory |
|
│ ├── train/ # Training data |
|
│ └── valid/ # Validation data |
|
├── tokenizer/ # Saved tokenizer files |
|
│ ├── encoder.json # Encoder state |
|
│ └── vocab_stats.json # Vocabulary statistics |
|
├── output/ # Visualization outputs |
|
├── byte_pair_encoder.py # Core BPE implementation |
|
├── hindi_bpe.py # Hindi-specific wrapper |
|
├── test_hindi_bpe.py # Test suite |
|
└── requirements.txt # Dependencies |
|
|
|
## Training stats |
|
- Iteration 4500: |
|
- Vocabulary size: 4,477 |
|
- Data size: 448,754 |
|
- Compression ratio: 3.66 |
|
- Max token length: 64 |
|
|
|
## File Descriptions |
|
|
|
1. **byte_pair_encoder.py** |
|
- Core BPE implementation |
|
- Trie-based tokenization |
|
- Training statistics tracking |
|
- Visualization utilities |
|
|
|
2. **hindi_bpe.py** |
|
- Hindi-specific tokenizer wrapper |
|
- Text preprocessing |
|
- Model saving/loading |
|
- Compression ratio calculation |
|
|
|
3. **app.py** |
|
- Interactive web interface |
|
- Real-time tokenization |
|
- Training visualization |
|
- Model parameter tuning |
|
|
|
4. **test_hindi_bpe.py** |
|
- Test suite for tokenizer |
|
- Performance benchmarks |
|
- Example usage |
|
|
|
## Installation |
|
- bash |
|
- Clone repository |
|
- git clone https://github.com/yourusername/hindi-bpe.git |
|
- cd hindi-bpe |
|
- pip install -r requirements.txt |
|
|
|
## Download and prepare dataset |
|
- python download_dataset.py |
|
|
|
### Web Interface |
|
- streamlit run app.py |
|
|
|
### Test- |
|
- python test_hindi_bpe.py |
|
- The test suite includes: |
|
- Training pipeline verification |
|
- Compression ratio validation |
|
- Token count requirements |
|
- Encoding/decoding accuracy |
|
|
|
## Performance Metrics |
|
|
|
The tokenizer aims to achieve: |
|
- Vocabulary size < 5000 tokens |
|
- Compression ratio ≥ 3.2 |
|
- Fast encoding/decoding |
|
- Memory-efficient operation |
|
|
|
## Contributing |
|
|
|
1. Fork the repository |
|
2. Create feature branch |
|
3. Commit changes |
|
4. Push to branch |
|
5. Create Pull Request |
|
|
|
## License |
|
|
|
This project is licensed under the MIT License - see the LICENSE file for details. |
|
|
