File size: 465 Bytes
c34bcb2 6e5e25e 383ed09 c34bcb2 f8bbc3d c34bcb2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
---
language: en
tags:
- tokenizer
- pytorch
- streaming
library_name: nano
pipeline_tag: token-classification
datasets:
- Salesforce/wikitext
---
# Nano Tokenizer
This tokenizer was trained using a Python-only pipeline (no `transformers` or `tokenizers`), on a dataset streamed from the Hugging Face Hub.
## Usage
```python
from transformers import PreTrainedTokenizerFast
tokenizer = PreTrainedTokenizerFast.from_pretrained("goabonga/wikitext-2-raw-v1")
```
|