I think hazm still performs better than this tokenizer

#1
by Waisullah - opened

The red highlighted output is what I got from mshojaei77/PersianBPETokenizer
image.png

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment