Stop the warning of error Decoding tokens

#11

by Beegbrain - opened 30 days ago

30 days ago

Hello, I'm currently using the FAST tokenizer in a custom pipeline to train a model. The "Error decoding tokens" is happening a lot in the beginning of the training and overflows my log file. Is there a way to silent the warning ?

Thanks in advance

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment