Update to tokenizers > 0.13

by gregoriopellegrino - opened Feb 1, 2024

Feb 1, 2024

Hi,
thanks for this great work. Unfortunately tokenizers 0.12.1 cannot be installed on mac with M1 or M2 chips. Would it be possible to update tokenizers dependencies to at least version 0.13?

Thank you,
Gregory

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment