KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications

Hi @mjbommar ,

I heavily think that the resources are really great. However, I heavily dislike the current structure of the datasets.

It first starts with the release of over 400 single datasets, with little to zero documentation, what is actually inside these datasets. Why not putting them into one dataset repo with good documentation and subsets?

Second: It is a bad idea to release a dataset that just has a link to an S3 bucket: it absolutely makes no sense. When the S3 bucket is deleted of whatever reason, it is completely gone. Why not releasing the content of these files directly here on Hugging Face?

Third: how can I download a single file from the dataset without an AWS account? And why do we need an extra client library for accessing the data?

Then I found this paper ( https://arxiv.org/pdf/2504.07854 ), sorry, but this is not democratization of AI:

As a non-profit, we unfortunately cannot afford to pay unrestricted egress fees for the raw data, but users within us-east-1 can access this data
for free and interested parties may contact us for assistance obtaining the data or coordinating alternative data transfer arrangements.

You could simply host the content here on HF.

The overall user experience is so frustrating, I am very close in scraping the S3 bucket and putting the content here on HF. This will cost some money for me, but at least I am doing something for democratization of AI.