Note Original dataset used to train tokenisers and models.

pietrolesci/tokenisers

Updated 26 days ago

Note Tokenisers trained on the MiniPile. The `_raw_tokenisers` folder contains the original tokenisers trained with a vocabulary size of 320k. Then, each folder is a `transformers`-compatible tokeniser of a smaller size.

Note Tokenised MiniPile dataset(s). Each split correponds to a tokeniser in `pietrolesci/tokenisers`.

Note This dataset is a subset of the `fineweb-edu/sample/100BT` dataset containing the initial 20.2M documents (or roughly 20B tokens). The default configuration is the raw data. Any other configuration corresponds to the tokenised version, where the configuration name corresponds to the tokeniser used to tokenise it (see `pietrolesci/tokenisers`).

pietrolesci/me57M-tied_minipile_bpe8064minipile

Updated 11 days ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me57M-tied_minipile_bpe32000minipile

Updated 11 days ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me57M-tied_minipile_bpe128000minipile

Updated 11 days ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me57M-tied_minipile_wordpiece32000minipile

Updated 11 days ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me57M-tied_minipile_bpe2wp32000minipile

Updated 11 days ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved every 2k steps. The bpe2wp nomenclature means that we choose the merges using the BPE objective, and we tokenised the MiniPile using the resulting vocabulary and the WordPiece tokenisation function (i.e., longest prefix match).

pietrolesci/me340M-tied_minipile_bpe32000minipile

Updated 11 days ago • 56

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me850M_minipile_bpe32000minipile

Updated 11 days ago • 54

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

Note Log-probabilities computed on the validation set of the MiniPile dataset using the models above.