|
--- |
|
license: apache-2.0 |
|
base_model: mistralai/Mistral-7B-v0.3 |
|
tags: |
|
- generated_from_trainer |
|
metrics: |
|
- accuracy |
|
language: |
|
- en |
|
datasets: |
|
- BEE-spoke-data/stepbasin-books |
|
--- |
|
|
|
[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/pszemraj/long-generation-tests/runs/o2w2acw6) |
|
# Mistral-7B-v0.3-stepbasin-books-20480 |
|
|
|
This model is a fine-tuned version of [mistralai/Mistral-7B-v0.3](https://huggingface.co/mistralai/Mistral-7B-v0.3) on [this dataset](https://github.com/stepbasin/books/tree/master/books) for the purpose of testing out super-long text generation. |
|
- fine-tuned at context length 20480, should consistently generate 8k+ tokens ([example](https://pastebin.com/raw/btzSQNVL)) |
|
|
|
|
|
--- |
|
|
|
It achieves the following results on the evaluation set: |
|
- Loss: 2.0784 |
|
- Accuracy: 0.5396 |
|
- Num Input Tokens Seen: 16384000 |
