---
license: apache-2.0
base_model: mistralai/Mistral-7B-v0.3
tags:
- generated_from_trainer
metrics:
- accuracy
language:
- en
datasets:
- BEE-spoke-data/stepbasin-books
---

[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/pszemraj/long-generation-tests/runs/o2w2acw6)
# Mistral-7B-v0.3-stepbasin-books-20480

This model is a fine-tuned version of [mistralai/Mistral-7B-v0.3](https://huggingface.co/mistralai/Mistral-7B-v0.3) on [this dataset](https://github.com/stepbasin/books/tree/master/books) for the purpose of testing out super-long text generation.
- fine-tuned at context length 20480, should consistently generate 8k+ tokens ([example](https://pastebin.com/raw/btzSQNVL))


---

It achieves the following results on the evaluation set:
- Loss: 2.0784
- Accuracy: 0.5396
- Num Input Tokens Seen: 16384000