|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
pipeline_tag: conversational |
|
tags: |
|
- not-for-all-audiences |
|
base_model: mistralai/Mistral-7B-Instruct-v0.2 |
|
--- |
|
|
|
Base: [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) |
|
|
|
Test model, do **not** use. |
|
|
|
It employs a different prompting format than the base's, and _not_ Alpaca. Not intended for |
|
public consumption yet, so no information will be given here in that regard. |
|
|
|
It's unlikely that the model will produce the intended outputs without the specific format it's |
|
been trained on. |
|
|
|
# Dataset |
|
Similar to LimaRP, but more niche. Flexible training sample length (from 4k to 32k tokens, |
|
at least). Might or might not be released in the future. |
|
|
|
# Training details |
|
## Hardware |
|
1x NVidia RTX 3090 24GB |
|
|
|
## Software |
|
[Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) |
|
|
|
## Training hyperparameters |
|
- load_in_4bit: true |
|
- adapter: qlora |
|
- sequence_len: 16384 |
|
- sample_packing: true |
|
- pad_to_sequence_len: false |
|
- gradient_accumulation_steps: 4 |
|
- micro_batch_size: 1 |
|
- eval_batch_size: 1 |
|
- num_epochs: 2 |
|
- optimizer: adamw_bnb_8bit |
|
- lr_scheduler: constant |
|
- learning_rate: 0.000085 |
|
- weight_decay: 0.05 |
|
- train_on_inputs: true |
|
- bf16: true |
|
- fp16: false |
|
- tf32: true |
|
- lora_r: 20 |
|
- lora_alpha: 16 |
|
- lora_dropout: 0.1 |
|
- lora_target_linear: true |
