See axolotl config
axolotl version: 0.4.1
adapter: qlora
auto_resume_from_checkpoints: false
base_model: bigscience/bloomz-560m
bf16: auto
chat_template: llama3
dataloader_num_workers: 12
dataset_prepared_path: null
datasets:
- data_files:
- 0fc9bc0ad0d49381_train_data.json
ds_type: json
format: custom
path: /workspace/input_data/0fc9bc0ad0d49381_train_data.json
type:
field_input: input
field_instruction: instruction
field_output: output
format: '{instruction} {input}'
no_input_format: '{instruction}'
system_format: '{system}'
system_prompt: ''
debug: null
deepspeed: null
early_stopping_patience: 4
eval_max_new_tokens: 128
eval_steps: 200
eval_table_size: null
evals_per_epoch: null
flash_attention: false
fp16: false
fsdp: null
fsdp_config: null
gradient_accumulation_steps: 4
gradient_checkpointing: true
group_by_length: true
hub_model_id: error577/6b3ffec3-8960-4633-9dd8-9344f00c879f
hub_repo: null
hub_strategy: checkpoint
hub_token: null
learning_rate: 0.0002
load_in_4bit: true
load_in_8bit: false
local_rank: null
logging_steps: 10
lora_alpha: 128
lora_dropout: 0.05
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 64
lora_target_linear: true
lr_scheduler: cosine
max_grad_norm: 1.0
max_steps: null
micro_batch_size: 2
mlflow_experiment_name: /tmp/0fc9bc0ad0d49381_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 3
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
save_steps: 200
sequence_len: 512
strict: false
tf32: false
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.002
wandb_entity: null
wandb_mode: online
wandb_name: 7af274cb-a9a3-45b8-b6c3-b1c837d298d4
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: 7af274cb-a9a3-45b8-b6c3-b1c837d298d4
warmup_steps: 10
weight_decay: 0.01
xformers_attention: null
6b3ffec3-8960-4633-9dd8-9344f00c879f
This model is a fine-tuned version of bigscience/bloomz-560m on the None dataset. It achieves the following results on the evaluation set:
- Loss: 2.2038
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 3
Training results
Training Loss | Epoch | Step | Validation Loss |
---|---|---|---|
No log | 0.0000 | 1 | 2.9006 |
9.4598 | 0.0032 | 200 | 2.4428 |
9.0193 | 0.0063 | 400 | 2.4032 |
9.1876 | 0.0095 | 600 | 2.3742 |
9.0006 | 0.0126 | 800 | 2.3595 |
8.471 | 0.0158 | 1000 | 2.3460 |
8.8708 | 0.0189 | 1200 | 2.3285 |
8.9666 | 0.0221 | 1400 | 2.3163 |
8.9781 | 0.0252 | 1600 | 2.3118 |
8.5027 | 0.0284 | 1800 | 2.3064 |
8.4862 | 0.0315 | 2000 | 2.2939 |
8.5331 | 0.0347 | 2200 | 2.2867 |
8.6053 | 0.0378 | 2400 | 2.2798 |
8.3577 | 0.0410 | 2600 | 2.2777 |
8.7329 | 0.0441 | 2800 | 2.2695 |
8.2758 | 0.0473 | 3000 | 2.2647 |
8.6138 | 0.0505 | 3200 | 2.2641 |
8.846 | 0.0536 | 3400 | 2.2620 |
8.4359 | 0.0568 | 3600 | 2.2576 |
8.7765 | 0.0599 | 3800 | 2.2510 |
8.6455 | 0.0631 | 4000 | 2.2499 |
8.6459 | 0.0662 | 4200 | 2.2417 |
8.6934 | 0.0694 | 4400 | 2.2431 |
8.8205 | 0.0725 | 4600 | 2.2390 |
8.5673 | 0.0757 | 4800 | 2.2372 |
8.3182 | 0.0788 | 5000 | 2.2328 |
8.8069 | 0.0820 | 5200 | 2.2327 |
8.3561 | 0.0851 | 5400 | 2.2285 |
8.4872 | 0.0883 | 5600 | 2.2258 |
8.9193 | 0.0914 | 5800 | 2.2191 |
8.6298 | 0.0946 | 6000 | 2.2243 |
8.764 | 0.0978 | 6200 | 2.2229 |
8.2044 | 0.1009 | 6400 | 2.2133 |
8.6036 | 0.1041 | 6600 | 2.2200 |
8.3937 | 0.1072 | 6800 | 2.2183 |
8.6775 | 0.1104 | 7000 | 2.2122 |
8.8183 | 0.1135 | 7200 | 2.2136 |
8.9629 | 0.1167 | 7400 | 2.2077 |
8.3259 | 0.1198 | 7600 | 2.2085 |
8.5166 | 0.1230 | 7800 | 2.2127 |
8.0724 | 0.1261 | 8000 | 2.2054 |
8.4386 | 0.1293 | 8200 | 2.2090 |
8.5342 | 0.1324 | 8400 | 2.2019 |
7.7389 | 0.1356 | 8600 | 2.2047 |
8.7464 | 0.1387 | 8800 | 2.2020 |
8.6213 | 0.1419 | 9000 | 2.2024 |
8.5823 | 0.1451 | 9200 | 2.2038 |
Framework versions
- PEFT 0.13.2
- Transformers 4.46.0
- Pytorch 2.5.0+cu124
- Datasets 3.0.1
- Tokenizers 0.20.1
- Downloads last month
- 1
Inference Providers
NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API:
The model has no pipeline_tag.
Model tree for error577/6b3ffec3-8960-4633-9dd8-9344f00c879f
Base model
bigscience/bloomz-560m