See axolotl config

axolotl version: 0.4.1

adapter: lora
base_model: fxmarty/really-tiny-falcon-testing
bf16: auto
chat_template: llama3
dataset_prepared_path: null
datasets:
- data_files:
  - 415dedee96180e11_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/415dedee96180e11_train_data.json
  type:
    field_instruction: instruction
    field_output: response
    format: '{instruction}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
debug: null
deepspeed: null
do_eval: true
early_stopping_patience: 3
eval_batch_size: 4
eval_max_new_tokens: 128
eval_steps: 500
evals_per_epoch: null
flash_attention: false
fp16: false
fsdp: null
fsdp_config: null
gradient_accumulation_steps: 8
gradient_checkpointing: true
group_by_length: true
hub_model_id: lesso02/561cc045-d57d-4d40-932e-390287a2eaac
hub_repo: null
hub_strategy: checkpoint
hub_token: null
learning_rate: 0.000202
load_in_4bit: false
load_in_8bit: false
local_rank: null
logging_steps: 50
lora_alpha: 128
lora_dropout: 0.15
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 64
lora_target_linear: true
lr_scheduler: cosine
max_grad_norm: 1.0
max_steps: 25000
micro_batch_size: 4
mlflow_experiment_name: /tmp/415dedee96180e11_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 10
optimizer: adamw_torch_fused
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
save_steps: 500
saves_per_epoch: null
seed: 20
sequence_len: 1024
strict: false
tf32: true
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.05
wandb_entity: null
wandb_mode: online
wandb_name: 5941f67d-1b56-4ae0-b76d-52a8681c66f9
wandb_project: 02a
wandb_run: your_name
wandb_runid: 5941f67d-1b56-4ae0-b76d-52a8681c66f9
warmup_steps: 100
weight_decay: 0.0
xformers_attention: null

561cc045-d57d-4d40-932e-390287a2eaac

This model is a fine-tuned version of fxmarty/really-tiny-falcon-testing on the None dataset. It achieves the following results on the evaluation set:

• Loss: 10.9172

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.000202
• train_batch_size: 4
• eval_batch_size: 4
• seed: 20
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• training_steps: 25000

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0002	1	11.0859
87.8237	0.1129	500	10.9719
87.7348	0.2258	1000	10.9600
87.6865	0.3386	1500	10.9512
87.6515	0.4515	2000	10.9449
87.6171	0.5644	2500	10.9406
87.5821	0.6773	3000	10.9366
87.5827	0.7902	3500	10.9341
87.5556	0.9031	4000	10.9321
87.5578	1.0159	4500	10.9300
87.5512	1.1288	5000	10.9284
87.5338	1.2417	5500	10.9271
87.5222	1.3546	6000	10.9265
87.5199	1.4675	6500	10.9258
87.5171	1.5804	7000	10.9247
87.5118	1.6932	7500	10.9238
87.5051	1.8061	8000	10.9233
87.508	1.9190	8500	10.9228
87.4979	2.0319	9000	10.9221
87.4973	2.1448	9500	10.9216
87.512	2.2577	10000	10.9213
87.4823	2.3705	10500	10.9210
87.4886	2.4834	11000	10.9207
87.4873	2.5963	11500	10.9202
87.4897	2.7092	12000	10.9200
87.4782	2.8221	12500	10.9196
87.4864	2.9350	13000	10.9196
87.4777	3.0478	13500	10.9194
87.4821	3.1607	14000	10.9189
87.479	3.2736	14500	10.9188
87.4648	3.3865	15000	10.9185
87.4757	3.4994	15500	10.9184
87.4546	3.6122	16000	10.9184
87.4722	3.7251	16500	10.9183
87.4617	3.8380	17000	10.9179
87.4607	3.9509	17500	10.9181
87.4602	4.0638	18000	10.9178
87.4577	4.1767	18500	10.9176
87.4592	4.2895	19000	10.9175
87.4784	4.4024	19500	10.9174
87.4644	4.5153	20000	10.9173
87.4677	4.6282	20500	10.9174
87.465	4.7411	21000	10.9174
87.4606	4.8540	21500	10.9172
87.4689	4.9668	22000	10.9173
87.4515	5.0797	22500	10.9172
87.4637	5.1926	23000	10.9172
87.4589	5.3055	23500	10.9172
87.4596	5.4184	24000	10.9172
87.461	5.5313	24500	10.9173
87.4639	5.6441	25000	10.9172

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1