metadata

library_name: transformers
tags:
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: llama_3b_step2_batch_v3
    results: []

llama_3b_step2_batch_v3

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

Model description

More information needed

More information needed

More information needed

The following hyperparameters were used during training:

learning_rate: 3e-05
train_batch_size: 4
eval_batch_size: 40
seed: 42
gradient_accumulation_steps: 4
total_train_batch_size: 16
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 2

Training Loss	Epoch	Step	Validation Loss
0.8424	0.0682	50	0.9705
0.8978	0.1363	100	0.7915
0.5886	0.2045	150	0.6863
0.7013	0.2727	200	0.6097
0.4795	0.3408	250	0.5510
0.5601	0.4090	300	0.5066
0.4968	0.4772	350	0.4675
0.3663	0.5453	400	0.4353
0.3754	0.6135	450	0.4074
0.3271	0.6817	500	0.3890
0.2731	0.7498	550	0.3744
0.3881	0.8180	600	0.3599
0.3295	0.8862	650	0.3489
0.3408	0.9543	700	0.3412
0.2966	1.0225	750	0.3418
0.2334	1.0907	800	0.3365
0.2457	1.1588	850	0.3321
0.2332	1.2270	900	0.3311
0.2279	1.2952	950	0.3285
0.2762	1.3633	1000	0.3245
0.1464	1.4315	1050	0.3233
0.2075	1.4997	1100	0.3211
0.2251	1.5678	1150	0.3184
0.1811	1.6360	1200	0.3167
0.2104	1.7042	1250	0.3164
0.224	1.7723	1300	0.3158
0.2483	1.8405	1350	0.3154
0.1773	1.9087	1400	0.3151
0.219	1.9768	1450	0.3151