collapse_gemma-2-2b_hs2_replace_iter15_sftsd1

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.5257
• Num Input Tokens Seen: 4546352

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 1
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.484	0.0511	5	1.2789	238752
0.9509	0.1021	10	1.2726	471216
0.5236	0.1532	15	1.4889	701272
0.305	0.2042	20	1.6939	944912
0.1397	0.2553	25	1.8924	1177984
0.0654	0.3063	30	2.1651	1414584
0.0448	0.3574	35	2.3155	1647904
0.0327	0.4084	40	2.3869	1880168
0.0255	0.4595	45	2.4274	2112992
0.0254	0.5105	50	2.4915	2339880
0.0288	0.5616	55	2.5298	2585520
0.0271	0.6126	60	2.5521	2817272
0.0216	0.6637	65	2.5396	3054120
0.0246	0.7147	70	2.5306	3288384
0.0237	0.7658	75	2.5104	3516200
0.0265	0.8168	80	2.4987	3753536
0.022	0.8679	85	2.5072	3992928
0.022	0.9190	90	2.5118	4222488
0.0232	0.9700	95	2.5225	4451528

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1