19a329a8-227f-4403-a594-52afbc5112e0

This model is a fine-tuned version of katuni4ka/tiny-random-qwen1.5-moe on the None dataset. It achieves the following results on the evaluation set:

• Loss: 11.7371

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.000204
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 500

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0000	1	11.9361
11.8371	0.0023	50	11.8614
11.6776	0.0047	100	11.7952
11.6579	0.0070	150	11.7724
11.6215	0.0094	200	11.7545
11.6344	0.0117	250	11.7462
11.6119	0.0141	300	11.7422
11.6267	0.0164	350	11.7389
11.6046	0.0188	400	11.7379
11.6064	0.0211	450	11.7372
11.5978	0.0235	500	11.7371

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1