15920e41-1a56-4e7e-b41d-e31c122b6d3f

This model is a fine-tuned version of katuni4ka/tiny-random-qwen1.5-moe on the None dataset. It achieves the following results on the evaluation set:

• Loss: 11.7366

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.000217
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 500

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0000	1	11.9361
11.8357	0.0023	50	11.8614
11.6733	0.0047	100	11.7928
11.6565	0.0070	150	11.7695
11.6208	0.0094	200	11.7531
11.6343	0.0117	250	11.7456
11.6128	0.0141	300	11.7412
11.6262	0.0164	350	11.7387
11.6039	0.0188	400	11.7371
11.6068	0.0211	450	11.7367
11.5984	0.0235	500	11.7366

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1