mistral_o1_10_full

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.1 on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 0.6912

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
0.79	0.0622	100	0.8509
0.7592	0.1245	200	0.9030
0.7628	0.1867	300	0.8635
0.7546	0.2489	400	0.8601
0.7427	0.3111	500	0.8204
0.7268	0.3734	600	0.8335
0.7084	0.4356	700	0.8025
0.6958	0.4978	800	0.7765
0.6789	0.5600	900	0.7588
0.6506	0.6223	1000	0.7463
0.645	0.6845	1100	0.7306
0.6335	0.7467	1200	0.7130
0.615	0.8090	1300	0.7026
0.608	0.8712	1400	0.6949
0.604	0.9334	1500	0.6919
0.6039	0.9956	1600	0.6911

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.0
• Tokenizers 0.19.1