mms-1b-bigcgen-male-20hrs-model

This model is a fine-tuned version of facebook/mms-1b-all on the BIGCGEN - BEM dataset. It achieves the following results on the evaluation set:

• Loss: 0.4819
• Wer: 0.4453

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 30.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
16.6224	0.0779	100	3.4136	1.0050
6.3668	0.1557	200	2.9155	1.0462
4.9805	0.2336	300	0.8884	0.6814
1.9813	0.3114	400	0.6719	0.5626
1.7501	0.3893	500	0.6440	0.5494
1.5396	0.4671	600	0.6134	0.5319
1.498	0.5450	700	0.6120	0.5162
1.475	0.6228	800	0.6036	0.5167
1.4093	0.7007	900	0.6058	0.5057
1.4896	0.7785	1000	0.6087	0.5121
1.3766	0.8564	1100	0.5707	0.5001
1.5787	0.9342	1200	0.5645	0.4907
1.5263	1.0117	1300	0.5231	0.4766
1.3527	1.0895	1400	0.5119	0.4780
1.3688	1.1674	1500	0.5115	0.4850
1.2191	1.2452	1600	0.5105	0.4746
1.1941	1.3231	1700	0.4952	0.4698
1.2454	1.4009	1800	0.4935	0.4780
1.2166	1.4788	1900	0.4931	0.4742
1.2543	1.5566	2000	0.4945	0.4609
1.1682	1.6345	2100	0.5006	0.4626
1.1835	1.7123	2200	0.4783	0.4655
1.1652	1.7902	2300	0.4827	0.4520
1.2329	1.8680	2400	0.4889	0.4549
1.1871	1.9459	2500	0.4753	0.4547
1.1469	2.0234	2600	0.4830	0.4532
1.1214	2.1012	2700	0.4740	0.4508
1.1185	2.1791	2800	0.4788	0.4516
1.1788	2.2569	2900	0.4865	0.4571
1.1214	2.3348	3000	0.4726	0.4434
1.1868	2.4126	3100	0.4846	0.4528
1.2368	2.4905	3200	0.4775	0.4335
1.214	2.5683	3300	0.4925	0.4422
1.1078	2.6462	3400	0.4820	0.4455

Framework versions

• Transformers 4.47.1
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0