w2v2-bert-Wolof-10-hours-ALFFA-dataset

This model is a fine-tuned version of facebook/w2v-bert-2.0 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5709
• Wer: 0.1743
• Cer: 0.0523

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
1.2343	1.5952	400	0.3992	0.3720	0.0987
0.5062	3.1904	800	0.3984	0.3519	0.0964
0.5063	4.7856	1200	0.5177	0.4211	0.1236
0.4841	6.3809	1600	0.5139	0.3837	0.1151
0.4184	7.9761	2000	0.4656	0.3576	0.1044
0.3573	9.5713	2400	0.4096	0.3080	0.0892
0.3076	11.1665	2800	0.3907	0.2882	0.0836
0.2697	12.7617	3200	0.4390	0.3265	0.0963
0.2364	14.3569	3600	0.3975	0.2941	0.0882
0.2081	15.9521	4000	0.3985	0.2907	0.0863
0.1724	17.5474	4400	0.3945	0.2676	0.0806
0.1502	19.1426	4800	0.4333	0.2634	0.0824
0.1239	20.7378	5200	0.3864	0.2283	0.0702
0.0988	22.3330	5600	0.3749	0.2349	0.0709
0.0832	23.9282	6000	0.3701	0.2270	0.0692
0.0662	25.5234	6400	0.3671	0.2215	0.0665
0.0553	27.1186	6800	0.4373	0.2151	0.0651
0.0436	28.7139	7200	0.4348	0.2153	0.0643
0.0344	30.3091	7600	0.4954	0.2245	0.0700
0.0271	31.9043	8000	0.3983	0.2007	0.0604
0.0196	33.4995	8400	0.4608	0.2199	0.0685
0.0182	35.0947	8800	0.4392	0.1948	0.0590
0.0115	36.6899	9200	0.4944	0.2078	0.0639
0.0104	38.2851	9600	0.4397	0.1910	0.0580
0.0065	39.8804	10000	0.4826	0.1827	0.0549
0.0061	41.4756	10400	0.4912	0.1836	0.0538
0.0045	43.0708	10800	0.4695	0.1859	0.0555
0.0026	44.6660	11200	0.5421	0.1834	0.0556
0.0018	46.2612	11600	0.5372	0.1799	0.0536
0.0008	47.8564	12000	0.5594	0.1768	0.0531
0.0005	49.4516	12400	0.5709	0.1743	0.0523

Framework versions

• Transformers 4.44.2
• Pytorch 2.1.0+cu118
• Datasets 2.17.0
• Tokenizers 0.19.1