w2v2-bert-wolof-mixed-75-hours

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the fleurs dataset. It achieves the following results on the evaluation set:

• Loss: 0.7945
• Model Preparation Time: 0.016
• Wer: 0.4378
• Cer: 0.1508

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss	Model Preparation Time	Wer	Cer
11.7766	0.3478	400	1.1257	0.016	0.6985	0.2268
5.0043	0.6957	800	0.9996	0.016	0.5908	0.2027
4.3873	1.0435	1200	0.9269	0.016	0.5708	0.1977
4.0347	1.3913	1600	0.9035	0.016	0.5655	0.1958
4.0726	1.7391	2000	1.0003	0.016	0.5852	0.2072
4.088	2.0870	2400	0.9824	0.016	0.5923	0.2199
4.0292	2.4348	2800	1.1359	0.016	0.6165	0.2262
4.0772	2.7826	3200	0.9909	0.016	0.6224	0.2381
4.1632	3.1304	3600	1.0027	0.016	0.6092	0.2172
4.2807	3.4783	4000	0.8705	0.016	0.5979	0.2087
4.2877	3.8261	4400	0.9515	0.016	0.6289	0.2395
4.2801	4.1739	4800	1.2157	0.016	0.6942	0.2623
4.4405	4.5217	5200	1.1690	0.016	0.7471	0.2766
4.5465	4.8696	5600	1.1639	0.016	0.8119	0.2750
4.5337	5.2174	6000	1.0600	0.016	0.6861	0.2694
4.5306	5.5652	6400	1.1896	0.016	0.8158	0.3098
4.4503	5.9130	6800	1.1112	0.016	0.7378	0.2844
4.3858	6.2609	7200	1.0705	0.016	0.6733	0.2545
4.3818	6.6087	7600	1.0729	0.016	0.7716	0.2802
4.2016	6.9565	8000	1.0411	0.016	0.6635	0.2514
4.1929	7.3043	8400	0.9196	0.016	0.6179	0.2297
4.2011	7.6522	8800	1.0356	0.016	0.6759	0.2732
4.1664	8.0	9200	1.0187	0.016	0.6679	0.2527
4.0612	8.3478	9600	1.0575	0.016	0.6697	0.2636
4.1004	8.6957	10000	1.0145	0.016	0.6989	0.2559
3.993	9.0435	10400	0.8966	0.016	0.6262	0.2293
3.946	9.3913	10800	0.9941	0.016	0.6476	0.2502
3.9723	9.7391	11200	0.8798	0.016	0.6116	0.2295
3.9375	10.0870	11600	1.0265	0.016	0.6918	0.2732
3.7855	10.4348	12000	0.9268	0.016	0.6131	0.2301
3.789	10.7826	12400	0.9128	0.016	0.6623	0.2470
3.6762	11.1304	12800	0.9188	0.016	0.6569	0.2459
3.6208	11.4783	13200	0.8584	0.016	0.6096	0.2302
3.7174	11.8261	13600	0.8551	0.016	0.6059	0.2367
3.5768	12.1739	14000	0.8994	0.016	0.5999	0.2301
3.4273	12.5217	14400	0.7724	0.016	0.5492	0.2075
3.5209	12.8696	14800	0.9623	0.016	0.6294	0.2384
3.5119	13.2174	15200	0.8470	0.016	0.6077	0.2274
3.2817	13.5652	15600	0.8757	0.016	0.6172	0.2295
3.4257	13.9130	16000	0.8080	0.016	0.5828	0.2159
3.2079	14.2609	16400	0.7876	0.016	0.5527	0.2005
3.2496	14.6087	16800	0.8186	0.016	0.5579	0.2107
3.2356	14.9565	17200	0.8017	0.016	0.5594	0.2105
3.1111	15.3043	17600	0.7971	0.016	0.5397	0.2017
3.1498	15.6522	18000	0.8149	0.016	0.5525	0.2098
3.0901	16.0	18400	0.7434	0.016	0.5560	0.2040
3.0073	16.3478	18800	0.7900	0.016	0.5631	0.2137
3.0489	16.6957	19200	0.9481	0.016	0.6334	0.2441
3.0164	17.0435	19600	0.7279	0.016	0.5256	0.1921
2.8813	17.3913	20000	0.7843	0.016	0.5331	0.2013
2.9653	17.7391	20400	0.7391	0.016	0.5340	0.2007
2.9241	18.0870	20800	0.8120	0.016	0.5337	0.1913
2.7475	18.4348	21200	0.7364	0.016	0.5384	0.1968
2.776	18.7826	21600	0.7634	0.016	0.5410	0.2051
2.8009	19.1304	22000	0.7331	0.016	0.5202	0.1881
2.6983	19.4783	22400	0.7233	0.016	0.5351	0.1909
2.7007	19.8261	22800	0.7191	0.016	0.5467	0.1962
2.6292	20.1739	23200	0.7249	0.016	0.5271	0.1958
2.6153	20.5217	23600	0.6891	0.016	0.5055	0.1841
2.6886	20.8696	24000	0.7109	0.016	0.5089	0.1896
2.5751	21.2174	24400	0.7151	0.016	0.5332	0.2115
2.5319	21.5652	24800	0.6984	0.016	0.5198	0.1884
2.5264	21.9130	25200	0.7594	0.016	0.5444	0.2035
2.4224	22.2609	25600	0.7153	0.016	0.4997	0.1845
2.5018	22.6087	26000	0.7060	0.016	0.5174	0.1907
2.4878	22.9565	26400	0.7114	0.016	0.5066	0.1898
2.3717	23.3043	26800	0.6762	0.016	0.5034	0.1839
2.3977	23.6522	27200	0.6972	0.016	0.5258	0.1869
2.3516	24.0	27600	0.7925	0.016	0.5279	0.1927
2.3072	24.3478	28000	0.7104	0.016	0.5046	0.1869
2.3526	24.6957	28400	0.8242	0.016	0.5398	0.1983
2.2715	25.0435	28800	0.7089	0.016	0.4961	0.1764
2.2415	25.3913	29200	0.7396	0.016	0.5289	0.1969
2.2458	25.7391	29600	0.7099	0.016	0.4936	0.1806
2.1919	26.0870	30000	0.6836	0.016	0.4767	0.1733
2.1406	26.4348	30400	0.6758	0.016	0.4955	0.1787
2.143	26.7826	30800	0.6664	0.016	0.4916	0.1751
2.1006	27.1304	31200	0.6869	0.016	0.4763	0.1703
2.0706	27.4783	31600	0.6942	0.016	0.5194	0.1851
2.0959	27.8261	32000	0.7162	0.016	0.5272	0.1861
2.0479	28.1739	32400	0.6295	0.016	0.4727	0.1714
1.9625	28.5217	32800	0.6404	0.016	0.4808	0.1740
1.9959	28.8696	33200	0.7074	0.016	0.5058	0.1824
1.9929	29.2174	33600	0.7636	0.016	0.5064	0.1882
1.9295	29.5652	34000	0.6882	0.016	0.4846	0.1755
1.9403	29.9130	34400	0.6595	0.016	0.4816	0.1734
1.8859	30.2609	34800	0.7068	0.016	0.4820	0.1721
1.877	30.6087	35200	0.6906	0.016	0.4850	0.1763
1.892	30.9565	35600	0.6596	0.016	0.4708	0.1654
1.8199	31.3043	36000	0.6752	0.016	0.4585	0.1603
1.8226	31.6522	36400	0.7320	0.016	0.4773	0.1722
1.7716	32.0	36800	0.6807	0.016	0.4622	0.1670
1.7435	32.3478	37200	0.7207	0.016	0.4947	0.1804
1.7791	32.6957	37600	0.6985	0.016	0.4693	0.1681
1.7286	33.0435	38000	0.7014	0.016	0.4817	0.1697
1.7015	33.3913	38400	0.6615	0.016	0.4859	0.1731
1.6884	33.7391	38800	0.6528	0.016	0.4695	0.1640
1.7252	34.0870	39200	0.6669	0.016	0.4698	0.1658
1.6353	34.4348	39600	0.6878	0.016	0.4908	0.1732
1.655	34.7826	40000	0.7076	0.016	0.4692	0.1655
1.6267	35.1304	40400	0.6742	0.016	0.4608	0.1627
1.5937	35.4783	40800	0.6456	0.016	0.4615	0.1590
1.5733	35.8261	41200	0.6525	0.016	0.4724	0.1629
1.58	36.1739	41600	0.6382	0.016	0.4600	0.1602
1.5158	36.5217	42000	0.6538	0.016	0.4499	0.1564
1.5284	36.8696	42400	0.6573	0.016	0.4508	0.1583
1.5195	37.2174	42800	0.6452	0.016	0.4597	0.1578
1.4839	37.5652	43200	0.6700	0.016	0.4572	0.1617
1.4793	37.9130	43600	0.6841	0.016	0.4668	0.1613
1.4587	38.2609	44000	0.7116	0.016	0.4542	0.1603
1.4233	38.6087	44400	0.6758	0.016	0.4472	0.1571
1.4962	38.9565	44800	0.6654	0.016	0.4555	0.1566
1.3871	39.3043	45200	0.6974	0.016	0.4573	0.1586
1.3926	39.6522	45600	0.6906	0.016	0.4704	0.1622
1.3852	40.0	46000	0.6375	0.016	0.4503	0.1572
1.3408	40.3478	46400	0.7045	0.016	0.4586	0.1613
1.3603	40.6957	46800	0.6581	0.016	0.4613	0.1596
1.3423	41.0435	47200	0.7167	0.016	0.4520	0.1562
1.296	41.3913	47600	0.6804	0.016	0.4517	0.1573
1.2604	41.7391	48000	0.7131	0.016	0.4414	0.1530
1.2873	42.0870	48400	0.7258	0.016	0.4509	0.1559
1.237	42.4348	48800	0.6957	0.016	0.4418	0.1525
1.2699	42.7826	49200	0.7115	0.016	0.4394	0.1563
1.2036	43.1304	49600	0.7259	0.016	0.4453	0.1539
1.226	43.4783	50000	0.7238	0.016	0.4331	0.1513
1.213	43.8261	50400	0.7210	0.016	0.4420	0.1530
1.1345	44.1739	50800	0.7782	0.016	0.4403	0.1529
1.1684	44.5217	51200	0.7361	0.016	0.4438	0.1536
1.214	44.8696	51600	0.7518	0.016	0.4407	0.1521
1.1141	45.2174	52000	0.7633	0.016	0.4443	0.1542
1.1284	45.5652	52400	0.7520	0.016	0.4398	0.1522
1.1172	45.9130	52800	0.7339	0.016	0.4446	0.1535
1.0723	46.2609	53200	0.7706	0.016	0.4387	0.1517
1.0499	46.6087	53600	0.7504	0.016	0.4382	0.1512
1.1076	46.9565	54000	0.7552	0.016	0.4460	0.1527
1.0547	47.3043	54400	0.7656	0.016	0.4397	0.1509
1.0231	47.6522	54800	0.7663	0.016	0.4409	0.1512
1.0573	48.0	55200	0.7506	0.016	0.4373	0.1502
1.0242	48.3478	55600	0.7745	0.016	0.4386	0.1497
1.0129	48.6957	56000	0.7814	0.016	0.4374	0.1505
0.975	49.0435	56400	0.7876	0.016	0.4392	0.1507
1.0072	49.3913	56800	0.7855	0.016	0.4408	0.1513
0.971	49.7391	57200	0.7945	0.016	0.4378	0.1508

Framework versions

• Transformers 4.46.1
• Pytorch 2.5.1+cu124
• Datasets 2.17.0
• Tokenizers 0.20.3