Hugging Face's logo

AmedeoBonatti
/
nlp_te_mlm_scibert

Safetensors
bert
Generated from Trainer
Model card Files Community
nlp_te_mlm_scibert / README.md
AmedeoBonatti's picture
AmedeoBonatti
AmedeoBonatti/nlp_te_mlm_scibert
c488321 verified
preview code
|
raw
history blame
28.8 kB
metadata
base_model: allenai/scibert_scivocab_uncased
tags:
  - generated_from_trainer
model-index:
  - name: mlm_scibert_uncased
    results: []

mlm_scibert_uncased

This model is a fine-tuned version of allenai/scibert_scivocab_uncased on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 1.2966

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 16
  • eval_batch_size: 8
  • seed: 1234
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 256
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 500
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
1.3835 0.9963 152 1.2575
1.3098 1.9992 305 1.2320
1.2883 2.9955 457 1.2204
1.2612 3.9984 610 1.2113
1.2501 4.9947 762 1.2043
1.2292 5.9975 915 1.1910
1.2269 6.9939 1067 1.1859
1.2096 7.9967 1220 1.1841
1.1992 8.9996 1373 1.1772
1.1973 9.9959 1525 1.1806
1.1801 10.9988 1678 1.1720
1.1822 11.9951 1830 1.1699
1.1693 12.9980 1983 1.1674
1.1677 13.9943 2135 1.1641
1.1529 14.9971 2288 1.1597
1.1448 16.0 2441 1.1613
1.1499 16.9963 2593 1.1579
1.1363 17.9992 2746 1.1580
1.1369 18.9955 2898 1.1640
1.1247 19.9984 3051 1.1560
1.1283 20.9947 3203 1.1505
1.1173 21.9975 3356 1.1497
1.1171 22.9939 3508 1.1522
1.1064 23.9967 3661 1.1505
1.1034 24.9996 3814 1.1471
1.1051 25.9959 3966 1.1447
1.0949 26.9988 4119 1.1414
1.0975 27.9951 4271 1.1470
1.0852 28.9980 4424 1.1497
1.0888 29.9943 4576 1.1472
1.0779 30.9971 4729 1.1418
1.0741 32.0 4882 1.1430
1.0761 32.9963 5034 1.1448
1.0674 33.9992 5187 1.1447
1.0712 34.9955 5339 1.1451
1.0604 35.9984 5492 1.1440
1.0612 36.9947 5644 1.1423
1.0549 37.9975 5797 1.1460
1.0553 38.9939 5949 1.1456
1.0469 39.9967 6102 1.1436
1.0411 40.9996 6255 1.1401
1.0474 41.9959 6407 1.1395
1.0373 42.9988 6560 1.1423
1.0399 43.9951 6712 1.1442
1.0317 44.9980 6865 1.1443
1.0355 45.9943 7017 1.1427
1.0259 46.9971 7170 1.1424
1.0228 48.0 7323 1.1396
1.0285 48.9963 7475 1.1434
1.0179 49.9992 7628 1.1407
1.0209 50.9955 7780 1.1427
1.0132 51.9984 7933 1.1418
1.0159 52.9947 8085 1.1344
1.0058 53.9975 8238 1.1401
1.0113 54.9939 8390 1.1429
1.0021 55.9967 8543 1.1424
0.9995 56.9996 8696 1.1426
1.0048 57.9959 8848 1.1389
0.9951 58.9988 9001 1.1387
1.0011 59.9951 9153 1.1410
0.9901 60.9980 9306 1.1399
0.9925 61.9943 9458 1.1416
0.9835 62.9971 9611 1.1416
0.9846 64.0 9764 1.1458
0.9878 64.9963 9916 1.1452
0.9792 65.9992 10069 1.1459
0.9813 66.9955 10221 1.1415
0.9747 67.9984 10374 1.1476
0.9764 68.9947 10526 1.1474
0.971 69.9975 10679 1.1509
0.9728 70.9939 10831 1.1441
0.9672 71.9967 10984 1.1466
0.9627 72.9996 11137 1.1425
0.9678 73.9959 11289 1.1445
0.9609 74.9988 11442 1.1435
0.9636 75.9951 11594 1.1408
0.9553 76.9980 11747 1.1468
0.9608 77.9943 11899 1.1460
0.9506 78.9971 12052 1.1475
0.9505 80.0 12205 1.1460
0.9535 80.9963 12357 1.1467
0.9471 81.9992 12510 1.1495
0.9509 82.9955 12662 1.1484
0.9412 83.9984 12815 1.1486
0.9456 84.9947 12967 1.1468
0.9385 85.9975 13120 1.1497
0.945 86.9939 13272 1.1501
0.9351 87.9967 13425 1.1483
0.9324 88.9996 13578 1.1497
0.9376 89.9959 13730 1.1501
0.9295 90.9988 13883 1.1469
0.9345 91.9951 14035 1.1554
0.9267 92.9980 14188 1.1485
0.931 93.9943 14340 1.1508
0.9225 94.9971 14493 1.1536
0.9208 96.0 14646 1.1495
0.9254 96.9963 14798 1.1522
0.9177 97.9992 14951 1.1550
0.9199 98.9955 15103 1.1575
0.9144 99.9984 15256 1.1563
0.9174 100.9947 15408 1.1518
0.911 101.9975 15561 1.1560
0.9135 102.9939 15713 1.1543
0.9044 103.9967 15866 1.1549
0.905 104.9996 16019 1.1568
0.9106 105.9959 16171 1.1567
0.902 106.9988 16324 1.1555
0.9068 107.9951 16476 1.1580
0.8973 108.9980 16629 1.1562
0.9038 109.9943 16781 1.1612
0.8957 110.9971 16934 1.1514
0.8949 112.0 17087 1.1571
0.8989 112.9963 17239 1.1634
0.8927 113.9992 17392 1.1621
0.8954 114.9955 17544 1.1572
0.8876 115.9984 17697 1.1604
0.8917 116.9947 17849 1.1660
0.8841 117.9975 18002 1.1564
0.8893 118.9939 18154 1.1624
0.8808 119.9967 18307 1.1668
0.8825 120.9996 18460 1.1608
0.8848 121.9959 18612 1.1600
0.878 122.9988 18765 1.1650
0.8818 123.9951 18917 1.1671
0.8748 124.9980 19070 1.1668
0.8787 125.9943 19222 1.1605
0.8727 126.9971 19375 1.1649
0.8701 128.0 19528 1.1675
0.875 128.9963 19680 1.1639
0.8669 129.9992 19833 1.1698
0.8714 130.9955 19985 1.1726
0.8657 131.9984 20138 1.1680
0.8682 132.9947 20290 1.1695
0.8623 133.9975 20443 1.1774
0.8659 134.9939 20595 1.1718
0.8606 135.9967 20748 1.1691
0.8587 136.9996 20901 1.1668
0.8635 137.9959 21053 1.1742
0.8567 138.9988 21206 1.1707
0.8607 139.9951 21358 1.1756
0.8519 140.9980 21511 1.1742
0.8558 141.9943 21663 1.1733
0.8518 142.9971 21816 1.1761
0.85 144.0 21969 1.1734
0.8536 144.9963 22121 1.1788
0.8469 145.9992 22274 1.1782
0.85 146.9955 22426 1.1773
0.8416 147.9984 22579 1.1731
0.8496 148.9947 22731 1.1767
0.842 149.9975 22884 1.1743
0.8452 150.9939 23036 1.1778
0.8379 151.9967 23189 1.1843
0.8379 152.9996 23342 1.1804
0.8425 153.9959 23494 1.1803
0.8332 154.9988 23647 1.1818
0.8394 155.9951 23799 1.1805
0.8307 156.9980 23952 1.1841
0.836 157.9943 24104 1.1835
0.8305 158.9971 24257 1.1823
0.8298 160.0 24410 1.1768
0.8329 160.9963 24562 1.1836
0.8271 161.9992 24715 1.1841
0.8316 162.9955 24867 1.1848
0.825 163.9984 25020 1.1807
0.8287 164.9947 25172 1.1866
0.821 165.9975 25325 1.1866
0.8249 166.9939 25477 1.1887
0.8188 167.9967 25630 1.1882
0.8192 168.9996 25783 1.1891
0.8215 169.9959 25935 1.1921
0.8162 170.9988 26088 1.1891
0.8213 171.9951 26240 1.1929
0.8145 172.9980 26393 1.1881
0.8177 173.9943 26545 1.1878
0.8123 174.9971 26698 1.1919
0.8097 176.0 26851 1.1922
0.8156 176.9963 27003 1.1957
0.8077 177.9992 27156 1.1945
0.812 178.9955 27308 1.1942
0.8069 179.9984 27461 1.1913
0.8108 180.9947 27613 1.1962
0.8041 181.9975 27766 1.1992
0.8072 182.9939 27918 1.1976
0.8021 183.9967 28071 1.1981
0.8018 184.9996 28224 1.1958
0.8041 185.9959 28376 1.2022
0.7978 186.9988 28529 1.1981
0.8019 187.9951 28681 1.1957
0.7966 188.9980 28834 1.1995
0.7989 189.9943 28986 1.1947
0.7928 190.9971 29139 1.1966
0.7915 192.0 29292 1.2022
0.7975 192.9963 29444 1.2062
0.7918 193.9992 29597 1.2031
0.7952 194.9955 29749 1.2034
0.7894 195.9984 29902 1.2060
0.791 196.9947 30054 1.2040
0.7868 197.9975 30207 1.2054
0.7899 198.9939 30359 1.2046
0.7859 199.9967 30512 1.2023
0.7851 200.9996 30665 1.2075
0.7885 201.9959 30817 1.2074
0.7822 202.9988 30970 1.2052
0.7868 203.9951 31122 1.2048
0.7809 204.9980 31275 1.2070
0.7847 205.9943 31427 1.2096
0.7778 206.9971 31580 1.2082
0.7782 208.0 31733 1.2147
0.7813 208.9963 31885 1.2137
0.775 209.9992 32038 1.2115
0.7785 210.9955 32190 1.2203
0.7733 211.9984 32343 1.2108
0.7771 212.9947 32495 1.2173
0.7711 213.9975 32648 1.2123
0.7765 214.9939 32800 1.2156
0.77 215.9967 32953 1.2182
0.7673 216.9996 33106 1.2223
0.774 217.9959 33258 1.2144
0.7666 218.9988 33411 1.2144
0.7721 219.9951 33563 1.2165
0.7646 220.9980 33716 1.2195
0.769 221.9943 33868 1.2157
0.7625 222.9971 34021 1.2166
0.7619 224.0 34174 1.2171
0.7662 224.9963 34326 1.2183
0.7585 225.9992 34479 1.2243
0.764 226.9955 34631 1.2159
0.76 227.9984 34784 1.2215
0.7619 228.9947 34936 1.2161
0.758 229.9975 35089 1.2174
0.7613 230.9939 35241 1.2236
0.7547 231.9967 35394 1.2234
0.7562 232.9996 35547 1.2258
0.7572 233.9959 35699 1.2218
0.7514 234.9988 35852 1.2235
0.7559 235.9951 36004 1.2264
0.7515 236.9980 36157 1.2243
0.7555 237.9943 36309 1.2245
0.7497 238.9971 36462 1.2238
0.7467 240.0 36615 1.2260
0.7524 240.9963 36767 1.2251
0.7448 241.9992 36920 1.2267
0.7498 242.9955 37072 1.2293
0.7433 243.9984 37225 1.2358
0.7468 244.9947 37377 1.2337
0.7431 245.9975 37530 1.2285
0.7474 246.9939 37682 1.2304
0.7413 247.9967 37835 1.2341
0.7385 248.9996 37988 1.2318
0.7453 249.9959 38140 1.2336
0.7377 250.9988 38293 1.2301
0.7415 251.9951 38445 1.2303
0.7388 252.9980 38598 1.2327
0.7397 253.9943 38750 1.2364
0.7347 254.9971 38903 1.2324
0.7334 256.0 39056 1.2358
0.7407 256.9963 39208 1.2335
0.7322 257.9992 39361 1.2353
0.7354 258.9955 39513 1.2348
0.7287 259.9984 39666 1.2342
0.7351 260.9947 39818 1.2341
0.7294 261.9975 39971 1.2317
0.7321 262.9939 40123 1.2390
0.7278 263.9967 40276 1.2386
0.7264 264.9996 40429 1.2357
0.7303 265.9959 40581 1.2428
0.7254 266.9988 40734 1.2405
0.7273 267.9951 40886 1.2439
0.7248 268.9980 41039 1.2351
0.7293 269.9943 41191 1.2394
0.7217 270.9971 41344 1.2433
0.7212 272.0 41497 1.2461
0.7256 272.9963 41649 1.2419
0.7189 273.9992 41802 1.2393
0.7247 274.9955 41954 1.2442
0.7186 275.9984 42107 1.2400
0.7242 276.9947 42259 1.2433
0.7165 277.9975 42412 1.2464
0.7208 278.9939 42564 1.2397
0.7142 279.9967 42717 1.2488
0.7161 280.9996 42870 1.2467
0.7182 281.9959 43022 1.2499
0.7145 282.9988 43175 1.2444
0.7182 283.9951 43327 1.2507
0.7117 284.9980 43480 1.2477
0.715 285.9943 43632 1.2499
0.7122 286.9971 43785 1.2483
0.7101 288.0 43938 1.2442
0.7138 288.9963 44090 1.2497
0.7078 289.9992 44243 1.2477
0.7111 290.9955 44395 1.2485
0.7053 291.9984 44548 1.2483
0.7105 292.9947 44700 1.2529
0.7056 293.9975 44853 1.2566
0.7088 294.9939 45005 1.2476
0.7054 295.9967 45158 1.2536
0.704 296.9996 45311 1.2519
0.7082 297.9959 45463 1.2581
0.7009 298.9988 45616 1.2609
0.7052 299.9951 45768 1.2549
0.6984 300.9980 45921 1.2517
0.7056 301.9943 46073 1.2585
0.7002 302.9971 46226 1.2567
0.6981 304.0 46379 1.2573
0.7016 304.9963 46531 1.2585
0.6971 305.9992 46684 1.2632
0.7008 306.9955 46836 1.2587
0.6975 307.9984 46989 1.2580
0.6984 308.9947 47141 1.2535
0.6946 309.9975 47294 1.2576
0.6982 310.9939 47446 1.2610
0.6922 311.9967 47599 1.2632
0.694 312.9996 47752 1.2518
0.6967 313.9959 47904 1.2588
0.6895 314.9988 48057 1.2643
0.6954 315.9951 48209 1.2630
0.6899 316.9980 48362 1.2620
0.6932 317.9943 48514 1.2606
0.6878 318.9971 48667 1.2632
0.6895 320.0 48820 1.2623
0.6916 320.9963 48972 1.2665
0.6873 321.9992 49125 1.2636
0.6914 322.9955 49277 1.2631
0.6852 323.9984 49430 1.2631
0.6891 324.9947 49582 1.2628
0.6843 325.9975 49735 1.2654
0.6875 326.9939 49887 1.2656
0.6818 327.9967 50040 1.2660
0.683 328.9996 50193 1.2654
0.6866 329.9959 50345 1.2701
0.6803 330.9988 50498 1.2647
0.6843 331.9951 50650 1.2735
0.68 332.9980 50803 1.2663
0.6836 333.9943 50955 1.2659
0.6792 334.9971 51108 1.2723
0.6775 336.0 51261 1.2719
0.681 336.9963 51413 1.2684
0.6772 337.9992 51566 1.2722
0.6806 338.9955 51718 1.2745
0.6749 339.9984 51871 1.2762
0.6778 340.9947 52023 1.2767
0.6752 341.9975 52176 1.2727
0.6783 342.9939 52328 1.2757
0.6725 343.9967 52481 1.2732
0.6744 344.9996 52634 1.2728
0.6756 345.9959 52786 1.2736
0.6709 346.9988 52939 1.2731
0.6763 347.9951 53091 1.2749
0.6708 348.9980 53244 1.2774
0.673 349.9943 53396 1.2710
0.6685 350.9971 53549 1.2692
0.6677 352.0 53702 1.2675
0.6711 352.9963 53854 1.2767
0.6683 353.9992 54007 1.2760
0.6732 354.9955 54159 1.2743
0.6676 355.9984 54312 1.2797
0.6713 356.9947 54464 1.2764
0.6651 357.9975 54617 1.2807
0.6689 358.9939 54769 1.2758
0.6632 359.9967 54922 1.2839
0.6632 360.9996 55075 1.2807
0.6659 361.9959 55227 1.2760
0.6622 362.9988 55380 1.2812
0.6669 363.9951 55532 1.2761
0.6616 364.9980 55685 1.2868
0.6656 365.9943 55837 1.2766
0.6606 366.9971 55990 1.2851
0.659 368.0 56143 1.2815
0.665 368.9963 56295 1.2810
0.6585 369.9992 56448 1.2818
0.6636 370.9955 56600 1.2826
0.658 371.9984 56753 1.2799
0.6633 372.9947 56905 1.2915
0.657 373.9975 57058 1.2803
0.6623 374.9939 57210 1.2872
0.6561 375.9967 57363 1.2847
0.656 376.9996 57516 1.2834
0.6595 377.9959 57668 1.2858
0.6546 378.9988 57821 1.2834
0.6572 379.9951 57973 1.2869
0.653 380.9980 58126 1.2772
0.6566 381.9943 58278 1.2936
0.6533 382.9971 58431 1.2910
0.6543 384.0 58584 1.2846
0.6555 384.9963 58736 1.2881
0.6508 385.9992 58889 1.2898
0.6547 386.9955 59041 1.2879
0.6496 387.9984 59194 1.2865
0.6531 388.9947 59346 1.2861
0.6481 389.9975 59499 1.2832
0.6539 390.9939 59651 1.2895
0.6476 391.9967 59804 1.2838
0.6489 392.9996 59957 1.2923
0.6519 393.9959 60109 1.2871
0.647 394.9988 60262 1.2846
0.6491 395.9951 60414 1.2914
0.6459 396.9980 60567 1.2886
0.6496 397.9943 60719 1.2891
0.6452 398.9971 60872 1.2861
0.6439 400.0 61025 1.2917
0.6484 400.9963 61177 1.2934
0.6446 401.9992 61330 1.2872
0.6493 402.9955 61482 1.2900
0.6423 403.9984 61635 1.2940
0.6469 404.9947 61787 1.2867
0.6412 405.9975 61940 1.2958
0.6468 406.9939 62092 1.2906
0.6428 407.9967 62245 1.2904
0.6409 408.9996 62398 1.2924
0.6464 409.9959 62550 1.2953
0.6404 410.9988 62703 1.2918
0.6452 411.9951 62855 1.2894
0.6406 412.9980 63008 1.2975
0.6442 413.9943 63160 1.2928
0.638 414.9971 63313 1.2948
0.6379 416.0 63466 1.2936
0.6416 416.9963 63618 1.2892
0.639 417.9992 63771 1.2959
0.6414 418.9955 63923 1.2940
0.6363 419.9984 64076 1.2949
0.6409 420.9947 64228 1.2943
0.6346 421.9975 64381 1.2974
0.6393 422.9939 64533 1.3000
0.6331 423.9967 64686 1.2944
0.636 424.9996 64839 1.2915
0.6383 425.9959 64991 1.2986
0.6338 426.9988 65144 1.2981
0.6378 427.9951 65296 1.2980
0.634 428.9980 65449 1.2958
0.6374 429.9943 65601 1.2959
0.6312 430.9971 65754 1.2918
0.6317 432.0 65907 1.2972
0.6352 432.9963 66059 1.2970
0.6319 433.9992 66212 1.2969
0.6334 434.9955 66364 1.2997
0.6296 435.9984 66517 1.2967
0.6352 436.9947 66669 1.2979
0.6302 437.9975 66822 1.2999
0.6323 438.9939 66974 1.2989
0.6287 439.9967 67127 1.2933
0.6295 440.9996 67280 1.2979
0.6335 441.9959 67432 1.2979
0.6273 442.9988 67585 1.2917
0.6308 443.9951 67737 1.3001
0.6278 444.9980 67890 1.2948
0.6303 445.9943 68042 1.3005
0.6278 446.9971 68195 1.2962
0.6274 448.0 68348 1.2969
0.6287 448.9963 68500 1.2953
0.6276 449.9992 68653 1.2983
0.629 450.9955 68805 1.3040
0.6249 451.9984 68958 1.2992
0.6307 452.9947 69110 1.2992
0.626 453.9975 69263 1.2975
0.6283 454.9939 69415 1.2983
0.6262 455.9967 69568 1.3002
0.6217 456.9996 69721 1.3029
0.6284 457.9959 69873 1.3001
0.6238 458.9988 70026 1.3011
0.6258 459.9951 70178 1.2993
0.6217 460.9980 70331 1.2971
0.6265 461.9943 70483 1.2996
0.622 462.9971 70636 1.2977
0.6228 464.0 70789 1.2981
0.6274 464.9963 70941 1.3028
0.6218 465.9992 71094 1.2995
0.6245 466.9955 71246 1.2990
0.621 467.9984 71399 1.3032
0.6254 468.9947 71551 1.2992
0.6217 469.9975 71704 1.2964
0.6236 470.9939 71856 1.3012
0.6216 471.9967 72009 1.3004
0.6191 472.9996 72162 1.3032
0.6234 473.9959 72314 1.3043
0.6202 474.9988 72467 1.3015
0.6248 475.9951 72619 1.3018
0.6194 476.9980 72772 1.3030
0.6217 477.9943 72924 1.3040
0.6193 478.9971 73077 1.3058
0.6198 480.0 73230 1.2999
0.6219 480.9963 73382 1.3016
0.6165 481.9992 73535 1.3048
0.6223 482.9955 73687 1.3044
0.6165 483.9984 73840 1.3040
0.6223 484.9947 73992 1.3059
0.6179 485.9975 74145 1.2996
0.621 486.9939 74297 1.3052
0.6173 487.9967 74450 1.3019
0.6179 488.9996 74603 1.3009
0.6195 489.9959 74755 1.3023
0.6177 490.9988 74908 1.2976
0.6214 491.9951 75060 1.3044
0.6168 492.9980 75213 1.3022
0.6189 493.9943 75365 1.3029
0.6182 494.9971 75518 1.3043
0.6168 496.0 75671 1.3027
0.6222 496.9963 75823 1.3022
0.6155 497.9992 75976 1.3005
0.6144 498.1565 76000 1.2966

Framework versions

  • Transformers 4.41.2
  • Pytorch 2.2.1
  • Datasets 2.19.2
  • Tokenizers 0.19.1