---
base_model: allenai/scibert_scivocab_uncased
tags:
- generated_from_trainer
model-index:
- name: nlp_te_mlm_scibert
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# nlp_te_mlm_scibert

This model is a fine-tuned version of [allenai/scibert_scivocab_uncased](https://huggingface.co/allenai/scibert_scivocab_uncased) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 1.1478

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 8
- seed: 5678
- gradient_accumulation_steps: 16
- total_train_batch_size: 256
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 500
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch   | Step | Validation Loss |
|:-------------:|:-------:|:----:|:---------------:|
| 1.3828        | 0.9963  | 152  | 1.2566          |
| 1.3087        | 1.9992  | 305  | 1.2295          |
| 1.289         | 2.9955  | 457  | 1.2237          |
| 1.262         | 3.9984  | 610  | 1.2054          |
| 1.2516        | 4.9947  | 762  | 1.1999          |
| 1.229         | 5.9975  | 915  | 1.1944          |
| 1.2272        | 6.9939  | 1067 | 1.1880          |
| 1.2066        | 7.9967  | 1220 | 1.1879          |
| 1.1991        | 8.9996  | 1373 | 1.1807          |
| 1.1978        | 9.9959  | 1525 | 1.1760          |
| 1.1803        | 10.9988 | 1678 | 1.1724          |
| 1.1819        | 11.9951 | 1830 | 1.1716          |
| 1.1659        | 12.9980 | 1983 | 1.1731          |
| 1.1658        | 13.9943 | 2135 | 1.1673          |
| 1.1524        | 14.9971 | 2288 | 1.1669          |
| 1.1481        | 16.0    | 2441 | 1.1590          |
| 1.1468        | 16.9963 | 2593 | 1.1626          |
| 1.1361        | 17.9992 | 2746 | 1.1623          |
| 1.1371        | 18.9955 | 2898 | 1.1582          |
| 1.125         | 19.9984 | 3051 | 1.1540          |
| 1.1276        | 20.9947 | 3203 | 1.1551          |
| 1.1143        | 21.9975 | 3356 | 1.1518          |
| 1.118         | 22.9939 | 3508 | 1.1550          |
| 1.104         | 23.9967 | 3661 | 1.1525          |
| 1.1011        | 24.9996 | 3814 | 1.1483          |
| 1.1061        | 25.9959 | 3966 | 1.1533          |
| 1.0941        | 26.9988 | 4119 | 1.1473          |
| 1.0951        | 27.9951 | 4271 | 1.1444          |
| 1.0866        | 28.9980 | 4424 | 1.1462          |
| 1.089         | 29.9943 | 4576 | 1.1453          |
| 1.0768        | 30.9971 | 4729 | 1.1496          |
| 1.0744        | 32.0    | 4882 | 1.1493          |
| 1.0773        | 32.9963 | 5034 | 1.1478          |


### Framework versions

- Transformers 4.41.2
- Pytorch 2.2.1
- Datasets 2.19.2
- Tokenizers 0.19.1