tohoku-nlp
/

tohokunlp-bert-500m-sq4096-alpha

Model card Files Files and versions Community

Onely7 commited on Nov 28, 2024

Commit

13f3c1f

·

verified ·

1 Parent(s): 1edc31c

Update README.md

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -90,7 +90,7 @@ Whole Word Masking 単語分割器には、[vibrato](https://github.com/daac-too
 | Batch Size (tokens) | 1146880 | 2293760 |
 | Max Learning Rate | 1.0E-4 | 1.0E-4 |
 | Min Learning Rate | 1.0E-6 | N/A |
-| Learning Rate Warmup Steps | 10000 | 10000 |
 | Scheduler | cosine | constant |
 | Optimizer | AdamW | AdamW |
 | Optimizer Config | beta_1 = 0.9, beta_2 = 0.999, eps = 1.0E-8 | beta_1 = 0.9, beta_2 = 0.999, eps = 1.0E-8 |
@@ -234,7 +234,7 @@ We only implemented Masked Language Modeling (MLM) during training, without Next
 | Batch Size (tokens) | 1146880 | 2293760 |
 | Max Learning Rate | 1.0E-4 | 1.0E-4 |
 | Min Learning Rate | 1.0E-6 | N/A |
-| Learning Rate Warmup Steps | 10000 | 10000 |
 | Scheduler | cosine | constant |
 | Optimizer | AdamW | AdamW |
 | Optimizer Config | beta_1 = 0.9, beta_2 = 0.999, eps = 1.0E-8 | beta_1 = 0.9, beta_2 = 0.999, eps = 1.0E-8 |

 | Batch Size (tokens) | 1146880 | 2293760 |
 | Max Learning Rate | 1.0E-4 | 1.0E-4 |
 | Min Learning Rate | 1.0E-6 | N/A |
+| Learning Rate Warmup Steps | 10000 | N/A |
 | Scheduler | cosine | constant |
 | Optimizer | AdamW | AdamW |
 | Optimizer Config | beta_1 = 0.9, beta_2 = 0.999, eps = 1.0E-8 | beta_1 = 0.9, beta_2 = 0.999, eps = 1.0E-8 |
 | Batch Size (tokens) | 1146880 | 2293760 |
 | Max Learning Rate | 1.0E-4 | 1.0E-4 |
 | Min Learning Rate | 1.0E-6 | N/A |
+| Learning Rate Warmup Steps | 10000 | N/A |
 | Scheduler | cosine | constant |
 | Optimizer | AdamW | AdamW |
 | Optimizer Config | beta_1 = 0.9, beta_2 = 0.999, eps = 1.0E-8 | beta_1 = 0.9, beta_2 = 0.999, eps = 1.0E-8 |