Hugging Face's logo

gokuls
/
mobilebert_sa_pre-training-complete

Fill-Mask
Transformers
PyTorch
TensorBoard
mobilebert
Generated from Trainer
Eval Results
Inference Endpoints
Model card Files Metrics Community
gokuls commited on
Commit
77306b1
·
1 Parent(s): e061222

update model card README.md

Browse files
Files changed (1) hide show
  1. README.md +55 -183
README.md CHANGED
@@ -13,13 +13,15 @@ model-index:
13
  name: Masked Language Modeling
14
  type: fill-mask
15
  dataset:
16
- name: wikitext wikitext-103-raw-v1
17
  type: wikitext
 
 
18
  args: wikitext-103-raw-v1
19
  metrics:
20
  - name: Accuracy
21
  type: accuracy
22
- value: 0.6427141769796747
23
  ---
24
 
25
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
@@ -27,10 +29,10 @@ should probably proofread and complete it, then remove this comment. -->
27
 
28
  # mobilebert_sa_pre-training-complete
29
 
30
- This model is a fine-tuned version of [google/mobilebert-uncased](https://huggingface.co/google/mobilebert-uncased) on the wikitext wikitext-103-raw-v1 dataset.
31
  It achieves the following results on the evaluation set:
32
- - Loss: nan
33
- - Accuracy: 0.6427
34
 
35
  ## Model description
36
 
@@ -50,196 +52,66 @@ More information needed
50
 
51
  The following hyperparameters were used during training:
52
  - learning_rate: 5e-05
53
- - train_batch_size: 64
54
- - eval_batch_size: 64
55
  - seed: 10
56
  - distributed_type: multi-GPU
57
- - num_devices: 2
58
- - total_train_batch_size: 128
59
- - total_eval_batch_size: 128
60
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
61
  - lr_scheduler_type: linear
62
  - lr_scheduler_warmup_steps: 100
63
  - training_steps: 300000
64
- - mixed_precision_training: Native AMP
65
 
66
  ### Training results
67
 
68
- | Training Loss | Epoch | Step | Validation Loss | Accuracy |
69
- |:-------------:|:------:|:------:|:---------------:|:--------:|
70
- | 0.0 | 1.0 | 1787 | nan | 0.6390 |
71
- | 0.0 | 2.0 | 3574 | nan | 0.6426 |
72
- | 0.0 | 3.0 | 5361 | nan | 0.6415 |
73
- | 0.0 | 4.0 | 7148 | nan | 0.6340 |
74
- | 0.0 | 5.0 | 8935 | nan | 0.6360 |
75
- | 0.0 | 6.0 | 10722 | nan | 0.6341 |
76
- | 0.0 | 7.0 | 12509 | nan | 0.6378 |
77
- | 0.0 | 8.0 | 14296 | nan | 0.6335 |
78
- | 0.0 | 9.0 | 16083 | nan | 0.6363 |
79
- | 0.0 | 10.0 | 17870 | nan | 0.6383 |
80
- | 0.0 | 11.0 | 19657 | nan | 0.6379 |
81
- | 0.0 | 12.0 | 21444 | nan | 0.6346 |
82
- | 0.0006 | 13.0 | 23231 | nan | 0.6409 |
83
- | 0.0 | 14.0 | 25018 | nan | 0.6406 |
84
- | 0.0 | 15.0 | 26805 | nan | 0.6323 |
85
- | 0.0 | 16.0 | 28592 | nan | 0.6402 |
86
- | 0.0 | 17.0 | 30379 | nan | 0.6400 |
87
- | 0.0 | 18.0 | 32166 | nan | 0.6328 |
88
- | 0.0 | 19.0 | 33953 | nan | 0.6352 |
89
- | 0.0 | 20.0 | 35740 | nan | 0.6380 |
90
- | 0.0 | 21.0 | 37527 | nan | 0.6463 |
91
- | 0.0 | 22.0 | 39314 | nan | 0.6313 |
92
- | 0.0 | 23.0 | 41101 | nan | 0.6386 |
93
- | 0.0 | 24.0 | 42888 | nan | 0.6413 |
94
- | 0.0 | 25.0 | 44675 | nan | 0.6323 |
95
- | 0.0008 | 26.0 | 46462 | nan | 0.6359 |
96
- | 0.0 | 27.0 | 48249 | nan | 0.6397 |
97
- | 0.0 | 28.0 | 50036 | nan | 0.6377 |
98
- | 0.0 | 29.0 | 51823 | nan | 0.6383 |
99
- | 0.0 | 30.0 | 53610 | nan | 0.6374 |
100
- | 0.0 | 31.0 | 55397 | nan | 0.6476 |
101
- | 0.0 | 32.0 | 57184 | nan | 0.6305 |
102
- | 0.0011 | 33.0 | 58971 | nan | 0.6451 |
103
- | 0.0 | 34.0 | 60758 | nan | 0.6372 |
104
- | 0.0 | 35.0 | 62545 | nan | 0.6368 |
105
- | 0.0006 | 36.0 | 64332 | nan | 0.6385 |
106
- | 0.0 | 37.0 | 66119 | nan | 0.6349 |
107
- | 0.0 | 38.0 | 67906 | nan | 0.6334 |
108
- | 0.0 | 39.0 | 69693 | nan | 0.6391 |
109
- | 0.0 | 40.0 | 71480 | nan | 0.6345 |
110
- | 0.0 | 41.0 | 73267 | nan | 0.6423 |
111
- | 0.0 | 42.0 | 75054 | nan | 0.6375 |
112
- | 0.0 | 43.0 | 76841 | nan | 0.6292 |
113
- | 0.0 | 44.0 | 78628 | nan | 0.6337 |
114
- | 0.0 | 45.0 | 80415 | nan | 0.6451 |
115
- | 0.0 | 46.0 | 82202 | nan | 0.6376 |
116
- | 0.0 | 47.0 | 83989 | nan | 0.6355 |
117
- | 0.0 | 48.0 | 85776 | nan | 0.6411 |
118
- | 0.0 | 49.0 | 87563 | nan | 0.6358 |
119
- | 0.0 | 50.0 | 89350 | nan | 0.6428 |
120
- | 0.0 | 51.0 | 91137 | nan | 0.6421 |
121
- | 0.004 | 52.0 | 92924 | nan | 0.6352 |
122
- | 0.0 | 53.0 | 94711 | nan | 0.6411 |
123
- | 0.0 | 54.0 | 96498 | nan | 0.6377 |
124
- | 0.0 | 55.0 | 98285 | nan | 0.6375 |
125
- | 0.0 | 56.0 | 100072 | nan | 0.6368 |
126
- | 0.0 | 57.0 | 101859 | nan | 0.6365 |
127
- | 0.0 | 58.0 | 103646 | nan | 0.6413 |
128
- | 0.0 | 59.0 | 105433 | nan | 0.6347 |
129
- | 0.0 | 60.0 | 107220 | nan | 0.6407 |
130
- | 0.0 | 61.0 | 109007 | nan | 0.6395 |
131
- | 0.0 | 62.0 | 110794 | nan | 0.6373 |
132
- | 0.0 | 63.0 | 112581 | nan | 0.6356 |
133
- | 0.0 | 64.0 | 114368 | nan | 0.6367 |
134
- | 0.0 | 65.0 | 116155 | nan | 0.6441 |
135
- | 0.0017 | 66.0 | 117942 | nan | 0.6380 |
136
- | 0.0 | 67.0 | 119729 | nan | 0.6348 |
137
- | 0.0 | 68.0 | 121516 | nan | 0.6356 |
138
- | 0.0 | 69.0 | 123303 | nan | 0.6391 |
139
- | 0.0006 | 70.0 | 125090 | nan | 0.6362 |
140
- | 0.0 | 71.0 | 126877 | nan | 0.6388 |
141
- | 0.0 | 72.0 | 128664 | nan | 0.6354 |
142
- | 0.0 | 73.0 | 130451 | nan | 0.6362 |
143
- | 0.0013 | 74.0 | 132238 | nan | 0.6347 |
144
- | 0.0 | 75.0 | 134025 | nan | 0.6327 |
145
- | 0.0 | 76.0 | 135812 | nan | 0.6382 |
146
- | 0.0 | 77.0 | 137599 | nan | 0.6411 |
147
- | 0.0 | 78.0 | 139386 | nan | 0.6404 |
148
- | 0.0 | 79.0 | 141173 | nan | 0.6392 |
149
- | 0.0 | 80.0 | 142960 | nan | 0.6404 |
150
- | 0.0 | 81.0 | 144747 | nan | 0.6421 |
151
- | 0.0 | 82.0 | 146534 | nan | 0.6364 |
152
- | 0.0 | 83.0 | 148321 | nan | 0.6364 |
153
- | 0.0 | 84.0 | 150108 | nan | 0.6370 |
154
- | 0.0 | 85.0 | 151895 | nan | 0.6357 |
155
- | 0.0 | 86.0 | 153682 | nan | 0.6353 |
156
- | 0.0 | 87.0 | 155469 | nan | 0.6393 |
157
- | 0.0 | 88.0 | 157256 | nan | 0.6397 |
158
- | 0.0006 | 89.0 | 159043 | nan | 0.6396 |
159
- | 0.0013 | 90.0 | 160830 | nan | 0.6378 |
160
- | 0.0 | 91.0 | 162617 | nan | 0.6386 |
161
- | 0.0 | 92.0 | 164404 | nan | 0.6415 |
162
- | 0.0 | 93.0 | 166191 | nan | 0.6342 |
163
- | 0.0 | 94.0 | 167978 | nan | 0.6356 |
164
- | 0.0 | 95.0 | 169765 | nan | 0.6410 |
165
- | 0.0 | 96.0 | 171552 | nan | 0.6366 |
166
- | 0.0 | 97.0 | 173339 | nan | 0.6329 |
167
- | 0.0013 | 98.0 | 175126 | nan | 0.6352 |
168
- | 0.0 | 99.0 | 176913 | nan | 0.6340 |
169
- | 0.0 | 100.0 | 178700 | nan | 0.6358 |
170
- | 0.0 | 101.0 | 180487 | nan | 0.6367 |
171
- | 0.0006 | 102.0 | 182274 | nan | 0.6368 |
172
- | 0.0 | 103.0 | 184061 | nan | 0.6353 |
173
- | 0.0 | 104.0 | 185848 | nan | 0.6370 |
174
- | 0.0 | 105.0 | 187635 | nan | 0.6333 |
175
- | 0.0 | 106.0 | 189422 | nan | 0.6316 |
176
- | 0.0006 | 107.0 | 191209 | nan | 0.6394 |
177
- | 0.0 | 108.0 | 192996 | nan | 0.6323 |
178
- | 0.0 | 109.0 | 194783 | nan | 0.6406 |
179
- | 0.0012 | 110.0 | 196570 | nan | 0.6331 |
180
- | 0.0 | 111.0 | 198357 | nan | 0.6398 |
181
- | 0.0 | 112.0 | 200144 | nan | 0.6402 |
182
- | 0.0 | 113.0 | 201931 | nan | 0.6345 |
183
- | 0.0 | 114.0 | 203718 | nan | 0.6416 |
184
- | 0.0 | 115.0 | 205505 | nan | 0.6352 |
185
- | 0.0 | 116.0 | 207292 | nan | 0.6357 |
186
- | 0.0032 | 117.0 | 209079 | nan | 0.6358 |
187
- | 0.0013 | 118.0 | 210866 | nan | 0.6406 |
188
- | 0.0 | 119.0 | 212653 | nan | 0.6354 |
189
- | 0.0 | 120.0 | 214440 | nan | 0.6345 |
190
- | 0.0 | 121.0 | 216227 | nan | 0.6433 |
191
- | 0.0 | 122.0 | 218014 | nan | 0.6326 |
192
- | 0.0 | 123.0 | 219801 | nan | 0.6358 |
193
- | 0.0 | 124.0 | 221588 | nan | 0.6409 |
194
- | 0.0 | 125.0 | 223375 | nan | 0.6405 |
195
- | 0.0 | 126.0 | 225162 | nan | 0.6376 |
196
- | 0.0 | 127.0 | 226949 | nan | 0.6396 |
197
- | 0.0 | 128.0 | 228736 | nan | 0.6356 |
198
- | 0.0 | 129.0 | 230523 | nan | 0.6432 |
199
- | 0.0 | 130.0 | 232310 | nan | 0.6385 |
200
- | 0.0 | 131.0 | 234097 | nan | 0.6337 |
201
- | 0.0 | 132.0 | 235884 | nan | 0.6390 |
202
- | 0.0 | 133.0 | 237671 | nan | 0.6362 |
203
- | 0.0 | 134.0 | 239458 | nan | 0.6332 |
204
- | 0.0 | 135.0 | 241245 | nan | 0.6367 |
205
- | 0.0016 | 136.0 | 243032 | nan | 0.6334 |
206
- | 0.0 | 137.0 | 244819 | nan | 0.6412 |
207
- | 0.0 | 138.0 | 246606 | nan | 0.6367 |
208
- | 0.0 | 139.0 | 248393 | nan | 0.6378 |
209
- | 0.0 | 140.0 | 250180 | nan | 0.6390 |
210
- | 0.0 | 141.0 | 251967 | nan | 0.6376 |
211
- | 0.0 | 142.0 | 253754 | nan | 0.6363 |
212
- | 0.0033 | 143.0 | 255541 | nan | 0.6425 |
213
- | 0.0 | 144.0 | 257328 | nan | 0.6360 |
214
- | 0.0 | 145.0 | 259115 | nan | 0.6377 |
215
- | 0.0 | 146.0 | 260902 | nan | 0.6302 |
216
- | 0.0 | 147.0 | 262689 | nan | 0.6320 |
217
- | 0.0 | 148.0 | 264476 | nan | 0.6358 |
218
- | 0.0 | 149.0 | 266263 | nan | 0.6381 |
219
- | 0.0 | 150.0 | 268050 | nan | 0.6414 |
220
- | 0.0 | 151.0 | 269837 | nan | 0.6401 |
221
- | 0.0012 | 152.0 | 271624 | nan | 0.6415 |
222
- | 0.0 | 153.0 | 273411 | nan | 0.6425 |
223
- | 0.0 | 154.0 | 275198 | nan | 0.6367 |
224
- | 0.0 | 155.0 | 276985 | nan | 0.6356 |
225
- | 0.0 | 156.0 | 278772 | nan | 0.6411 |
226
- | 0.0 | 157.0 | 280559 | nan | 0.6343 |
227
- | 0.0007 | 158.0 | 282346 | nan | 0.6369 |
228
- | 0.0 | 159.0 | 284133 | nan | 0.6361 |
229
- | 0.0013 | 160.0 | 285920 | nan | 0.6396 |
230
- | 0.0008 | 161.0 | 287707 | nan | 0.6381 |
231
- | 0.0 | 162.0 | 289494 | nan | 0.6352 |
232
- | 0.0 | 163.0 | 291281 | nan | 0.6370 |
233
- | 0.0 | 164.0 | 293068 | nan | 0.6399 |
234
- | 0.0031 | 165.0 | 294855 | nan | 0.6401 |
235
- | 0.0 | 166.0 | 296642 | nan | 0.6358 |
236
- | 0.0 | 167.0 | 298429 | nan | 0.6390 |
237
- | 0.0 | 167.88 | 300000 | nan | 0.6354 |
238
 
239
 
240
  ### Framework versions
241
 
242
- - Transformers 4.25.1
243
  - Pytorch 1.14.0a0+410ce96
244
- - Datasets 2.8.0
245
  - Tokenizers 0.13.2
 
13
  name: Masked Language Modeling
14
  type: fill-mask
15
  dataset:
16
+ name: wikitext
17
  type: wikitext
18
+ config: wikitext-103-raw-v1
19
+ split: validation
20
  args: wikitext-103-raw-v1
21
  metrics:
22
  - name: Accuracy
23
  type: accuracy
24
+ value: 0.7186174960946218
25
  ---
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 
29
 
30
  # mobilebert_sa_pre-training-complete
31
 
32
+ This model is a fine-tuned version of [google/mobilebert-uncased](https://huggingface.co/google/mobilebert-uncased) on the wikitext dataset.
33
  It achieves the following results on the evaluation set:
34
+ - Loss: 1.3074
35
+ - Accuracy: 0.7186
36
 
37
  ## Model description
38
 
 
52
 
53
  The following hyperparameters were used during training:
54
  - learning_rate: 5e-05
55
+ - train_batch_size: 32
56
+ - eval_batch_size: 32
57
  - seed: 10
58
  - distributed_type: multi-GPU
 
 
 
59
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
60
  - lr_scheduler_type: linear
61
  - lr_scheduler_warmup_steps: 100
62
  - training_steps: 300000
 
63
 
64
  ### Training results
65
 
66
+ | Training Loss | Epoch | Step | Validation Loss | Accuracy |
67
+ |:-------------:|:-----:|:------:|:---------------:|:--------:|
68
+ | 1.6028 | 1.0 | 7145 | 1.4525 | 0.6935 |
69
+ | 1.5524 | 2.0 | 14290 | 1.4375 | 0.6993 |
70
+ | 1.5323 | 3.0 | 21435 | 1.4194 | 0.6993 |
71
+ | 1.5191 | 4.0 | 28580 | 1.4110 | 0.7027 |
72
+ | 1.5025 | 5.0 | 35725 | 1.4168 | 0.7014 |
73
+ | 1.4902 | 6.0 | 42870 | 1.3931 | 0.7012 |
74
+ | 1.4813 | 7.0 | 50015 | 1.3738 | 0.7057 |
75
+ | 1.4751 | 8.0 | 57160 | 1.4237 | 0.6996 |
76
+ | 1.4689 | 9.0 | 64305 | 1.3969 | 0.7047 |
77
+ | 1.4626 | 10.0 | 71450 | 1.3916 | 0.7068 |
78
+ | 1.4566 | 11.0 | 78595 | 1.3686 | 0.7072 |
79
+ | 1.451 | 12.0 | 85740 | 1.3811 | 0.7060 |
80
+ | 1.4478 | 13.0 | 92885 | 1.3598 | 0.7092 |
81
+ | 1.4441 | 14.0 | 100030 | 1.3790 | 0.7054 |
82
+ | 1.4379 | 15.0 | 107175 | 1.3794 | 0.7066 |
83
+ | 1.4353 | 16.0 | 114320 | 1.3609 | 0.7102 |
84
+ | 1.43 | 17.0 | 121465 | 1.3685 | 0.7083 |
85
+ | 1.4278 | 18.0 | 128610 | 1.3953 | 0.7036 |
86
+ | 1.4219 | 19.0 | 135755 | 1.3756 | 0.7085 |
87
+ | 1.4197 | 20.0 | 142900 | 1.3597 | 0.7090 |
88
+ | 1.4169 | 21.0 | 150045 | 1.3673 | 0.7061 |
89
+ | 1.4146 | 22.0 | 157190 | 1.3753 | 0.7073 |
90
+ | 1.4109 | 23.0 | 164335 | 1.3696 | 0.7082 |
91
+ | 1.4073 | 24.0 | 171480 | 1.3563 | 0.7092 |
92
+ | 1.4054 | 25.0 | 178625 | 1.3712 | 0.7103 |
93
+ | 1.402 | 26.0 | 185770 | 1.3528 | 0.7113 |
94
+ | 1.4001 | 27.0 | 192915 | 1.3367 | 0.7123 |
95
+ | 1.397 | 28.0 | 200060 | 1.3508 | 0.7118 |
96
+ | 1.3955 | 29.0 | 207205 | 1.3572 | 0.7117 |
97
+ | 1.3937 | 30.0 | 214350 | 1.3566 | 0.7095 |
98
+ | 1.3901 | 31.0 | 221495 | 1.3515 | 0.7117 |
99
+ | 1.3874 | 32.0 | 228640 | 1.3445 | 0.7118 |
100
+ | 1.386 | 33.0 | 235785 | 1.3611 | 0.7097 |
101
+ | 1.3833 | 34.0 | 242930 | 1.3502 | 0.7087 |
102
+ | 1.3822 | 35.0 | 250075 | 1.3657 | 0.7108 |
103
+ | 1.3797 | 36.0 | 257220 | 1.3576 | 0.7108 |
104
+ | 1.3793 | 37.0 | 264365 | 1.3472 | 0.7106 |
105
+ | 1.3763 | 38.0 | 271510 | 1.3323 | 0.7156 |
106
+ | 1.3762 | 39.0 | 278655 | 1.3325 | 0.7145 |
107
+ | 1.3748 | 40.0 | 285800 | 1.3243 | 0.7138 |
108
+ | 1.3733 | 41.0 | 292945 | 1.3218 | 0.7170 |
109
+ | 1.3722 | 41.99 | 300000 | 1.3074 | 0.7186 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
110
 
111
 
112
  ### Framework versions
113
 
114
+ - Transformers 4.26.0
115
  - Pytorch 1.14.0a0+410ce96
116
+ - Datasets 2.9.0
117
  - Tokenizers 0.13.2