Hugging Face's logo

aiface
/
CafeBERT_massive_crf_v2

Transformers
Safetensors
Generated from Trainer
Model card Files
xet
Community
aiface commited on
Commit
f78aa93
·
verified ·
1 Parent(s): 7142ead

Model save

Browse files
Files changed (5) hide show
  1. README.md +80 -0
  2. intent_report_test.txt +75 -0
  3. model.safetensors +1 -1
  4. model_predict_test.csv +0 -0
  5. slot_report_test.txt +60 -0
README.md ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ license: apache-2.0
4
+ base_model: uitnlp/CafeBERT
5
+ tags:
6
+ - generated_from_trainer
7
+ model-index:
8
+ - name: CafeBERT_massive_crf_v2
9
+ results: []
10
+ ---
11
+
12
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
+ should probably proofread and complete it, then remove this comment. -->
14
+
15
+ # CafeBERT_massive_crf_v2
16
+
17
+ This model is a fine-tuned version of [uitnlp/CafeBERT](https://huggingface.co/uitnlp/CafeBERT) on an unknown dataset.
18
+ It achieves the following results on the evaluation set:
19
+ - Loss: 5.5963
20
+ - Slot P: 0.0077
21
+ - Slot R: 0.0082
22
+ - Slot F1: 0.0079
23
+ - Slot Exact Match: 0.3246
24
+ - Intent Acc: 0.8751
25
+
26
+ ## Model description
27
+
28
+ More information needed
29
+
30
+ ## Intended uses & limitations
31
+
32
+ More information needed
33
+
34
+ ## Training and evaluation data
35
+
36
+ More information needed
37
+
38
+ ## Training procedure
39
+
40
+ ### Training hyperparameters
41
+
42
+ The following hyperparameters were used during training:
43
+ - learning_rate: 5e-05
44
+ - train_batch_size: 128
45
+ - eval_batch_size: 128
46
+ - seed: 42
47
+ - gradient_accumulation_steps: 2
48
+ - total_train_batch_size: 256
49
+ - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
50
+ - lr_scheduler_type: cosine
51
+ - lr_scheduler_warmup_ratio: 0.06
52
+ - num_epochs: 30
53
+ - mixed_precision_training: Native AMP
54
+
55
+ ### Training results
56
+
57
+ | Training Loss | Epoch | Step | Validation Loss | Slot P | Slot R | Slot F1 | Slot Exact Match | Intent Acc |
58
+ |:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:-------:|:----------------:|:----------:|
59
+ | No log | 1.0 | 45 | 15.8863 | 0.0 | 0.0 | 0.0 | 0.4088 | 0.0733 |
60
+ | 69.6998 | 2.0 | 90 | 7.8039 | 0.0080 | 0.0076 | 0.0078 | 0.3438 | 0.5032 |
61
+ | 22.3589 | 3.0 | 135 | 4.5345 | 0.0091 | 0.0100 | 0.0095 | 0.3227 | 0.7846 |
62
+ | 10.4218 | 4.0 | 180 | 4.0667 | 0.0110 | 0.0111 | 0.0111 | 0.3384 | 0.8406 |
63
+ | 6.8199 | 5.0 | 225 | 3.8871 | 0.0092 | 0.0100 | 0.0096 | 0.3261 | 0.8623 |
64
+ | 5.4068 | 6.0 | 270 | 3.9234 | 0.0106 | 0.0117 | 0.0111 | 0.3212 | 0.8633 |
65
+ | 4.2552 | 7.0 | 315 | 4.0332 | 0.0115 | 0.0129 | 0.0122 | 0.3168 | 0.8657 |
66
+ | 3.5197 | 8.0 | 360 | 4.2753 | 0.0080 | 0.0088 | 0.0084 | 0.3222 | 0.8647 |
67
+ | 2.8374 | 9.0 | 405 | 4.6031 | 0.0099 | 0.0106 | 0.0102 | 0.3256 | 0.8701 |
68
+ | 2.2784 | 10.0 | 450 | 4.7992 | 0.0118 | 0.0129 | 0.0123 | 0.3237 | 0.8652 |
69
+ | 2.2784 | 11.0 | 495 | 5.0575 | 0.0118 | 0.0129 | 0.0123 | 0.3222 | 0.8652 |
70
+ | 1.8204 | 12.0 | 540 | 5.1371 | 0.0088 | 0.0094 | 0.0091 | 0.3266 | 0.8731 |
71
+ | 1.5073 | 13.0 | 585 | 5.4768 | 0.0109 | 0.0123 | 0.0116 | 0.3133 | 0.8677 |
72
+ | 1.275 | 14.0 | 630 | 5.5963 | 0.0077 | 0.0082 | 0.0079 | 0.3246 | 0.8751 |
73
+
74
+
75
+ ### Framework versions
76
+
77
+ - Transformers 4.55.0
78
+ - Pytorch 2.7.0+cu126
79
+ - Datasets 3.6.0
80
+ - Tokenizers 0.21.4
intent_report_test.txt ADDED
@@ -0,0 +1,75 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ precision recall f1-score support
2
+
3
+ 0 0.90 0.97 0.93 88
4
+ 1 0.85 0.94 0.89 36
5
+ 2 1.00 0.94 0.97 35
6
+ 3 0.91 0.86 0.88 35
7
+ 4 0.77 0.92 0.84 26
8
+ 5 0.00 0.00 0.00 1
9
+ 6 0.77 0.79 0.78 43
10
+ 7 1.00 0.50 0.67 4
11
+ 8 1.00 0.83 0.91 18
12
+ 9 0.96 0.92 0.94 72
13
+ 10 0.97 0.97 0.97 39
14
+ 11 0.83 1.00 0.91 15
15
+ 12 0.71 0.55 0.62 169
16
+ 13 0.95 0.96 0.95 156
17
+ 14 0.79 0.85 0.81 13
18
+ 15 0.71 0.83 0.77 12
19
+ 16 0.83 0.86 0.84 22
20
+ 17 0.65 0.85 0.73 26
21
+ 18 0.89 0.89 0.89 27
22
+ 19 0.78 1.00 0.87 31
23
+ 20 0.88 0.88 0.88 41
24
+ 21 0.86 0.92 0.89 39
25
+ 22 0.80 0.86 0.83 124
26
+ 23 1.00 0.88 0.94 34
27
+ 24 1.00 0.90 0.95 10
28
+ 25 0.95 1.00 0.97 19
29
+ 26 0.94 0.86 0.90 57
30
+ 27 0.87 0.80 0.83 25
31
+ 28 0.33 0.33 0.33 6
32
+ 29 1.00 0.50 0.67 6
33
+ 30 0.91 0.94 0.93 67
34
+ 31 0.89 0.76 0.82 21
35
+ 32 0.73 0.82 0.77 126
36
+ 33 0.95 0.93 0.94 114
37
+ 34 0.96 0.88 0.92 26
38
+ 35 0.91 0.91 0.91 11
39
+ 36 0.78 0.96 0.86 72
40
+ 37 0.00 0.00 0.00 0
41
+ 38 0.79 0.73 0.76 15
42
+ 39 0.88 0.92 0.90 25
43
+ 40 0.95 0.98 0.97 43
44
+ 41 0.67 0.67 0.67 3
45
+ 42 0.84 0.90 0.87 51
46
+ 43 0.84 0.89 0.86 36
47
+ 44 0.96 0.93 0.94 119
48
+ 45 0.91 0.90 0.91 176
49
+ 46 0.88 0.94 0.91 32
50
+ 47 0.99 0.91 0.95 81
51
+ 48 0.95 0.98 0.96 41
52
+ 49 0.77 0.81 0.79 141
53
+ 50 0.95 0.91 0.93 209
54
+ 51 0.94 0.94 0.94 35
55
+ 52 0.95 1.00 0.98 21
56
+ 53 0.92 0.92 0.92 52
57
+ 54 0.92 1.00 0.96 23
58
+ 55 0.80 0.80 0.80 20
59
+ 56 1.00 0.97 0.99 36
60
+ 57 0.89 0.89 0.89 35
61
+ 58 0.92 0.73 0.81 63
62
+ 59 0.91 0.78 0.84 51
63
+
64
+ accuracy 0.88 2974
65
+ macro avg 0.84 0.83 0.83 2974
66
+ weighted avg 0.88 0.88 0.87 2974
67
+
68
+ Confusion matrix:
69
+ [[85 0 0 ... 0 0 0]
70
+ [ 0 34 0 ... 0 0 0]
71
+ [ 0 0 33 ... 0 0 0]
72
+ ...
73
+ [ 0 0 0 ... 31 0 0]
74
+ [ 0 0 0 ... 0 46 0]
75
+ [ 0 0 0 ... 0 1 40]]
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:0162639ab844dd9e7b92b6a36d6d09373d72f898d2b4fa651359f7ea4b0e3550
3
  size 2240362200
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d0a4df39d307944d6a8f628272d6dc58413c6723db25dac6df6aac1f686020d0
3
  size 2240362200
model_predict_test.csv ADDED
The diff for this file is too large to render. See raw diff
 
slot_report_test.txt ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ precision recall f1-score support
2
+
3
+ alarm_type 0.00 0.00 0.00 2
4
+ app_name 0.00 0.00 0.00 5
5
+ artist_name 0.02 0.02 0.02 52
6
+ audiobook_author 0.00 0.00 0.00 5
7
+ audiobook_name 0.00 0.00 0.00 22
8
+ business_name 0.00 0.00 0.00 89
9
+ business_type 0.00 0.00 0.00 27
10
+ change_amount 0.00 0.00 0.00 7
11
+ coffee_type 0.00 0.00 0.00 3
12
+ color_type 0.00 0.00 0.00 9
13
+ cooking_type 0.00 0.00 0.00 8
14
+ currency_name 0.00 0.00 0.00 50
15
+ date 0.01 0.01 0.01 365
16
+ definition_word 0.00 0.00 0.00 51
17
+ device_type 0.00 0.00 0.00 28
18
+ drink_type 0.00 0.00 0.00 1
19
+ email_address 0.00 0.00 0.00 9
20
+ email_folder 0.00 0.00 0.00 5
21
+ event_name 0.00 0.00 0.00 237
22
+ food_type 0.03 0.03 0.03 63
23
+ game_name 0.00 0.00 0.00 23
24
+ general_frequency 0.05 0.06 0.05 17
25
+ house_place 0.00 0.00 0.00 14
26
+ ingredient 0.00 0.00 0.00 5
27
+ joke_type 0.00 0.00 0.00 11
28
+ list_name 0.00 0.00 0.00 61
29
+ meal_type 0.00 0.00 0.00 13
30
+ media_type 0.00 0.00 0.00 119
31
+ movie_name 0.00 0.00 0.00 2
32
+ movie_type 0.00 0.00 0.00 3
33
+ music_album 0.00 0.00 0.00 1
34
+ music_descriptor 0.00 0.00 0.00 4
35
+ music_genre 0.03 0.02 0.02 44
36
+ news_topic 0.00 0.00 0.00 43
37
+ order_type 0.00 0.00 0.00 9
38
+ person 0.01 0.01 0.01 212
39
+ personal_info 0.00 0.00 0.00 13
40
+ place_name 0.00 0.00 0.00 257
41
+ player_setting 0.00 0.00 0.00 35
42
+ playlist_name 0.00 0.00 0.00 12
43
+ podcast_descriptor 0.00 0.00 0.00 21
44
+ podcast_name 0.00 0.00 0.00 16
45
+ radio_name 0.05 0.07 0.06 29
46
+ relation 0.00 0.00 0.00 53
47
+ song_name 0.00 0.00 0.00 34
48
+ sport_type 0.00 0.00 0.00 0
49
+ time 0.02 0.02 0.02 161
50
+ time_zone 0.00 0.00 0.00 13
51
+ timeofday 0.00 0.00 0.00 48
52
+ transport_agency 0.00 0.00 0.00 9
53
+ transport_descriptor 0.00 0.00 0.00 2
54
+ transport_name 0.00 0.00 0.00 4
55
+ transport_type 0.00 0.00 0.00 63
56
+ weather_descriptor 0.02 0.02 0.02 48
57
+
58
+ micro avg 0.01 0.01 0.01 2437
59
+ macro avg 0.00 0.00 0.00 2437
60
+ weighted avg 0.01 0.01 0.01 2437