phi3_on_korean_summary

README.md

@@ -1,5 +1,5 @@
 ---
-license: other
+license: mit
 library_name: peft
 tags:
 - generated_from_trainer
@@ -12,12 +12,11 @@ model-index:
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment. -->
 
-[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/ghost_in_the_lab/Korean-fine-tune-models/runs/g1zumzc4?nw=nwuserghost_in_the_lab)
 # phi3_on_korean_summary
 
 This model is a fine-tuned version of [microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.5903
+- Loss: 0.6044
 
 ## Model description
 
@@ -37,24 +36,52 @@ More information needed
 
 The following hyperparameters were used during training:
 - learning_rate: 5e-05
-- train_batch_size: 1
-- eval_batch_size: 1
+- train_batch_size: 2
+- eval_batch_size: 2
 - seed: 42
 - gradient_accumulation_steps: 5
-- total_train_batch_size: 5
+- total_train_batch_size: 10
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: linear
 - lr_scheduler_warmup_steps: 50
-- training_steps: 1400
+- training_steps: 500
 - mixed_precision_training: Native AMP
 
 ### Training results
 
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 1.2363        | 0.26  | 20   | 1.1356          |
+| 1.0765        | 0.53  | 40   | 1.0056          |
+| 0.9569        | 0.79  | 60   | 0.8976          |
+| 0.8639        | 1.05  | 80   | 0.8329          |
+| 0.8059        | 1.32  | 100  | 0.7890          |
+| 0.7728        | 1.58  | 120  | 0.7568          |
+| 0.7474        | 1.84  | 140  | 0.7310          |
+| 0.7053        | 2.11  | 160  | 0.7122          |
+| 0.6654        | 2.37  | 180  | 0.6928          |
+| 0.6464        | 2.63  | 200  | 0.6775          |
+| 0.6407        | 2.89  | 220  | 0.6652          |
+| 0.6192        | 3.16  | 240  | 0.6582          |
+| 0.5811        | 3.42  | 260  | 0.6478          |
+| 0.5817        | 3.68  | 280  | 0.6382          |
+| 0.5717        | 3.95  | 300  | 0.6308          |
+| 0.5493        | 4.21  | 320  | 0.6277          |
+| 0.5352        | 4.47  | 340  | 0.6202          |
+| 0.5287        | 4.74  | 360  | 0.6155          |
+| 0.5204        | 5.0   | 380  | 0.6105          |
+| 0.4992        | 5.26  | 400  | 0.6132          |
+| 0.4891        | 5.53  | 420  | 0.6090          |
+| 0.4872        | 5.79  | 440  | 0.6060          |
+| 0.4875        | 6.05  | 460  | 0.6039          |
+| 0.4629        | 6.32  | 480  | 0.6061          |
+| 0.4782        | 6.58  | 500  | 0.6044          |
+
 
 ### Framework versions
 
-- PEFT 0.7.1
-- Transformers 4.41.0.dev0
-- Pytorch 2.1.2
-- Datasets 2.15.0
-- Tokenizers 0.19.1
+- PEFT 0.8.2
+- Transformers 4.38.2
+- Pytorch 2.1.2+cu121
+- Datasets 2.17.0
+- Tokenizers 0.15.0

adapter_config.json

@@ -19,10 +19,11 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "o_proj",
     "qkv_proj",
     "down_proj",
+    "o_proj",
     "gate_up_proj"
   ],
-  "task_type": "CAUSAL_LM"
+  "task_type": "CAUSAL_LM",
+  "use_rslora": false
 }

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d93fd911e76ba4aaf4dc925090fbe94c3e8f5ffde030ed875ddffb6098233679
+oid sha256:9dd2d1d50d827e6f916b726a5f4e46e1e97935ef0652d28306cec87b3d86a934
 size 100697728

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d48a4bfd03c5b213eadb0d4ea8bc6e9b8912d11c9e1ec2de5cca71a449f7138b
-size 5112
+oid sha256:f04837b222bab4c6b3f0d7d3bf4b2968b57842b3a1400556cb6b7fd37c2f35cd
+size 4920