ysr/hyperparam-rust-sft-lora

README.md

@@ -20,7 +20,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [deepseek-ai/deepseek-coder-1.3b-base](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base) on the generator dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.4725
+- Loss: 0.4360
 
 ## Model description
 
@@ -49,15 +49,20 @@ The following hyperparameters were used during training:
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_ratio: 0.05
 - lr_scheduler_warmup_steps: 1
-- num_epochs: 2
+- num_epochs: 5
 
 ### Training results
 
 | Training Loss | Epoch | Step | Validation Loss |
 |:-------------:|:-----:|:----:|:---------------:|
-| 0.6643        | 0.59  | 25   | 0.5277          |
-| 0.4808        | 1.18  | 50   | 0.4792          |
-| 0.4569        | 1.77  | 75   | 0.4725          |
+| 0.6641        | 0.59  | 25   | 0.5224          |
+| 0.4747        | 1.18  | 50   | 0.4685          |
+| 0.4392        | 1.77  | 75   | 0.4519          |
+| 0.4378        | 2.37  | 100  | 0.4437          |
+| 0.4189        | 2.96  | 125  | 0.4389          |
+| 0.4209        | 3.55  | 150  | 0.4367          |
+| 0.4201        | 4.14  | 175  | 0.4360          |
+| 0.4163        | 4.73  | 200  | 0.4360          |
 
 
 ### Framework versions

adapter_config.json

@@ -20,10 +20,10 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "q_proj",
-    "k_proj",
+    "v_proj",
     "o_proj",
-    "v_proj"
+    "k_proj",
+    "q_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fabbe8ef8028413c720fc343bd9bf525408afa2f108c5b0d474578ff93d5ce0d
+oid sha256:2fd6460db179ca1736b9c7447f6f1ac3ef8d5ca5bc0c962a0a70f4f375627c71
 size 50357632

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e621433630116961a3628891344142314658f4cb308cd6d231a64cb923ac369d
+oid sha256:d3d201d74214f993646ae033925c94201c44cfcc9da3d21d28f5ab25ce99f8ac
 size 4920