Hugging Face's logo

fage13141
/
fage

Safetensors
Chinese
English
deepseek
lora
chinese
roleplay
chat
Model card Files Community
fage13141 commited on
Commit
2e05d59
·
verified ·
1 Parent(s): 7fef133

Upload folder using huggingface_hub

Browse files
Files changed (8) hide show
  1. README.md +115 -0
  2. adapter_config.json +26 -0
  3. adapter_model.safetensors +3 -0
  4. optimizer.pt +3 -0
  5. rng_state.pth +3 -0
  6. scheduler.pt +3 -0
  7. trainer_state.json +379 -0
  8. training_args.bin +3 -0
README.md ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # DeepSeek-7B-Chat LoRA 微调模型
2
+
3
+ 这是一个基于 DeepSeek-7B-Chat 使用 LoRA 技术微调甄嬛体的模型。
4
+
5
+ ## 模型信息
6
+ - 基础模型: deepseek-ai/deepseek-llm-7b-chat
7
+ - 训练方法: LoRA
8
+ - 检查点: checkpoint-600
9
+ - 上传时间: 2025-02-26 02:31:40
10
+
11
+ ## 环境要求
12
+
13
+ ### Python 版本
14
+ - Python 3.8 或更高版本
15
+
16
+ ### 必需依赖
17
+ ```bash
18
+ pip install torch>=2.0.0
19
+ pip install transformers>=4.35.2
20
+ pip install peft>=0.7.0
21
+ pip install accelerate>=0.25.0
22
+ pip install safetensors>=0.4.1
23
+ ```
24
+
25
+ ### GPU 要求
26
+ - NVIDIA GPU with CUDA support
27
+ - 至少 16GB 显存（推理时）
28
+ - 推荐使用 24GB 或更大显存的 GPU
29
+
30
+ ## 使用方法
31
+
32
+ ### 1. 安装依赖
33
+ ```bash
34
+ # 安装基本依赖
35
+ pip install torch transformers peft accelerate safetensors
36
+
37
+ # 或者指定版本安装
38
+ pip install torch>=2.0.0
39
+ pip install transformers>=4.35.2
40
+ pip install peft>=0.7.0
41
+ pip install accelerate>=0.25.0
42
+ pip install safetensors>=0.4.1
43
+ ```
44
+
45
+ ### 2. 加载模型
46
+ ```python
47
+ from transformers import AutoTokenizer, AutoModelForCausalLM
48
+ from peft import PeftModel
49
+ import torch
50
+
51
+ # 加载基础模型
52
+ base_model = AutoModelForCausalLM.from_pretrained(
53
+ "deepseek-ai/deepseek-llm-7b-chat",
54
+ trust_remote_code=True,
55
+ torch_dtype=torch.half,
56
+ device_map="auto"
57
+ )
58
+
59
+ # 加载 tokenizer
60
+ tokenizer = AutoTokenizer.from_pretrained(
61
+ "deepseek-ai/deepseek-llm-7b-chat",
62
+ use_fast=False,
63
+ trust_remote_code=True
64
+ )
65
+
66
+ # 加载 LoRA 权重
67
+ model = PeftModel.from_pretrained(
68
+ base_model,
69
+ "fage13141/fage",
70
+ torch_dtype=torch.half,
71
+ device_map="auto"
72
+ )
73
+
74
+ # 使用示例
75
+ prompt = "你的提示词"
76
+ inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
77
+ outputs = model.generate(**inputs, max_new_tokens=512)
78
+ response = tokenizer.decode(outputs[0], skip_special_tokens=True)
79
+ print(response)
80
+ ```
81
+
82
+ ### 3. 生成参数说明
83
+ 在 `generate` 函数中，你可以调整以下参数来控制生成效果：
84
+ - max_new_tokens: 生成的最大token数
85
+ - temperature: 温度参数，控制随机性（0.0-1.0）
86
+ - top_p: 控制采样的概率阈值
87
+ - repetition_penalty: 重复惩罚参数
88
+
89
+ 示例：
90
+ ```python
91
+ outputs = model.generate(
92
+ **inputs,
93
+ max_new_tokens=512,
94
+ temperature=0.7,
95
+ top_p=0.9,
96
+ repetition_penalty=1.1
97
+ )
98
+ ```
99
+
100
+ ## 常见问题
101
+
102
+ 1. 显存不足
103
+ - 尝试减小 batch_size
104
+ - 使用 8-bit 量化: `load_in_8bit=True`
105
+ - 使用 CPU 加载: `device_map="cpu"`
106
+
107
+ 2. 模型加载失败
108
+ - 确保已安装所有必需依赖
109
+ - 检查 GPU 显存是否足够
110
+ - 确保网络连接正常
111
+
112
+ ## 引用和致谢
113
+ - 基础模型: [DeepSeek-7B-Chat](https://huggingface.co/deepseek-ai/deepseek-llm-7b-chat)
114
+ - LoRA 方法: [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)
115
+ ```
adapter_config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "auto_mapping": null,
3
+ "base_model_name_or_path": "deepseek-ai/deepseek-llm-7b-chat",
4
+ "bias": "none",
5
+ "fan_in_fan_out": false,
6
+ "inference_mode": true,
7
+ "init_lora_weights": true,
8
+ "layers_pattern": null,
9
+ "layers_to_transform": null,
10
+ "lora_alpha": 32,
11
+ "lora_dropout": 0.1,
12
+ "modules_to_save": null,
13
+ "peft_type": "LORA",
14
+ "r": 8,
15
+ "revision": null,
16
+ "target_modules": [
17
+ "q_proj",
18
+ "k_proj",
19
+ "v_proj",
20
+ "o_proj",
21
+ "gate_proj",
22
+ "up_proj",
23
+ "down_proj"
24
+ ],
25
+ "task_type": "CAUSAL_LM"
26
+ }
adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:32f1cc12dff3e781003365aeb22e2b1745397972d2aa8f80784eb6a659fc7b71
3
+ size 75012288
optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b4f4f3620d95aa7585b93cfc3320a37346a26c8e291f5393aa40284943ed7791
3
+ size 150266878
rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f2fccc77ae50f77bde5b2fb70427571be6dd52ad98309758c52b1d6bd1f83a7f
3
+ size 14244
scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:da3df0af35c9db15fc5f444af87d7cdadb0500803f630c1b72a7a2e301fdd06f
3
+ size 1064
trainer_state.json ADDED
@@ -0,0 +1,379 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 2.569593147751606,
5
+ "eval_steps": 500,
6
+ "global_step": 600,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.04,
13
+ "learning_rate": 9.856938483547926e-05,
14
+ "loss": 3.6649,
15
+ "step": 10
16
+ },
17
+ {
18
+ "epoch": 0.09,
19
+ "learning_rate": 9.713876967095852e-05,
20
+ "loss": 2.813,
21
+ "step": 20
22
+ },
23
+ {
24
+ "epoch": 0.13,
25
+ "learning_rate": 9.570815450643778e-05,
26
+ "loss": 2.7133,
27
+ "step": 30
28
+ },
29
+ {
30
+ "epoch": 0.17,
31
+ "learning_rate": 9.427753934191703e-05,
32
+ "loss": 2.3768,
33
+ "step": 40
34
+ },
35
+ {
36
+ "epoch": 0.21,
37
+ "learning_rate": 9.284692417739628e-05,
38
+ "loss": 2.3128,
39
+ "step": 50
40
+ },
41
+ {
42
+ "epoch": 0.26,
43
+ "learning_rate": 9.141630901287554e-05,
44
+ "loss": 2.2792,
45
+ "step": 60
46
+ },
47
+ {
48
+ "epoch": 0.3,
49
+ "learning_rate": 8.99856938483548e-05,
50
+ "loss": 2.2207,
51
+ "step": 70
52
+ },
53
+ {
54
+ "epoch": 0.34,
55
+ "learning_rate": 8.855507868383405e-05,
56
+ "loss": 2.2776,
57
+ "step": 80
58
+ },
59
+ {
60
+ "epoch": 0.39,
61
+ "learning_rate": 8.71244635193133e-05,
62
+ "loss": 2.3522,
63
+ "step": 90
64
+ },
65
+ {
66
+ "epoch": 0.43,
67
+ "learning_rate": 8.569384835479256e-05,
68
+ "loss": 2.201,
69
+ "step": 100
70
+ },
71
+ {
72
+ "epoch": 0.47,
73
+ "learning_rate": 8.426323319027182e-05,
74
+ "loss": 2.2058,
75
+ "step": 110
76
+ },
77
+ {
78
+ "epoch": 0.51,
79
+ "learning_rate": 8.283261802575107e-05,
80
+ "loss": 2.2312,
81
+ "step": 120
82
+ },
83
+ {
84
+ "epoch": 0.56,
85
+ "learning_rate": 8.140200286123033e-05,
86
+ "loss": 2.2807,
87
+ "step": 130
88
+ },
89
+ {
90
+ "epoch": 0.6,
91
+ "learning_rate": 7.997138769670959e-05,
92
+ "loss": 2.339,
93
+ "step": 140
94
+ },
95
+ {
96
+ "epoch": 0.64,
97
+ "learning_rate": 7.854077253218884e-05,
98
+ "loss": 2.1615,
99
+ "step": 150
100
+ },
101
+ {
102
+ "epoch": 0.69,
103
+ "learning_rate": 7.71101573676681e-05,
104
+ "loss": 2.1912,
105
+ "step": 160
106
+ },
107
+ {
108
+ "epoch": 0.73,
109
+ "learning_rate": 7.567954220314736e-05,
110
+ "loss": 2.1207,
111
+ "step": 170
112
+ },
113
+ {
114
+ "epoch": 0.77,
115
+ "learning_rate": 7.424892703862662e-05,
116
+ "loss": 2.2326,
117
+ "step": 180
118
+ },
119
+ {
120
+ "epoch": 0.81,
121
+ "learning_rate": 7.281831187410587e-05,
122
+ "loss": 2.2286,
123
+ "step": 190
124
+ },
125
+ {
126
+ "epoch": 0.86,
127
+ "learning_rate": 7.138769670958512e-05,
128
+ "loss": 2.03,
129
+ "step": 200
130
+ },
131
+ {
132
+ "epoch": 0.9,
133
+ "learning_rate": 6.995708154506438e-05,
134
+ "loss": 2.2001,
135
+ "step": 210
136
+ },
137
+ {
138
+ "epoch": 0.94,
139
+ "learning_rate": 6.852646638054364e-05,
140
+ "loss": 2.2102,
141
+ "step": 220
142
+ },
143
+ {
144
+ "epoch": 0.99,
145
+ "learning_rate": 6.70958512160229e-05,
146
+ "loss": 2.1974,
147
+ "step": 230
148
+ },
149
+ {
150
+ "epoch": 1.03,
151
+ "learning_rate": 6.566523605150215e-05,
152
+ "loss": 2.046,
153
+ "step": 240
154
+ },
155
+ {
156
+ "epoch": 1.07,
157
+ "learning_rate": 6.42346208869814e-05,
158
+ "loss": 1.9384,
159
+ "step": 250
160
+ },
161
+ {
162
+ "epoch": 1.11,
163
+ "learning_rate": 6.280400572246066e-05,
164
+ "loss": 2.1286,
165
+ "step": 260
166
+ },
167
+ {
168
+ "epoch": 1.16,
169
+ "learning_rate": 6.137339055793991e-05,
170
+ "loss": 2.0289,
171
+ "step": 270
172
+ },
173
+ {
174
+ "epoch": 1.2,
175
+ "learning_rate": 5.9942775393419173e-05,
176
+ "loss": 2.0393,
177
+ "step": 280
178
+ },
179
+ {
180
+ "epoch": 1.24,
181
+ "learning_rate": 5.8512160228898425e-05,
182
+ "loss": 2.0035,
183
+ "step": 290
184
+ },
185
+ {
186
+ "epoch": 1.28,
187
+ "learning_rate": 5.7081545064377684e-05,
188
+ "loss": 2.0714,
189
+ "step": 300
190
+ },
191
+ {
192
+ "epoch": 1.33,
193
+ "learning_rate": 5.565092989985694e-05,
194
+ "loss": 2.0226,
195
+ "step": 310
196
+ },
197
+ {
198
+ "epoch": 1.37,
199
+ "learning_rate": 5.4220314735336195e-05,
200
+ "loss": 2.0695,
201
+ "step": 320
202
+ },
203
+ {
204
+ "epoch": 1.41,
205
+ "learning_rate": 5.278969957081545e-05,
206
+ "loss": 1.989,
207
+ "step": 330
208
+ },
209
+ {
210
+ "epoch": 1.46,
211
+ "learning_rate": 5.135908440629471e-05,
212
+ "loss": 2.0688,
213
+ "step": 340
214
+ },
215
+ {
216
+ "epoch": 1.5,
217
+ "learning_rate": 4.992846924177397e-05,
218
+ "loss": 1.8838,
219
+ "step": 350
220
+ },
221
+ {
222
+ "epoch": 1.54,
223
+ "learning_rate": 4.8497854077253216e-05,
224
+ "loss": 2.0109,
225
+ "step": 360
226
+ },
227
+ {
228
+ "epoch": 1.58,
229
+ "learning_rate": 4.7067238912732475e-05,
230
+ "loss": 2.0371,
231
+ "step": 370
232
+ },
233
+ {
234
+ "epoch": 1.63,
235
+ "learning_rate": 4.563662374821173e-05,
236
+ "loss": 2.0012,
237
+ "step": 380
238
+ },
239
+ {
240
+ "epoch": 1.67,
241
+ "learning_rate": 4.420600858369099e-05,
242
+ "loss": 2.0461,
243
+ "step": 390
244
+ },
245
+ {
246
+ "epoch": 1.71,
247
+ "learning_rate": 4.2775393419170244e-05,
248
+ "loss": 1.9601,
249
+ "step": 400
250
+ },
251
+ {
252
+ "epoch": 1.76,
253
+ "learning_rate": 4.13447782546495e-05,
254
+ "loss": 1.9282,
255
+ "step": 410
256
+ },
257
+ {
258
+ "epoch": 1.8,
259
+ "learning_rate": 3.991416309012876e-05,
260
+ "loss": 2.0841,
261
+ "step": 420
262
+ },
263
+ {
264
+ "epoch": 1.84,
265
+ "learning_rate": 3.848354792560801e-05,
266
+ "loss": 2.0851,
267
+ "step": 430
268
+ },
269
+ {
270
+ "epoch": 1.88,
271
+ "learning_rate": 3.7052932761087265e-05,
272
+ "loss": 2.039,
273
+ "step": 440
274
+ },
275
+ {
276
+ "epoch": 1.93,
277
+ "learning_rate": 3.5622317596566524e-05,
278
+ "loss": 1.9102,
279
+ "step": 450
280
+ },
281
+ {
282
+ "epoch": 1.97,
283
+ "learning_rate": 3.419170243204578e-05,
284
+ "loss": 2.0091,
285
+ "step": 460
286
+ },
287
+ {
288
+ "epoch": 2.01,
289
+ "learning_rate": 3.2761087267525034e-05,
290
+ "loss": 2.0263,
291
+ "step": 470
292
+ },
293
+ {
294
+ "epoch": 2.06,
295
+ "learning_rate": 3.133047210300429e-05,
296
+ "loss": 1.7926,
297
+ "step": 480
298
+ },
299
+ {
300
+ "epoch": 2.1,
301
+ "learning_rate": 2.9899856938483552e-05,
302
+ "loss": 1.8643,
303
+ "step": 490
304
+ },
305
+ {
306
+ "epoch": 2.14,
307
+ "learning_rate": 2.8469241773962807e-05,
308
+ "loss": 1.7537,
309
+ "step": 500
310
+ },
311
+ {
312
+ "epoch": 2.18,
313
+ "learning_rate": 2.7038626609442062e-05,
314
+ "loss": 1.8578,
315
+ "step": 510
316
+ },
317
+ {
318
+ "epoch": 2.23,
319
+ "learning_rate": 2.5608011444921314e-05,
320
+ "loss": 1.7847,
321
+ "step": 520
322
+ },
323
+ {
324
+ "epoch": 2.27,
325
+ "learning_rate": 2.4177396280400573e-05,
326
+ "loss": 1.7704,
327
+ "step": 530
328
+ },
329
+ {
330
+ "epoch": 2.31,
331
+ "learning_rate": 2.2746781115879828e-05,
332
+ "loss": 1.864,
333
+ "step": 540
334
+ },
335
+ {
336
+ "epoch": 2.36,
337
+ "learning_rate": 2.1316165951359084e-05,
338
+ "loss": 1.7659,
339
+ "step": 550
340
+ },
341
+ {
342
+ "epoch": 2.4,
343
+ "learning_rate": 1.9885550786838342e-05,
344
+ "loss": 1.6955,
345
+ "step": 560
346
+ },
347
+ {
348
+ "epoch": 2.44,
349
+ "learning_rate": 1.8454935622317597e-05,
350
+ "loss": 1.7299,
351
+ "step": 570
352
+ },
353
+ {
354
+ "epoch": 2.48,
355
+ "learning_rate": 1.7024320457796853e-05,
356
+ "loss": 1.7581,
357
+ "step": 580
358
+ },
359
+ {
360
+ "epoch": 2.53,
361
+ "learning_rate": 1.5593705293276108e-05,
362
+ "loss": 1.8381,
363
+ "step": 590
364
+ },
365
+ {
366
+ "epoch": 2.57,
367
+ "learning_rate": 1.4163090128755365e-05,
368
+ "loss": 1.731,
369
+ "step": 600
370
+ }
371
+ ],
372
+ "logging_steps": 10,
373
+ "max_steps": 699,
374
+ "num_train_epochs": 3,
375
+ "save_steps": 100,
376
+ "total_flos": 3.739811903756698e+16,
377
+ "trial_name": null,
378
+ "trial_params": null
379
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:be5afeef83251025b2ed9b8cef77039ea44f65ae2dceae4b63b07461fac97f87
3
+ size 4536