mohit9999 commited on
Commit
7b230eb
·
verified ·
1 Parent(s): eb894d8

End of training

Browse files
Files changed (1) hide show
  1. README.md +139 -0
README.md ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: peft
3
+ license: llama2
4
+ base_model: meta-llama/Llama-2-7b-hf
5
+ tags:
6
+ - axolotl
7
+ - generated_from_trainer
8
+ datasets:
9
+ - mohit9999/all_news_finance_sm_1h2023_custom
10
+ model-index:
11
+ - name: all_news_finance_sm_1h2023_custom_model_3
12
+ results: []
13
+ ---
14
+
15
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
16
+ should probably proofread and complete it, then remove this comment. -->
17
+
18
+ [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
19
+ <details><summary>See axolotl config</summary>
20
+
21
+ axolotl version: `0.7.0`
22
+ ```yaml
23
+ adapter: qlora
24
+ base_model: meta-llama/Llama-2-7b-hf
25
+ bf16: auto
26
+ dataset_prepared_path: null
27
+ datasets:
28
+ - path: mohit9999/all_news_finance_sm_1h2023_custom
29
+ type: alpaca
30
+ debug: null
31
+ deepspeed: null
32
+ early_stopping_patience: null
33
+ eval_sample_packing: true
34
+ eval_table_size: null
35
+ evals_per_epoch: 1
36
+ flash_attention: false
37
+ fp16: null
38
+ fsdp: null
39
+ fsdp_config: null
40
+ gradient_accumulation_steps: 2
41
+ gradient_checkpointing: true
42
+ group_by_length: false
43
+ hub_model_id: mohit9999/all_news_finance_sm_1h2023_custom_model_3
44
+ learning_rate: 2e-5
45
+ load_in_4bit: true
46
+ load_in_8bit: false
47
+ logging_steps: 1
48
+ lora_alpha: 16
49
+ lora_dropout: 0.05
50
+ lora_fan_in_fan_out: null
51
+ lora_model_dir: null
52
+ lora_modules_to_save:
53
+ - embed_tokens
54
+ - lm_head
55
+ lora_r: 32
56
+ lora_target_linear: true
57
+ lr_scheduler: cosine
58
+ max_steps: 25
59
+ micro_batch_size: 1
60
+ model_type: LlamaForCausalLM
61
+ num_epochs: 1
62
+ optimizer: paged_adamw_8bit
63
+ output_dir: ./outputs/lora-out-3
64
+ pad_to_sequence_len: true
65
+ resume_from_checkpoint: null
66
+ sample_packing: true
67
+ saves_per_epoch: 1
68
+ sdp_attention: true
69
+ sequence_len: 2048
70
+ special_tokens: null
71
+ strict: false
72
+ tf32: false
73
+ tokenizer_type: LlamaTokenizer
74
+ train_on_inputs: false
75
+ val_set_size: 0.1
76
+ wandb_entity: null
77
+ wandb_log_model: null
78
+ wandb_name: null
79
+ wandb_project: null
80
+ wandb_watch: null
81
+ warmup_steps: 1
82
+ weight_decay: 0.0
83
+ xformers_attention: null
84
+
85
+ ```
86
+
87
+ </details><br>
88
+
89
+ # all_news_finance_sm_1h2023_custom_model_3
90
+
91
+ This model is a fine-tuned version of [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) on the mohit9999/all_news_finance_sm_1h2023_custom dataset.
92
+ It achieves the following results on the evaluation set:
93
+ - Loss: 4.9536
94
+
95
+ ## Model description
96
+
97
+ More information needed
98
+
99
+ ## Intended uses & limitations
100
+
101
+ More information needed
102
+
103
+ ## Training and evaluation data
104
+
105
+ More information needed
106
+
107
+ ## Training procedure
108
+
109
+ ### Training hyperparameters
110
+
111
+ The following hyperparameters were used during training:
112
+ - learning_rate: 2e-05
113
+ - train_batch_size: 1
114
+ - eval_batch_size: 1
115
+ - seed: 42
116
+ - distributed_type: multi-GPU
117
+ - num_devices: 8
118
+ - gradient_accumulation_steps: 2
119
+ - total_train_batch_size: 16
120
+ - total_eval_batch_size: 8
121
+ - optimizer: Use paged_adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
122
+ - lr_scheduler_type: cosine
123
+ - lr_scheduler_warmup_steps: 2
124
+ - training_steps: 14
125
+
126
+ ### Training results
127
+
128
+ | Training Loss | Epoch | Step | Validation Loss |
129
+ |:-------------:|:------:|:----:|:---------------:|
130
+ | 4.9043 | 0.9655 | 14 | 4.9536 |
131
+
132
+
133
+ ### Framework versions
134
+
135
+ - PEFT 0.14.0
136
+ - Transformers 4.48.3
137
+ - Pytorch 2.6.0+cu124
138
+ - Datasets 3.2.0
139
+ - Tokenizers 0.21.0