lesso02
/

561cc045-d57d-4d40-932e-390287a2eaac

+---
+library_name: peft
+license: mit
+base_model: fxmarty/really-tiny-falcon-testing
+tags:
+- axolotl
+- generated_from_trainer
+model-index:
+- name: 561cc045-d57d-4d40-932e-390287a2eaac
+  results: []
+---
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
+<details><summary>See axolotl config</summary>
+axolotl version: `0.4.1`
+```yaml
+adapter: lora
+base_model: fxmarty/really-tiny-falcon-testing
+bf16: auto
+chat_template: llama3
+dataset_prepared_path: null
+datasets:
+- data_files:
+  - 415dedee96180e11_train_data.json
+  ds_type: json
+  format: custom
+  path: /workspace/input_data/415dedee96180e11_train_data.json
+  type:
+    field_instruction: instruction
+    field_output: response
+    format: '{instruction}'
+    no_input_format: '{instruction}'
+    system_format: '{system}'
+    system_prompt: ''
+debug: null
+deepspeed: null
+do_eval: true
+early_stopping_patience: 3
+eval_batch_size: 4
+eval_max_new_tokens: 128
+eval_steps: 500
+evals_per_epoch: null
+flash_attention: false
+fp16: false
+fsdp: null
+fsdp_config: null
+gradient_accumulation_steps: 8
+gradient_checkpointing: true
+group_by_length: true
+hub_model_id: lesso02/561cc045-d57d-4d40-932e-390287a2eaac
+hub_repo: null
+hub_strategy: checkpoint
+hub_token: null
+learning_rate: 0.000202
+load_in_4bit: false
+load_in_8bit: false
+local_rank: null
+logging_steps: 50
+lora_alpha: 128
+lora_dropout: 0.15
+lora_fan_in_fan_out: null
+lora_model_dir: null
+lora_r: 64
+lora_target_linear: true
+lr_scheduler: cosine
+max_grad_norm: 1.0
+max_steps: 25000
+micro_batch_size: 4
+mlflow_experiment_name: /tmp/415dedee96180e11_train_data.json
+model_type: AutoModelForCausalLM
+num_epochs: 10
+optimizer: adamw_torch_fused
+output_dir: miner_id_24
+pad_to_sequence_len: true
+resume_from_checkpoint: null
+s2_attention: null
+sample_packing: false
+save_steps: 500
+saves_per_epoch: null
+seed: 20
+sequence_len: 1024
+strict: false
+tf32: true
+tokenizer_type: AutoTokenizer
+train_on_inputs: false
+trust_remote_code: true
+val_set_size: 0.05
+wandb_entity: null
+wandb_mode: online
+wandb_name: 5941f67d-1b56-4ae0-b76d-52a8681c66f9
+wandb_project: 02a
+wandb_run: your_name
+wandb_runid: 5941f67d-1b56-4ae0-b76d-52a8681c66f9
+warmup_steps: 100
+weight_decay: 0.0
+xformers_attention: null
+```
+</details><br>
+# 561cc045-d57d-4d40-932e-390287a2eaac
+This model is a fine-tuned version of [fxmarty/really-tiny-falcon-testing](https://huggingface.co/fxmarty/really-tiny-falcon-testing) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 10.9172
+## Model description
+More information needed
+## Intended uses & limitations
+More information needed
+## Training and evaluation data
+More information needed
+## Training procedure
+### Training hyperparameters
+The following hyperparameters were used during training:
+- learning_rate: 0.000202
+- train_batch_size: 4
+- eval_batch_size: 4
+- seed: 20
+- gradient_accumulation_steps: 8
+- total_train_batch_size: 32
+- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 100
+- training_steps: 25000
+### Training results
+| Training Loss | Epoch  | Step  | Validation Loss |
+|:-------------:|:------:|:-----:|:---------------:|
+| No log        | 0.0002 | 1     | 11.0859         |
+| 87.8237       | 0.1129 | 500   | 10.9719         |
+| 87.7348       | 0.2258 | 1000  | 10.9600         |
+| 87.6865       | 0.3386 | 1500  | 10.9512         |
+| 87.6515       | 0.4515 | 2000  | 10.9449         |
+| 87.6171       | 0.5644 | 2500  | 10.9406         |
+| 87.5821       | 0.6773 | 3000  | 10.9366         |
+| 87.5827       | 0.7902 | 3500  | 10.9341         |
+| 87.5556       | 0.9031 | 4000  | 10.9321         |
+| 87.5578       | 1.0159 | 4500  | 10.9300         |
+| 87.5512       | 1.1288 | 5000  | 10.9284         |
+| 87.5338       | 1.2417 | 5500  | 10.9271         |
+| 87.5222       | 1.3546 | 6000  | 10.9265         |
+| 87.5199       | 1.4675 | 6500  | 10.9258         |
+| 87.5171       | 1.5804 | 7000  | 10.9247         |
+| 87.5118       | 1.6932 | 7500  | 10.9238         |
+| 87.5051       | 1.8061 | 8000  | 10.9233         |
+| 87.508        | 1.9190 | 8500  | 10.9228         |
+| 87.4979       | 2.0319 | 9000  | 10.9221         |
+| 87.4973       | 2.1448 | 9500  | 10.9216         |
+| 87.512        | 2.2577 | 10000 | 10.9213         |
+| 87.4823       | 2.3705 | 10500 | 10.9210         |
+| 87.4886       | 2.4834 | 11000 | 10.9207         |
+| 87.4873       | 2.5963 | 11500 | 10.9202         |
+| 87.4897       | 2.7092 | 12000 | 10.9200         |
+| 87.4782       | 2.8221 | 12500 | 10.9196         |
+| 87.4864       | 2.9350 | 13000 | 10.9196         |
+| 87.4777       | 3.0478 | 13500 | 10.9194         |
+| 87.4821       | 3.1607 | 14000 | 10.9189         |
+| 87.479        | 3.2736 | 14500 | 10.9188         |
+| 87.4648       | 3.3865 | 15000 | 10.9185         |
+| 87.4757       | 3.4994 | 15500 | 10.9184         |
+| 87.4546       | 3.6122 | 16000 | 10.9184         |
+| 87.4722       | 3.7251 | 16500 | 10.9183         |
+| 87.4617       | 3.8380 | 17000 | 10.9179         |
+| 87.4607       | 3.9509 | 17500 | 10.9181         |
+| 87.4602       | 4.0638 | 18000 | 10.9178         |
+| 87.4577       | 4.1767 | 18500 | 10.9176         |
+| 87.4592       | 4.2895 | 19000 | 10.9175         |
+| 87.4784       | 4.4024 | 19500 | 10.9174         |
+| 87.4644       | 4.5153 | 20000 | 10.9173         |
+| 87.4677       | 4.6282 | 20500 | 10.9174         |
+| 87.465        | 4.7411 | 21000 | 10.9174         |
+| 87.4606       | 4.8540 | 21500 | 10.9172         |
+| 87.4689       | 4.9668 | 22000 | 10.9173         |
+| 87.4515       | 5.0797 | 22500 | 10.9172         |
+| 87.4637       | 5.1926 | 23000 | 10.9172         |
+| 87.4589       | 5.3055 | 23500 | 10.9172         |
+| 87.4596       | 5.4184 | 24000 | 10.9172         |
+| 87.461        | 5.5313 | 24500 | 10.9173         |
+| 87.4639       | 5.6441 | 25000 | 10.9172         |
+### Framework versions
+- PEFT 0.13.2
+- Transformers 4.46.0
+- Pytorch 2.5.0+cu124
+- Datasets 3.0.1
+- Tokenizers 0.20.1

adapter_model.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:edf56a0cc92455f1c9262fb79086620c0d1803589b4b8e2f27f64b6db1cbe8fc
+size 252062