--- library_name: peft license: apache-2.0 base_model: mistralai/Mistral-7B-v0.1 tags: - axolotl - generated_from_trainer datasets: - AiAF/input_output_master_list.json model-index: - name: UFOs-Mistral-7B-v0.1-V4 results: [] --- [Built with Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config axolotl version: `0.6.0` ```yaml base_model: mistralai/Mistral-7B-v0.1 # optionally might have model_type or tokenizer_type model_type: AutoModelForCausalLM tokenizer_type: LlamaTokenizer # Automatically upload checkpoint and final model to HF hub_model_id: AiAF/UFOs-Mistral-7B-v0.1-V4 trust_remote_code: true load_in_8bit: false load_in_4bit: true strict: false #chat_template: none datasets: - path: AiAF/input_output_master_list.json type: input_output train_on_inputs: True # datasets: # - path: AiAF/master_list_jsonl # type: chat_template # field_messages: conversations # message_field_role: from # message_field_content: value dataset_prepared_path: last_run_prepared val_set_size: 0.1 output_dir: ./outputs/qlora-out ## You can optionally freeze the entire model and unfreeze a subset of parameters unfrozen_parameters: # - ^lm_head.weight$ # - ^model.embed_tokens.weight$[:32000] # - model.layers.2[0-9]+.block_sparse_moe.gate # - model.layers.2[0-9]+.block_sparse_moe.experts # - model.layers.3[0-9]+.block_sparse_moe.gate # - model.layers.3[0-9]+.block_sparse_moe.experts model_config: output_router_logits: true adapter: qlora lora_model_dir: sequence_len: 2048 sample_packing: true pad_to_sequence_len: true lora_r: 32 lora_alpha: 16 lora_dropout: 0.05 lora_target_linear: true lora_fan_in_fan_out: #lora_target_modules: # - gate # - q_proj # - k_proj # - v_proj # - o_proj # - w1 # - w2 # - w3 wandb_project: wandb_entity: wandb_watch: wandb_name: wandb_log_model: gradient_accumulation_steps: 2 micro_batch_size: 1 num_epochs: 1 optimizer: adamw_bnb_8bit lr_scheduler: cosine learning_rate: 0.0002 ```

# UFOs-Mistral-7B-v0.1-V4 This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the AiAF/input_output_master_list.json dataset. It achieves the following results on the evaluation set: - Loss: 2.0053 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0002 - train_batch_size: 1 - eval_batch_size: 1 - seed: 42 - gradient_accumulation_steps: 2 - total_train_batch_size: 2 - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 14 - num_epochs: 1.0 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:----:|:---------------:| | 0.755 | 1.0 | 500 | 2.0053 | ### Framework versions - PEFT 0.14.0 - Transformers 4.48.1 - Pytorch 2.5.1+cu124 - Datasets 3.2.0 - Tokenizers 0.21.0