Visualize in Weights & Biases

Meta-Llama-3-8B-Instruct-ORPO-QLoRA

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the HuggingFaceH4/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:

  • Loss: 0.5734
  • Rewards/chosen: -0.0085
  • Rewards/rejected: -0.0105
  • Rewards/accuracies: 0.6070
  • Rewards/margins: 0.0020
  • Logps/rejected: -1.0492
  • Logps/chosen: -0.8470
  • Logits/rejected: -0.2321
  • Logits/chosen: -0.2275
  • Nll Loss: 0.5669
  • Log Odds Ratio: -0.6615
  • Log Odds Chosen: 0.3163

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 7e-06
  • train_batch_size: 4
  • eval_batch_size: 8
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 2
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 32
  • total_eval_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss Rewards/chosen Rewards/rejected Rewards/accuracies Rewards/margins Logps/rejected Logps/chosen Logits/rejected Logits/chosen Nll Loss Log Odds Ratio Log Odds Chosen
0.8633 0.0524 100 0.7181 -0.0135 -0.0158 0.6060 0.0023 -1.5779 -1.3476 -0.4503 -0.4466 0.7126 -0.6965 0.2913
0.7831 0.1048 200 0.6487 -0.0105 -0.0125 0.6140 0.0020 -1.2499 -1.0520 -0.3621 -0.3619 0.6432 -0.6627 0.2691
0.7146 0.1572 300 0.6238 -0.0102 -0.0122 0.6140 0.0020 -1.2194 -1.0173 -0.3196 -0.3169 0.6181 -0.6594 0.2790
0.7361 0.2096 400 0.6137 -0.0100 -0.0120 0.6140 0.0020 -1.2012 -1.0014 -0.2841 -0.2811 0.6078 -0.6618 0.2770
0.7382 0.2620 500 0.6066 -0.0099 -0.0119 0.6120 0.0020 -1.1884 -0.9868 -0.3023 -0.2982 0.6006 -0.6603 0.2812
0.7339 0.3143 600 0.6009 -0.0097 -0.0118 0.6100 0.0020 -1.1751 -0.9714 -0.2544 -0.2490 0.5948 -0.6587 0.2859
0.7133 0.3667 700 0.5968 -0.0096 -0.0116 0.6070 0.0020 -1.1590 -0.9588 -0.2830 -0.2764 0.5906 -0.6590 0.2828
0.6988 0.4191 800 0.5926 -0.0095 -0.0115 0.6070 0.0020 -1.1491 -0.9451 -0.2817 -0.2745 0.5864 -0.6576 0.2898
0.7493 0.4715 900 0.5882 -0.0093 -0.0114 0.6080 0.0021 -1.1357 -0.9301 -0.2547 -0.2476 0.5820 -0.6552 0.2952
0.7022 0.5239 1000 0.5842 -0.0091 -0.0111 0.6070 0.0020 -1.1110 -0.9090 -0.2588 -0.2514 0.5780 -0.6569 0.2962
0.6805 0.5763 1100 0.5807 -0.0089 -0.0108 0.6020 0.0020 -1.0833 -0.8865 -0.2590 -0.2519 0.5744 -0.6608 0.2937
0.6427 0.6287 1200 0.5780 -0.0087 -0.0107 0.6070 0.0020 -1.0670 -0.8682 -0.2483 -0.2430 0.5717 -0.6609 0.3024
0.6762 0.6811 1300 0.5762 -0.0086 -0.0106 0.6070 0.0020 -1.0576 -0.8586 -0.2376 -0.2322 0.5698 -0.6618 0.3069
0.6944 0.7335 1400 0.5750 -0.0085 -0.0105 0.6070 0.0020 -1.0548 -0.8542 -0.2468 -0.2420 0.5686 -0.6609 0.3102
0.6695 0.7859 1500 0.5742 -0.0085 -0.0105 0.6080 0.0020 -1.0505 -0.8493 -0.2426 -0.2372 0.5678 -0.6616 0.3135
0.7258 0.8382 1600 0.5738 -0.0085 -0.0105 0.6080 0.0020 -1.0497 -0.8485 -0.2418 -0.2371 0.5673 -0.6619 0.3140
0.7193 0.8906 1700 0.5735 -0.0085 -0.0105 0.6050 0.0020 -1.0499 -0.8477 -0.2403 -0.2352 0.5671 -0.6610 0.3162
0.7038 0.9430 1800 0.5734 -0.0085 -0.0105 0.6090 0.0020 -1.0493 -0.8471 -0.2360 -0.2311 0.5670 -0.6615 0.3164
0.6723 0.9954 1900 0.5734 -0.0085 -0.0105 0.6070 0.0020 -1.0493 -0.8470 -0.2369 -0.2320 0.5669 -0.6615 0.3168

Framework versions

  • PEFT 0.11.1
  • Transformers 4.41.0
  • Pytorch 2.3.0+cu121
  • Datasets 2.19.1
  • Tokenizers 0.19.1
Downloads last month
11
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model has no pipeline_tag.

Model tree for statking/Meta-Llama-3-8B-Instruct-ORPO-QLoRA

Adapter
(756)
this model

Dataset used to train statking/Meta-Llama-3-8B-Instruct-ORPO-QLoRA